- What is AVIATOR?
- I. Installation
- II. LLM Configuration
- III. Usage
- CWE selector (beta)
- Script Arguments
- Repository Structure
- Datasets
- Downstream Vulnerability Detection
- Abstract
- How to cite
- License
- Disclaimer
AVIATOR is an AI-agentic framework for automated vulnerability injection in source code. It produces high-fidelity, large-scale code security datasets by orchestrating specialized AI agents, RAG, and tool-based analysis in a multi-step workflow.
Reproducible runs with no local Python setup. The image includes Python dependencies, ESBMC, cppcheck, PrimeVul data, and the RAG index.
Prerequisites: Docker, Docker Compose
docker compose buildConfigure the LLM via .env before running; see LLM Configuration → 1. .env file.
For a lean image without PrimeVul baked in: docker compose build --build-arg SKIP_PRIMEVUL_DOWNLOAD=1, then mount ./data at runtime.
Prerequisites: Python ≥ 3.11, uv
git clone https://github.com/your-username/aviator.git
cd AVIATOR/
./scripts/setup_aviator.shOptions: --skip-esbmc, --skip-cppcheck, --skip-download (use existing data in ./data/PrimeVul_v0.1).
Copy .env.example to .env and set values. Docker Compose loads them automatically. No JSON edits needed.
| Model source | Variable | Description |
|---|---|---|
| OpenAI-compatible API (option 1) | AVIATOR_LLM_BASE_URL |
API base URL |
AVIATOR_LLM_API_KEY |
API key | |
AVIATOR_LLM_MODEL |
Model name | |
| HuggingFace (option 2) | AVIATOR_LLM_PATH |
Model path for in-process HuggingFace model |
| Fine-tuned model (optional) | AVIATOR_LLM_FINETUNED_PATH |
LoRA adapter path (FT workflow vul_inject_SFT agent) |
Per-LLM: AVIATOR_LLM_<ID>_* (e.g. AVIATOR_LLM_MAIN_LLM_BASE_URL, AVIATOR_LLM_VUL_INJECT_SFT_PATH).
Edit workflow JSON files in the codebase directly (e.g. vul_code_gen/AVIATOR_13steps_full_workflow/vul_code_gen_workflow_noFT.json).
Use any hosted API (OpenAI, Together, Groq, etc.) or a local model served via vLLM:
"llm": {
"type": "OpenAICompatibleLLM",
"model": "Qwen/Qwen2.5-Coder-32B-Instruct",
"base_url": "host-url",
"api_key": "your-api-key"
}Example for local vLLM:
vllm serve Qwen/Qwen2.5-Coder-32B-Instruct --port 18446 --dtype bfloat16 --tensor-parallel-size 2 --max_model_len 32000 --disable-log-requestsSet base_url to http://localhost:18446/v1 in the workflow JSON.
Run a model in-process (no separate server):
"llm": {
"type": "HFpipeline",
"llm_path": "Qwen/Qwen2.5-Coder-32B-Instruct"
}For the FT workflow, set llm_path in the vul_inject_SFT LLM section to your LoRA adapter output (e.g. outputs or Qwen2.5-coder-GRPO/full_GRPO). See LoRA_FT/README.md.
Run injection. Output is saved to injected_vul_code/<dataset_basename>/ by default.
# Quick test (45 samples, always available):
docker compose run aviator scripts/run_injection.sh \
--dataset-path validation_dataset/primevul_paired_test_45.jsonl
# Full PrimeVul test (requires data/ in image):
docker compose run aviator scripts/run_injection.sh \
--dataset-path /AVIATOR/data/PrimeVul_v0.1/primevul_test.jsonlRun injection with evaluation (FormAI, SARD100 with CodeBLEU and optional ESBMC):
docker compose run aviator scripts/run_injection_with_eval.sh \
--dataset-path validation_dataset/formai_with_benign_37.jsonl --run-esbmcRun vulnerability injection:
./scripts/run_injection.sh --dataset-path /path/to/primevul_test.jsonlRun injection with evaluation (CodeBLEU; optional ESBMC for sard100/formai):
./scripts/run_injection_with_eval.sh --dataset-path validation_dataset/formai_with_benign_37.jsonl --run-esbmcThe 14-steps CWE selector workflow (vul_code_gen/AVIATOR_14steps_CWEselector/) adds an extra step to automatically find a suitable vulnerability type (CWE) to inject when your dataset does not include a pre-defined target CWE per example. This is useful for datasets that contain only benign code (e.g. FormAI), where you want the workflow to choose which vulnerability to inject.
CWE selection modes:
-
Weighted random selection (
vulnInjectID_selector_probabilistic): Samples a CWE using weights fromvulnerability_probabilities.json. Runs as a single all-in-one workflow. -
Agentic selection (
vulnInjectID_selector): An LLM analyzes the benign code and selects a contextually suitable CWE. Uses a two-step pipeline: first select CWEs, then run injection.
Single workflow — CWE selection and injection in one run:
./scripts/run_injection.sh --dataset-path /path/to/benign_only.jsonl \
--workflow-json vul_code_gen/AVIATOR_14steps_CWEselector/vul_code_gen_workflow_noFT_CWEselector_random.jsonTwo-step pipeline — first obtain CWEs via AI analysis, then run injection:
# Step 1: Select CWEs (AI agent analyzes code and picks suitable vulnerability types)
./scripts/run_injection.sh --dataset-path /path/to/benign_only.jsonl \
--workflow-json vul_code_gen/AVIATOR_14steps_CWEselector/vul_code_gen_workflow_noFT_before_CWEselector.json \
--output-file /path/to/with_cwe_ids.jsonl
# Step 2: Run injection using the CWE-enriched output from step 1
./scripts/run_injection.sh --dataset-path /path/to/with_cwe_ids.jsonl \
--workflow-json vul_code_gen/AVIATOR_14steps_CWEselector/vul_code_gen_workflow_noFT_after_CWEselector.jsonThe intermediate output from step 1 must include benign_code, vul_inject_id, function_purpose, and the other fields expected by the after workflow. You may need to adapt the runner or use a custom script to capture and pass this intermediate output.
Beta disclaimer: The CWE selector is in beta. Results are less robust than the regular 13-steps workflow where the target CWE is provided in the input dataset (e.g. PrimeVul). Use it for exploratory runs or when CWE labels are not available.
| Argument | Required | Description |
|---|---|---|
--dataset-path |
Yes | Input dataset (JSONL) |
--output-file |
No | Output JSONL (default: injected_vul_code/<input_basename>/aviator_injection.jsonl) |
--workflow-json |
No | Workflow config (default: vul_code_gen_workflow_noFT.json) |
--percent |
No | Percentage of dataset to process (default: 100.0) |
--dataset-type |
No | primevul, sard100, or formai (default: primevul) |
| Argument | Required | Description |
|---|---|---|
--dataset-path |
Yes | Input dataset (JSONL) |
--output-file |
No | Output JSONL |
--workflow-json |
No | Workflow config |
--percent |
No | Percentage of dataset (default: 100.0) |
--dataset-type |
No | primevul, sard100, or formai (default: primevul) |
--run-codebleu |
No | Compute CodeBLEU (default: on) |
--run-esbmc |
No | Run ESBMC (sard100/formai only) |
| Argument | Required | Description |
|---|---|---|
--dataset_path |
Yes | Path to dataset file |
--workflow_json |
Yes | Path to workflow JSON |
--output_file |
Yes | Output JSONL path |
--percent |
No | Percentage to process (default: 100.0) |
--dataset_type |
No | primevul, sard100, or formai (default: primevul) |
| Argument | Required | Description |
|---|---|---|
--data_to_test_path |
Yes | Path to generated/injected code (JSONL) |
--reference_dataset_path |
Yes | Path to reference ground truth |
--dataset_type |
Yes | primevul, sard100, or formai |
--run_codebleu |
No | Compute CodeBLEU |
--run_esbmc |
No | Run ESBMC (sard100/formai only) |
| Argument | Required | Description |
|---|---|---|
--primevul_paired_path |
Yes | Path to primevul_train_paired.jsonl |
| Directory | Description |
|---|---|
| awe | Low-code library for AI agentic workflows |
| LoRA_FT | Fine-tuning code (SFT, GRPO, LoRA) |
| vul_code_gen | Injection workflow, agents, RAG, scripts |
| vul_code_gen/AVIATOR_14steps_CWEselector | Beta CWE-selector workflow (auto-selects vulnerability type) |
| validation_dataset | Benchmark data links |
| scripts/ | Setup and run scripts |
- PrimeVul (training and RAG): Google Drive — downloaded by setup to
./data/PrimeVul_v0.1 - FormAI: FormAI Dataset
- SARD100: SARD
For the experiments evaluating impact on downstream vulnerability detection, we used VulScribeR, the official repository for the paper VulScribeR: Exploring RAG-based Vulnerability Augmentation with LLMs.
The increasing complexity of software systems and the sophistication of cyber-attacks have underscored the critical need for reliable automated software vulnerability detection. Data-driven approaches using deep learning models show promise but critically depend on the availability of large, accurately labeled datasets. Automated vulnerability injection provides a way to address these limitations, but existing techniques remain limited in coverage, contextual fidelity, or injection success.
AVIATOR is the first AI-agentic vulnerability injection framework. It decomposes vulnerability injection into a coordinated workflow of specialized AI agents, tool-based analysis, and iterative self-correction, with RAG and LoRA-based fine-tuning to produce realistic, category-specific vulnerabilities.
@article{aviator2025,
title = {AI Agentic Vulnerability Injection And Transformation with Optimized Reasoning},
author = {Lbath, Amine and Amini, Massih-Reza and Delaitre, Aurelien and Okun, Vadim},
year = {2025},
eprint = {2508.20866},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2508.20866},
}MIT License — see LICENSE.
NIST-developed software is provided by NIST as a public service. You may use, copy, and distribute copies of the software in any medium, provided that you keep intact this entire notice. You may improve, modify, and create derivative works of the software or any portion of the software, and you may copy and distribute such modifications or works. Modified works should carry a notice stating that you changed the software and should note the date and nature of any such change. Please explicitly acknowledge the National Institute of Standards and Technology as the source of the software.
NIST-developed software is expressly provided "AS IS." NIST MAKES NO WARRANTY OF ANY KIND, EXPRESS, IMPLIED, IN FACT, OR ARISING BY OPERATION OF LAW, INCLUDING, WITHOUT LIMITATION, THE IMPLIED WARRANTY OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE, NON-INFRINGEMENT, AND DATA ACCURACY. NIST NEITHER REPRESENTS NOR WARRANTS THAT THE OPERATION OF THE SOFTWARE WILL BE UNINTERRUPTED OR ERROR-FREE, OR THAT ANY DEFECTS WILL BE CORRECTED. NIST DOES NOT WARRANT OR MAKE ANY REPRESENTATIONS REGARDING THE USE OF THE SOFTWARE OR THE RESULTS THEREOF, INCLUDING BUT NOT LIMITED TO THE CORRECTNESS, ACCURACY, RELIABILITY, OR USEFULNESS OF THE SOFTWARE.
You are solely responsible for determining the appropriateness of using and distributing the software and you assume all risks associated with its use, including but not limited to the risks and costs of program errors, compliance with applicable laws, damage to or loss of data, programs or equipment, and the unavailability or interruption of operation. This software is not intended to be used in any situation where a failure could cause risk of injury or damage to property. The software developed by NIST employees is not subject to copyright protection within the United States.