AutoSec-RAN: Specification-Grounded LLM Fine-Tuning for Automated Security Analysis of O-RAN

I. INTRODUCTION

• The framework targets threats to open fronthaul, including spoofed control-plane traffic, DoS attacks, and timing disruption that may evade conventional alarm thresholds.The motivating passage also describes correlating eCPRI payload metadata, wire-direction labels, receive-window counters, and volumetric statistics.Source: §I · p. 1
• O-RAN fronthaul security analysis must integrate threat-model, cryptographic, and eCPRI specifications to distinguish spoofing from misconfiguration and transient degradation from benign load variation.The cited governing sources include O-RAN WG11, 3GPP TS 33.501, and eCPRI.Source: §I · p. 1
• AutoSec-RAN combines RAG over a curated specification corpus with LoRA inference in a two-stage supervised fine-tuning pipeline, producing structured security triage from scenario metadata without raw packet evidence.Stage 1.1 trains on 3,163 analyst question-answer pairs generated from 69 curated O-RAN, 3GPP, and adjacent standards documents. Stage 1.2 warm-starts from the Stage 1.1 checkpoint and adds 1,200 observation-driven pairs from the 5G-based ORAN dataset.Source: §I · pp. 1–2
• The evaluation uses a 5G/O-RAN testbed and labeled fronthaul DDoS replay scenarios, scoring answers against references authored from full PCAP and log evidence.At inference, the model receives only an investigator-style question and a seven-field scenario metadata block; the judge scores faithfulness, correctness, and actionability.Source: §I · p. 1, Figure 1 · p. 2
• The contributions include a multi-axis evaluation rubric, six-class performance analysis, metadata-only triage, and reproducible synthetic-pair generation grounded in security specifications.The framework is positioned as a scenario triage assistant, not a packet-level classifier or anomaly detector.Source: §I · p. 2

II. PRIOR WORK

• Prior work covers O-RAN knowledge benchmarking, conformance testing, specification-guided fuzzing, runtime detection –[14], and telecom-standard question answering.The approaches include benchmark evaluation, cooperative LLM testing, FSM-guided fuzzing, detector-explainer systems, rule-based detection, and zero-shot DoS analysis.Source: §II · p. 2
• AutoSec-RAN differs through specification-grounded LoRA fine-tuning, standalone triage from metadata, a judge rubric plus semantic check, and end-to-end physical-testbed validation.This contrasts with zero-shot prompting [14], pretrained-explainer chaining, detector-coupled explanations, and signature-based alerts.Source: §II · p. 2

III. AUTOSEC-RAN TESTING METHODOLOGY AND COMPONENTS

• The pipeline curates standards, generates analyst-style question-answer pairs, fine-tunes Mistral-7B-Instruct with two-stage LoRA, then retrieves from the same indexes and evaluates outputs with a Gemma judge.The architecture includes separate offline training and inference/evaluation paths.Source: §III · p. 2, Figure 2 · p. 3

A. Specification Corpus: Curation and Indexing

• The reviewed corpus retains 69 security-relevant documents from 85 candidate paths and indexes child chunks for FAISS search, with parent chunks supplied as RAG context.It covers O-RAN WG11, 3GPP TS 33.501, and complementary NIST, ETSI, IETF, and GSMA material; the two stores contain 4,175 parent chunks total.Source: A. Specification Corpus: Curation and… · p. 3

B. Training Data Generation

• Gemma generates specification-grounded analyst pairs across five taxonomy classes, while attack-direction reasoning is withheld for evaluation; one in five pairs receives a source-support audit.The 5,448-pair manifest includes 3,163 specification pairs and 1,200 observation-driven pairs, with 815 validation and 270 test pairs.Source: B. Training Data Generation · pp. 3–4

C. Two-Stage LoRA Fine-Tuning

• Stage 1.1 trains a specification-grounded adapter, then Stage 1.2 warm-starts from it at a reduced learning rate while adding empirical flow observations.Both stages train for three epochs; the learning rates are 2 × 10^-4 for Stage 1.1 and 5 × 10^-5 for Stage 1.2.Source: C. Two-Stage LoRA Fine-Tuning · p. 4

A. Evaluation Criteria

• The framework evaluates analyst triage on distinct dimensions because technical accuracy alone does not ensure faithful claims or actionable guidance.The section introduces four interpretable sub-scores for response quality.Source: A. Evaluation Criteria · p. 4
• Faithfulness penalizes unsupported claims, correctness checks technical claims against references grounded in PCAP and logs, and actionability measures direct usefulness for analyst work.Fabricated protocol behavior or nonexistent requirements may lead analysts to misconfigure defenses or dismiss genuine threats.Source: A. Evaluation Criteria · p. 4
• Coverage tracks required must-mention concepts but is excluded from the weighted overall score because rubric lists vary across taxonomy classes.It is retained as a diagnostic for systematic gaps in conceptual completeness.Source: A. Evaluation Criteria · p. 4

• Each evaluation unit is scored for faithfulness, correctness, actionability, and coverage on a continuous [0, 1] scale.The dimensions assess unsupported claims, agreement with the reference, analyst usability, and coverage of required concepts, respectively.Source: B. Evaluation Score · p. 4
• The per-item overall score weights faithfulness at 0.3, correctness at 0.5, and actionability at 0.2; coverage is reported diagnostically but excluded.Source: B. Evaluation Score · p. 4
• Raw cosine similarity is rescaled to the unit interval for use in the blended score.Source: B. Evaluation Score · p. 4
• A set-level pass rate counts items whose overall score meets threshold τ, calibrated to the minimum quality useful for analyst decision-making.Source: B. Evaluation Score · p. 4

C. Reference Answer and Rubric Construction

• Reference answers are authored offline from PCAP, gNodeB and RU logs, outcome labels, and timing statistics, giving the benchmark a richer evidence base than the model receives.The model's seven-field metadata constraint is intentional; references represent what a fully informed analyst would produce.Source: C. Reference Answer and Rubric… · pp. 4–5
• Per-item rubrics specify required concepts and prohibited evidence-specific values or log tokens, distinguishing completeness from potential data leakage.Source: C. Reference Answer and Rubric… · p. 5

D. Judge LLM

• Gemma-4-31B-IT judges candidates using the investigator question, reference answer, candidate answer, and item rubric, returning four scores and a rationale.The final output omits chain-of-thought scratchpad to prevent the rationale from influencing score calibration.Source: D. Judge LLM · p. 5
• The judge prompt keeps the reference and candidate order fixed; malformed outputs are excluded, and reported scores receive no manual corrections.Source: D. Judge LLM · p. 5
• The evaluation uses fixed prompt ordering, strict parsing with up to three low-temperature retries, and no manual correction to mitigate known single-judge biases.The BGE cosine signal is also used as an independent semantic check on judge verdicts.Source: D. Judge LLM · p. 5
• Two Qwen2.5 reference judges rescore the same candidates with the same references, prompts, and weights, varying only the judge model.Source: D. Judge LLM · p. 5
• Judge agreement is assessed by Spearman rank correlation over per-item overall scores, testing whether judges order evaluation items consistently.Source: D. Judge LLM · p. 5

E. Cosine Similarity and Blended Scoring

• The blended score combines mean judge overall score and normalized BGE cosine at weights of 0.7 and 0.3, respectively.Low cosine alongside a high judge score may flag thematically divergent answers; high cosine with low judgment may flag verbatim specification repetition without synthesis.Source: E. Cosine Similarity and Blended Scoring · p. 5

V. EXPERIMENTAL DEPLOYMENT AND RESULTS

• The results section covers deployment configuration, attack execution and replay, aggregate and class-level results, and qualitative success and failure cases.Source: §V · p. 5

A. Testbed Architecture and Data Collection

• A purpose-built 5G/O-RAN testbed uses a CU-DU-RU stack and eCPRI open fronthaul; 313 labeled DDoS replay items form the evaluation reference set.PCAPs and gNodeB/RU logs inform references, while the model receives high-level scenario descriptors without raw packet or log content.Source: A. Testbed Architecture and Data… · p. 5

B. Deployment Configuration

• Retrieval runs only for the spec-grounding class: BGE embeds the question, FAISS retrieves candidates, a cross-encoder reranks the top 20, and parent chunks precede metadata in the prompt.The other four supervised classes skip retrieval; judge scoring follows answer generation as a stateless postprocessing step.Source: B. Deployment Configuration · pp. 5–6

C. Attack Scenario Configuration

• Controlled replays inject traffic at the DU or RU side of the eCPRI link, with direction, packet size, rate, source volume, plane, and link direction set as metadata.Source: C. Attack Scenario Configuration · p. 6

D. Aggregate Results

• 0.713 mean judge overall score across 313 items, compared with 0.755 for the five supervised classes; blended scores are 0.778 and 0.808, respectively.The five-class aggregate excludes the intentionally withheld attack-direction reasoning class.Source: D. Aggregate Results · p. 6
• Table I reports per-class evaluation results; Fig. 3 plots mean judge overall score by taxonomy class and annotates each bar with its sample size.Source: Table I · p. 6, Figure 3 · p. 6
• Two reference judges score overall at 0.728 and 0.776, and all three judges rank the six taxonomy classes in the same order.Per-item score orderings have Spearman correlations of 0.773 and 0.801 between the primary judge and each reference judge, and 0.780 between the reference judges.Source: D. Aggregate Results · p. 6
• Mitigation and threat-model pass rates are 0.875 and 0.756, while the held-out attack-direction class passes at 0.200.Spec grounding and detection signals pass at 0.525 and 0.534; scenario-severity expectation passes at 0.341.Source: D. Aggregate Results · p. 6

E. Class-Level Analysis

• Mitigation and Threat Model benefit from normative guidance on mitigations, and both have mean faithfulness above 0.95.The cited guidance includes ingress filtering, MACsec, IPsec, and zero-trust interface policies.Source: E. Class-Level Analysis · pp. 6–7
• Fig. 4 reports mean overall, normalised cosine, and blended scores for all 313 items and the 258-item subset excluding held-out Attack Direction Reasoning.Source: Figure 4 · p. 7
• Mitigation and Threat Model are the only classes above 70% judge pass rate; held-out Attack Direction Reasoning scores 20.0% pass rate and 0.515 mean overall.The attack-direction class was excluded from training by design.Source: Figure 5 · p. 7, E. Class-Level Analysis · p. 7
• Spec Grounding and Detection Signals expose different precision gaps: retrieved requirements are not guaranteed to appear in top-k, while detection answers can omit deviation direction or magnitude.Their correctness sub-scores average 0.63 for spec grounding and 0.58 for detection signals.Source: E. Class-Level Analysis · p. 7
• Scenario Severity Expectation has a 0.604 mean overall, with errors concentrated on recovery-without-crash cases that the model tends to conflate with more severe outcomes.The needed operational attack-threshold experience is absent from specifications and only partly represented in the NetsLab flow data.Source: E. Class-Level Analysis · p. 7
• Fig. 6 places only Mitigation and Threat Model inside the target quadrant for acceptable score quality and consistency.The joint distribution compares mean overall score with pass rate across six classes.Source: E. Class-Level Analysis · p. 7

F. Qualitative Case Studies

• The example uses a 10 MB baseline C-plane downlink PCAP and RU–DU traffic replayed at 10 Mbps from the DU-side interface.Fig. 7 shows judge sub-scores for both items.Source: F. Qualitative Case Studies · p. 7

1) Detection Signals Item:

• The model identified the eCPRI direction mismatch and late-packet counters, earning faithfulness 1.000 and mean overall 0.770, but called the attack man-in-the-middle rather than spoofing.It also omitted the expected increase in late packets followed by recovery; correctness was 0.700 and actionability 0.600.Source: 1) Detection Signals Item · pp. 7–8
• Fig. 6 plots mean overall score against pass rate across taxonomy classes, with bubble area encoding item count and shading marking the target quadrant.Source: Figure 6 · p. 8

2) Mitigation Item:

• 0.810 mean overall on the mitigation item accompanies full faithfulness and 0.800 actionability, but the response recommends an imprecise mutual-TLS mitigation and omits eCPRI data-direction validation.The figure’s weighted Overall incorporates faithfulness, correctness, and actionability; Coverage is diagnostic only.Source: 2) Mitigation Item · p. 8, Figure 7 · p. 8

G. Discussion

• Specification-grounded fine-tuning yields analyst-quality mitigation and threat-model outputs where normative structure is strong, while operational-judgment tasks degrade systematically; cosine similarity complements, but cannot resolve, judge-identified correctness gaps.The detection-signals example has cosine similarity of 0.832.Source: G. Discussion · p. 8
• Judge bias is not fully eliminated, so scores are calibrated relative measures rather than ground truth; evaluation on Fronthaul DDoS replays constrains generalization across interfaces and attack families.The evaluation uses 313 scenarios on one 5G/O-RAN testbed.Source: G. Discussion · pp. 8–9, §VI · p. 9

VI. CONCLUSIONS

• 0.713 mean judge overall and 0.778 blended scores accompany strongest performance on mitigation and threat-model reasoning, while attack-direction reasoning remains a training-data gap.The reported class means are 0.849 for mitigation, 0.848 for threat-model reasoning, and 0.515 for attack-direction reasoning.Source: §VI · p. 9