Source-linked AI summary
AI Slop and Hallucinations in Vulnerability Assessment: A Survey on Reasoning Failures and Trustworthy Mitigation
Junchen Ding, Jialiang Dong, Yichen Zhu, Yi Liu, Gelei Deng, Willy Susilo, Siqi Ma, Yuekang Li
TL;DR
AI-generated security artifacts can appear fluent and plausible while lacking semantic validity or executable grounding, challenging vulnerability-triage reliability. This survey formalizes the phenomenon, synthesizes verification architectures, and specifies evaluation instruments, concluding that trustworthy triage requires checkable, reproducible, formally grounded outputs.
Problem
Current vulnerability-assessment research lacks sufficiently grounded architectures that treat LLM-generated security outputs as hypotheses requiring deterministic validation.
Method
The survey develops an AI-slop taxonomy, operationalizes the reasoning gap with measurable criteria, analyzes verification architectures, and specifies benchmark and scoring instruments.
Results
Across hallucinated vulnerabilities, incorrect patches, and semantic repackaging, probabilistic generation substitutes for causal verification, with no reviewed study providing execution-level validation.
Takeaways & Limitations
Trustworthy AI-assisted vulnerability triage requires pairing generation with deterministic evaluators whose blind spots are explicit and whose outputs are checkable, reproducible, and grounded.
Takeaways & Limitations
Passive statistical detectors cannot determine whether a vulnerability claim is true because template-like human reports can produce the same low-perplexity signals.
Abstract
from arXiv · showhide
The integration of Large Language Models (LLMs) into cybersecurity has transformed vulnerability assessment, but it has also produced a trustworthiness crisis driven by the unchecked proliferation of "AI slop." These artifacts, hallucinated vulnerabilities, plausible but incorrect patches, and semantically repackaged bug reports, impose a cognitive burden on human triage pipelines that mirrors a denial-of-service attack. This paper surveys the empirical evidence, identifies a unifying mechanism, and traces a path toward trustworthy triage. We formalize a taxonomy of AI slop grounded in a structured literature review and dissect its root cause: the gap between the causal deductive reasoning of security experts and the autoregressive probabilistic generation of current LLMs. We operationalize this gap through a measurable proxy, the Deductive Coverage Score, and show that chain-of-thought prompting and tool-using agents narrow but do not close it. We review mitigation strategies and argue that passive detection and watermarking target provenance rather than correctness, facing fundamental entropy constraints. We instead advocate for active neuro-symbolic verification, mapping each pipeline component to prior systems with documented limits on security inputs. Finally, we specify two evaluation instruments, CVE-Bench and Slop-Score, including dataset construction, metric formulas, and anti-gaming provisions. By shifting evaluation from linguistic fluency to mathematical verifiability, this survey provides a roadmap for securing emerging AI-driven triage systems.
1 Introduction
LLM integration is reshaping vulnerability assessment but introduces fluent, plausible, semantically invalid “AI Slop” caused by a reasoning mismatch between human deduction and probabilistic generation. The paper formalizes this problem, surveys its manifestations and mitigations, and advocates active verification for trustworthy triage.
- Problem Definition: AI Slop comprises fluent, structurally plausible security artifacts lacking semantic validity or executable grounding, including hallucinated vulnerabilities, incorrect patches, and semantic repackaging.These artifacts are difficult to invalidate and differ from traditional rule-based false positives.
- Root Cause: The paper attributes AI Slop to a reasoning gap: experts use causal, multi-step deduction and exploitability constraints, whereas LLMs generate outputs probabilistically.This mismatch systematically produces unreliable vulnerability detection and patch synthesis.
- Trustworthiness Gap: Existing accuracy improvements and post hoc detection remain limited without grounding, motivating architectures that treat generated outputs as hypotheses requiring deterministic validation.The paper argues that generated conclusions should not be treated as authoritative without verification.
- Contributions: The paper formalizes AI Slop and its taxonomy, analyzes the human–LLM reasoning divergence, reviews passive detection and watermarking limits, and advocates active neuro-symbolic verification.Its roadmap targets trustworthy, human-aligned triage systems and identifies open challenges for their development.
2 Background
LLMs are increasingly integrated into vulnerability assessment, augmenting expert workflows with semantic analysis, explanations, exploit sketches, and patch suggestions. This shift makes triage more difficult because fluent, plausible reports can conceal semantic failures, while much real remediation remains outside visible disclosure infrastructures.
- LLMs in Vulnerability Assessment: LLMs increasingly support vulnerability assessment by interpreting code semantically and assisting anomaly detection, explanations, exploit sketching, and patch generation.Their higher-level analysis can capture code structure, naming conventions, and developer intent beyond purely syntactic SAST rules.
- Reporting and Triage: LLM-generated reports weaken linguistic clarity as a trust signal because inexperienced or adversarial contributors can imitate established reporting conventions and superficial indicators.This undermines the historical correlation between well-written reports, careful analysis, and reproducible findings.
- Reporting and Triage: The disclosure ecosystem is incomplete: Dong et al. (2025) found that substantial vulnerability remediation occurs through “silent” fixes lacking CVEs, coordinated disclosure, or security advisories.These omissions mean visible reporting infrastructures do not capture all real vulnerability remediation.
- Reporting and Triage: Triage has shifted from spotting obviously flawed reports to verifying whether well-structured submissions describe actual vulnerabilities.Analysts must reconstruct or refute each report’s implicit reasoning, increasing cognitive load.
- AI Slop: AI slop consists of fluent, syntactically plausible security artifacts whose hallucinated technical content causes catastrophic semantic failures.It differs from conventional false positives, which typically arise from deterministic tools applying conservative or overly broad rules.
- Reasoning Differences: Human researchers use causal, multi-step deduction to model systems, trace data flow, and verify exploit conditions, whereas LLMs generate likely token sequences resembling known reports.This reasoning difference underlies the distinction between expert vulnerability analysis and probabilistic text generation.
3 Literature Review: A Taxonomy of AI Slop
A systematic review of empirical security studies organizes AI slop into hallucinated vulnerabilities, incorrect patches, and report spam. Across these branches, probabilistic generation substitutes for causal verification, while existing evaluations rarely provide execution-level validation.
- Taxonomy: The review identifies three AI-slop branches: hallucinated vulnerabilities, incorrect patches, and report spam, each describing a distinct drift from grounded reasoning.The taxonomy was built from screened studies with empirical evidence of LLM failure modes in security contexts, and Table 1 anchors the categories in representative evidence.
- Hallucinated Vulnerabilities: Hallucination includes fabricated CVEs, APIs, and exploit paths; benign code can trigger vulnerability reports, while semantics-preserving renaming can change judgments.Prompt-injection hallucination and agent failures involving incorrect tool arguments or structured-output interpretations are identified as known but under-studied sub-branches.
- Incorrect Patches: Incorrect patches often address symptoms rather than causes, fail to converge through iterative dialogue, or introduce new security weaknesses despite functional success.These regressions can remain invisible to functional benchmarks while creating a new CWE or leaving the underlying vulnerability unresolved.
- Report Spam: Report spam exploits fluency and scale: readers prefer clear but incorrect explanations, while automated submissions can flood triage with unverifiable reports.Silent fixes can also be repackaged as novel findings without CVEs or public contradictions, making lookup-based validation ineffective.
- Cross-Branch Synthesis: Across all branches, probabilistic generation replaces causal verification, and no reviewed study provides execution-level validation of LLM claims.Compile checking and static analysis represent the strongest grounding identified, but both fall short of security reasoning requirements.
4 The Empirical Divide: Human Deductive Reasoning versus AI Probabilistic Generation
Human vulnerability analysts build and test evolving execution models, whereas LLMs generate probabilistic, coherent narratives without consistently preserving long-range dependencies. Chain-of-thought, tool use, and retrieval narrow this epistemic gap but do not replace execution-grounded verification, motivating the Deductive Coverage Score as a proxy for explicit evidence grounding.
- Human Deductive Reasoning: Human analysts trace code, data, dependencies, and constraints while revising hypotheses against observed execution, although real triage also relies on heuristics and is affected by time pressure and cognitive bias.Their reasoning carries forward implicit invariants and tests whether a suspected vulnerability survives the complete execution chain without contradiction.
- LLM Agent Probabilistic Generation: LLMs predict likely continuations rather than construct execution traces, and longer contexts can weaken attention to critical information through the “lost in the middle” effect.The resulting output may reconstruct a coherent narrative from context fragments without consistently preserving cross-procedural relationships.
- LLM Agent Probabilistic Generation: Larger models can produce more coherent and persuasive narratives while remaining anchored to the same probabilistic foundation.Thus, increased fluency does not resolve the underlying reasoning tension.
- Gap-Narrowing Modifications: CoT helps mainly on simple single-function vulnerabilities, while its benefit diminishes sharply on cross-procedural cases requiring at least three hops and can yield longer but still incorrect reasoning chains.Tool-using agents and RAG provide additional structure, but agents may formulate incorrect tool queries, while retrieval can reduce nonexistent-CVE fabrication without confirming that analyzed code instantiates the retrieved vulnerability.
- Deductive Coverage Score: The Deductive Coverage Score measures whether a vulnerability claim is grounded in explicit, verifiable evidence by decomposing it into atomic constraints that must each hold.Acceptable anchors include specific code references, execution traces, concrete input specifications, or formal proof fragments; vague data-flow assertions do not count.
5 Mitigating AI Slop: Content Provenance and Verification Strategies
The section argues that trustworthy mitigation must target correctness rather than provenance, because detectors, watermarks, and attestations cannot establish whether a security artifact is true. It presents verification-first neuro-symbolic triage, in which LLM hypotheses undergo deterministic checks before human review, while acknowledging each component’s blind spots.
- Historically misaligned approaches: Statistical detectors confuse formulaic human security reports with machine text, while their ROC AUC converges to chance as model and human distributions converge [21, 24].Vulnerability reports use rigid templates, stack traces, and standardized formatting, producing false positives and remaining vulnerable to adversarially engineered burstiness.
- Historically misaligned approaches: Watermarking can corrupt constrained code and patch generation by inducing incorrect API calls, invalid variable names, or broken payloads.In deterministic logic chains, the available vocabulary is too constrained for provenance marking without risking functional errors.
- Ecosystem-level attestation: Cryptographic attestation proves submitter identity or execution provenance but cannot establish that a reported exploit path is true.The section treats ecosystem-level attestation as a pragmatic infrastructure defense that still targets provenance rather than correctness.
- Verification-first paradigm: Verification-first triage shifts the target from provenance to correctness by requiring generated hypotheses to survive deterministic checks before reaching human analysts.The architecture raises the floor by requiring at least two independent verification stages, with most slop failing at the first hurdle.
- Verification-first paradigm: RAG, CodeQL, fuzzing, symbolic execution, and Reflexion each constrain hallucinations but retain important limits, including retrieval noise, false negatives, path constraints, path explosion, timeouts, and absent convergence guarantees [16, 1, 18, 20, 17, 3, 26, 25].RAG confirms vulnerability classes rather than specific code instances; CodeQL cannot reliably refute complex claims; fuzzing depends on the generated harness; symbolic execution struggles with exponential path growth and difficult constraints; Reflexion depends on feedback granularity and lacks guaranteed convergence.
6 Open Challenges and Future Directions
This section proposes CVE-Bench and Slop-Score to evaluate verifiable security reasoning beyond linguistic fluency, while warning that adversaries can exploit component-level verification blind spots. It argues for verification-centered systems that expose independently checkable reasoning artifacts rather than relying on confident explanations.
- Evaluation gap: Existing benchmarks reward apparent correctness while overlooking whether security reasoning is sound or whether benign triage prompts trigger invented vulnerabilities [2].HumanEval and MBPP emphasize binary functional outcomes, SWE-bench omits patch-reasoning soundness, and CyberSecEval focuses on malicious-intent compliance.
- CVE-Bench: CVE-Bench tests whether triage systems distinguish verified vulnerability reports from fluent phantom exploit paths using classification, localization, and ranking tasks.Its design uses verified recent CVEs with reproducible proofs-of-concept, fluent LLM-generated phantom reports, temporal holdouts, identifier-free inputs, and paraphrased positives.
- Slop-Score: Slop-Score quantifies the gap between linguistic polish and verifiable substance using fluency, evidence density, and constraint satisfaction.Validation compares expert-written and LLM-generated phantom reports with AUC-ROC and component ablations.
- Adversarial robustness: Adversaries can engineer phantom reports to exploit parser edge cases and blind spots in CodeQL heap coverage, symbolic execution, and fuzzing magic-byte comparisons.Such reports may present plausible but unverifiable use-after-free assumptions that no individual verification component can conclusively refute.
- Verification-centered systems: Because generative models reinforce confident errors rather than detecting contradictions, systems should prioritize independently validated intermediate representations, execution traces, or formal exploitability proofs.Natural language should serve as a secondary explanation layer, with the model judged by verifiability rather than persuasive fluency.
7 Conclusion
LLM-based vulnerability assessment creates a widening trust gap because fluent outputs can contain hallucinated vulnerabilities, incorrect patches, and recycled reports. Trustworthy systems must pair generation with deterministic verification while exposing evaluator blind spots.
- LLM deployment is reshaping vulnerability assessment, but hallucinated vulnerabilities, incorrect patches, and recycled bug reports expose a widening trust gap.These failures reflect a mismatch between expert reasoning and model generation.
- Human analysts use constraint, verification, and deduction, whereas language models predict likely patterns; prompting, tools, and retrieval help simpler cases but do not solve security’s multi-step reasoning.Statistical filtering and watermarking address provenance rather than correctness.
- The path forward treats generation as the start of a verification pipeline, pairing generative models with deterministic evaluators while specifying each evaluator’s blind spots.The survey structures this agenda through a taxonomy, a measurable reasoning gap, component-level verification analysis, and concrete benchmark specifications.