Source-linked AI summary
Large Language Models at the Intersection of Software Engineering and Software Security:An Evidence-Centered Structured Survey and Research Agenda
Wei Lin, Tao Zhou, Zhaofei Xie, Changgui Hong
TL;DR
LLM software engineering and security evidence remains divided across functional, vulnerability, and exploit-oriented evaluations, leaving unclear what supports trustworthy repository-scale agents. This survey synthesizes the evidence through a cross-domain taxonomy and assurance framework, finding that credible capability claims require task-appropriate, reproducible, threat-model-aligned, and governance-aware evidence rather than a single benchmark score.
Problem
Existing reviews leave open what evidence is sufficient when development artifacts must be functional, secure, reproducible, and produced by agents with non-trivial authority.
Method
The survey uses a structured cross-domain synthesis organized by tasks, artifacts, adaptation mechanisms, evaluation configurations, and agent authority.
Results
The synthesis finds that evidence validity depends on the deployment claim: function tests, vulnerability labels, and repository or operational claims require progressively different evidence.
Takeaways & Limitations
Trustworthy LLM-based software systems should be judged through reproducible, threat-model-aligned, governance-aware assurance cases supported by task-appropriate evidence.
Takeaways & Limitations
Selection, construct, conclusion, and external validity remain constrained by the narrative review process, inconsistent terminology, heterogeneous configurations, and dominance of visible English-language work.
Abstract
from arXiv · showhide
Large Language Models (LLMs) are moving from code completion toward repository-scale agents that retrieve context, edit files, execute tools, and participate in security-sensitive workflows. The evidence for these systems, however, remains divided between software engineering evaluations centered on functional task completion and software security evaluations centered on vulnerability detection, secure generation, or exploit-oriented validation. This evidence-centered structured survey synthesizes representative work available through May 31, 2026 across software engineering tasks, software security tasks, adaptation mechanisms, artifact granularity, and evaluation design. In addition to a task taxonomy, we introduce an assurance framework that separates functional correctness, security, operational reliability, evidence provenance, and agent authority. The review shows that execution feedback and repository access can substantially improve engineering task completion, but do not by themselves establish security; conversely, static-analysis labels or vulnerability-classification scores rarely establish deployable correctness. We identify recurring validity threats--weak test oracles, duplicated and temporally leaked data, changing agent harnesses, proxy-only security checks, and under-reported budgets and human intervention--and derive a minimum reporting protocol for cross-study comparison. The resulting research agenda prioritizes jointly secure-and-functional benchmarks, repository-scale threat models, calibrated human oversight, longitudinal maintainability evidence, and reproducible agent evaluation. The central conclusion is that model capability should be judged as an assurance case supported by task-appropriate evidence, rather than by a single benchmark score.
1 Introduction
This survey bridges software engineering and software security by comparing their tasks, artifacts, adaptation mechanisms, and evaluation evidence. It frames trustworthy agentic development as requiring functional, security, operational, reproducibility, and governance evidence rather than a single task-success score.
- Motivation: Existing reviews cover LLM-based software engineering, agentic systems, code review, cybersecurity, vulnerability detection, and repair, but leave cross-domain evidence requirements unresolved.The open question concerns artifacts that must be functional, secure, reproducible, and produced by agents with non-trivial authority.
- Scope: The survey treats software engineering and software security as co-equal pillars and connects them through generation reliability, security analysis, and joint evaluation criteria.It includes agentic systems spanning repository-oriented development and autonomous security workflows.
- Contributions: Its taxonomy relates engineering and security tasks to artifact granularity, context requirements, adaptation mechanisms, and agent authority.The framework organizes tasks across code generation, repair, testing, review, documentation, vulnerability analysis, malware analysis, and security testing.
- Contributions: The assurance framework separates functional, security, operational, reproducibility, and governance evidence instead of reducing task success to one scalar score.This distinction supports evaluation of systems whose outputs and actions have different assurance requirements.
- Contributions: The survey adds a claim–evidence ladder, a minimum reporting protocol, and a research agenda linking secure generation, repository-scale analysis, human oversight, maintainability, and standardized evaluation.These contributions target comparison across model-centric and agentic studies.
2 Review Methodology
The review uses a traceable, evidence-centered narrative synthesis to compare representative LLM software engineering and security research. It preserves study-specific context and configuration differences rather than pooling heterogeneous results into a meta-analysis.
- Scope: The paper is an evidence-centered structured narrative and tertiary survey, with a search cutoff of May 31, 2026.It prioritizes foundational models and benchmarks, systematic reviews, empirical evaluations, and relevant methodological work.
- Research questions: The review asks which tasks and artifacts are addressed, how adaptation changes capabilities and failures, which evaluation evidence is credible, and how engineering and security requirements intersect.These questions span task context, prompting, fine-tuning, retrieval, agentic tools, benchmarks, metrics, and governance.
- Search and selection: Sources were identified through backward chaining and targeted searches across arXiv, ACM, and IEEE indexes using four concept blocks.Retention required consequential models, datasets, benchmarks, methods, relevant empirical evidence, bounded synthesis, or exposed validity threats.
- Data extraction: The seed corpus was automatically generated from citation keys, deduplicated by DOI, arXiv identifier, and normalized title, and coded across task, artifact, model, adaptation, benchmark, metric, result, and limitation fields.A separate rubric scores question clarity, provenance, split validity, configuration reporting, baselines, metrics, uncertainty, and leakage analysis.
- Synthesis: Findings are compared at the claim level rather than pooled statistically because studies differ in models, data, prompts, tools, and evaluation protocols.The package includes a prospective protocol, machine-readable corpus, search and extraction templates, quality rubric, and scripts for corpus maintenance.
3 Background and Preliminaries
LLM-based code systems span three architectural paradigms and four adaptation strategies, supporting a unified view of software engineering and security applications. The taxonomy organizes these approaches across engineering tasks, security tasks, and adaptation families.
- Architectural paradigms: Encoder-only models support code understanding tasks, while encoder-decoder models support generation-oriented tasks such as summarization, translation, and repair.CodeBERT exemplifies encoder-only architectures, whereas CodeT5 combines bidirectional encoding with autoregressive decoding.
- Architectural paradigms: Decoder-only models enable open-ended code generation and tool-mediated interaction.Codex is identified as an early demonstration of this paradigm.
- Adaptation strategies: Zero-shot or few-shot prompting guides generation without parameter updates, whereas fine-tuning adapts models through full or parameter-efficient updates.LoRA and QLoRA are highlighted as parameter-efficient methods for domain-specific and security-sensitive tasks.
- Unified taxonomy: The taxonomy spans engineering tasks, security tasks, and adaptation strategies as intersecting dimensions.Engineering tasks include generation, repair, testing, review, and documentation; security tasks include vulnerability detection, repair, malware analysis, and security testing or fuzzing.
- Historical context: The review situates code-model developments from CodeBERT in 2020 through later systems along a timeline whose colors denote architectural paradigms.Figure 2 provides the historical timeline, while Figure 1 presents the hierarchical taxonomy.
4 An Assurance-Centered Framework
The assurance-centered framework treats an LLM workflow as a compound evaluation configuration rather than a model-only score. It requires evidence that matches the task, threat model, environment, budgets, human intervention, and agent authority.
- Evaluation configuration: An evaluation configuration combines the artifact and task, context, model and adaptation, tools and permissions, validators, budget, and human intervention.The framework represents these components as E = ⟨A, C, M, T, V, B, H⟩.
- Evaluation configuration: Scores are directly comparable only when differences in workflow components are controlled or explicitly analyzed.Changing the harness or environment can change benchmark leaderboards, making model-only labels insufficient for agent results.
- Joint functional and security success: Joint success requires generated candidates to satisfy both functional tests and threat-model-aligned security checks.Functional and security scores should still be reported separately because identical joint scores can conceal different failure profiles.
- Security validation: Security claims require validators matched to the threat model, such as exploit reproduction, dynamic proof-of-vulnerability tests, expert review, or property checking.Absence of analyzer warnings alone does not establish security.
- Evidence ladder: Evidence levels differ in realism and comparative precision, so mature assurance cases should combine complementary evidence rather than substitute one score for another.Controlled benchmarks can support precise comparisons, while operational deployments provide stronger external-validity evidence.
- Authority and governance: Agent authority ranges from read-only retrieval to actions affecting repositories, dependencies, credentials, or deployment, with governance requirements increasing across tiers.Higher-authority tiers require controls including least-privilege scopes, isolation, logging, approval gates, rollback, and incident handling.
5 LLMs for Software Engineering
LLMs support software-engineering tasks from function generation and repair to testing, review, documentation, and repository-scale agents. Evidence indicates that broader context and execution feedback improve task workflows, but robust correctness and maintainability require stronger oracles and lifecycle evidence.
- Code Generation and Synthesis: Function-generation benchmarks can overstate correctness because solutions passing limited test suites may fail additional edge-case tests.HumanEval+ exposes this gap by augmenting original tasks with many additional tests.
- Automated Program Repair: Automated repair combines fault localization, context construction, patch generation, and validation, while generative flexibility expands the space of plausible but incorrect patches.Compilation, tests, and additional analyses provide validation, but test quality and retry budgets affect observed gains.
- Testing: Execution feedback can repair generated test scaffolding, but tests may reproduce implementation behavior rather than validate intended specifications.Project-level evaluations expose compilation and cascading-context failures beyond function-level tasks.
- Review, Documentation, and Lifecycle Tasks: Code-review, documentation, requirements, and architecture applications need task-specific validation because fluent outputs do not establish factual consistency or lifecycle value.Maintainability evidence should include later defects, rework, churn, dependency risk, architectural conformance, and reversions.
- Code Generation and Synthesis: Repository-level agents resolve issues through coordinated interpretation, retrieval, planning, editing, execution, and revision across entire codebases.Their results depend on retrieval quality, harness behavior, execution environment, and retry limits.
6 LLMs for Software Security
LLMs for software security span vulnerability detection, repair, behavioral analysis, fuzzing, and exploit-oriented agents. The section emphasizes that security claims require threat-model-matched validation and that autonomous security workflows remain constrained by reliability and governance requirements.
- Vulnerability Detection: Vulnerability detection requires semantic reasoning about flaws that may span multiple functions or files, unlike general bug detection.Approaches range from representation learning and structure-aware models to generative classifications and explanations.
- Vulnerability Detection: Detection results across security datasets are not interchangeable because resources differ in labeling quality, class balance, deduplication, and temporal splitting.Strong benchmark scores alone do not establish production readiness.
- Vulnerability Repair: Security repair must remove the targeted vulnerability without breaking intended behavior or creating a new attack surface.SEC-bench and repository-level secure-generation benchmarks combine project context with functional and dynamic security validation.
- Security Testing: Hybrid security systems pair precise static or dynamic analysis with LLM semantic reasoning, including for vulnerability reachability analysis.Coverage claims for LLM-guided fuzzing must separate model contributions from underlying execution infrastructure.
- Security Testing and Exploit Generation: Fully autonomous security-agent deployment has limited evidence, making human approval, audit logs, scoped permissions, and reproducible execution important evaluation requirements.False positives increase analyst workload, while false negatives can miss critical threats.
7 Benchmarks and Evaluation
Benchmark evidence spans function-level coding, repair, security detection, and executable repository tasks, but scores support only the claims their artifacts and validators can justify. Cross-study comparison therefore requires configuration reporting, joint security-and-functionality outcomes, and explicit validity controls.
- Code Generation Benchmarks: LLM4SE benchmarks cover function generation, realistic data-science problems, complex instructions, library calls, and continuously refreshed contest tasks.HumanEval and MBPP commonly use pass@k, while HumanEval+ adds tests designed to expose weak-suite overfitting.
- Benchmark Scope and Validity: Repair and security datasets differ in language, realism, fault complexity, labeling, balance, split strategy, and test adequacy, limiting direct score comparison.PrimeVul illustrates the importance of deduplication, chronological splitting, and realistic class imbalance.
- Security Benchmarks: Newer benchmarks jointly evaluate functionality and security through repository context, developer tests, dynamic proof-of-vulnerability inputs, or executable patching tasks.These resources narrow construct-validity gaps but assess different languages, artifacts, authority levels, and threat models.
- Metric Limitations and Evaluation Gaps: Pass@k, classification metrics, static analyzers, and repeated sampling each provide incomplete evidence about specification compliance, exploitability, triage cost, or agent reliability.Best-of-k gains may increase cost, latency, and insecure outputs, while analyzer labels can disagree with human-validated security labels.
- Minimum Reporting Protocol: The minimum reporting protocol identifies whether gains arise from the model, context, harness, tools, budget, validator, or human assistance.It applies to both model-centric and agentic studies and permits fields to be marked not applicable without silent omission.
8 Challenges and Open Problems
Open problems center on reliability, dataset and environment validity, cost, adversarial robustness, governance, industrial realism, and the survey’s own evidence boundaries. The proposed agenda favors falsifiable studies that preserve failures and jointly measure engineering value and security risk.
- Reliability and Validation: Repository-scale hallucinations include nonexistent symbols, stale APIs, incompatible dependencies, misunderstood build assumptions, and patches that violate implicit invariants.Execution feedback reduces some errors but can encourage validator overfitting to observed tests or analyzer warnings.
- Data and Benchmark Validity: Dataset artifacts such as duplicates, unrelated fixing edits, unrealistic negatives, and noisy labels can dominate apparent model progress.Deduplication, chronological splitting, and realistic imbalance are required for more credible comparisons.
- Reproducibility: Exact replication is difficult because models, APIs, repositories, registries, tools, network access, timeouts, concurrency, and retry policies evolve.Results should include both a logical task specification and an executable environment whenever licensing permits.
- Efficiency and Human Effort: Effectiveness must include cost, latency, tool use, failed candidates, and human review rather than task success alone.A Pareto frontier over quality, security, cost, and latency is more informative than a single ranking.
- Adversarial Robustness and Governance: Security evaluations must test both robustness to natural code evolution and robustness to deliberate manipulation of code, retrieval content, and tool outputs.Repository agents additionally face prompt injections in untrusted files and issue text.
- Industrial and Lifecycle Evidence: Industrial validity remains limited by small public benchmarks that omit private dependencies, legacy systems, organizational conventions, regulated data, and downstream maintenance outcomes.Longitudinal studies should measure rejected and reverted outputs beyond merge time.
- Threats to Validity: The survey’s conclusions are bounded by non-exhaustive selection, heterogeneous configurations, limited external validity, and rapidly changing models, APIs, scaffolds, and leaderboards.The authors recommend attaching the cutoff date and benchmark versions when reusing conclusions.
- Research Agenda: The research agenda favors studies that isolate mechanisms, preserve failure evidence, and jointly evaluate engineering value and security risk.The agenda translates broad directions into falsifiable questions and minimum evidence requirements.
9 Conclusion
The survey concludes that credible assessment of LLM-based software systems requires evidence aligned with the deployment claim and jointly considers engineering, security, operational, and governance dimensions. Repository access and iterative tools can improve completion, but trustworthy progress depends on reproducible, threat-model-aligned assurance cases rather than isolated scores.
- Tasks span function-level generation through repository issue resolution, security patching, fuzzing, and tool-mediated analysis, with artifact and context often more decisive than task labels.
- Prompting and fine-tuning remain important for bounded tasks, while retrieval, execution feedback, and agents become more consequential as repository context and iteration increase.These mechanisms also change failure modes, so their effects require controlled ablations rather than attribution to the base model alone.
- Credible evidence must match the deployment claim: unit tests support test-relative functional claims, vulnerability labels support classification claims, and operational claims require broader validation.The proposed evidence ladder and minimum reporting protocol make versioned environments, realistic context, regression checks, security oracles, budgets, and human-effort evidence explicit.
- Security is a constraint on generation, repair, testing, review, and agent authority, so joint outcomes such as J@k should accompany separate functional and security results.
- Repository access and iterative tool use can improve task completion but introduce harness dependence, larger attack surfaces, higher cost, and reproducibility challenges.Static analyzers, tests, and LLM judges contribute to assurance cases but are not individually sufficient ground truth; maintainability and oversight costs remain under-evidenced.
- The research priority is cumulative evidence infrastructure, including versioned benchmarks, executable environments, contamination-aware splits, joint validation, transparent permissions, complete logs, and longitudinal outcomes.Trustworthiness therefore depends on a reproducible, threat-model-aligned, and governance-aware assurance case rather than a plausible patch or high scalar score.