Source-linked AI summary
WeSCE: A Benchmark for Measuring Security Drift in LLM-Driven Code Editing
Zhiyu Zhang, Tingyue Wen, Senke Sun, Dengxiang Liang, Enhao Huang
TL;DR
LLM-driven code edits increasingly affect software security, yet existing approaches do not model security changes when edits pursue only functional goals. WeSCE addresses this gap with a benchmark and continuous, multi-scale drift measures, finding that stronger models reduce risk in over 80% of samples but achieve complete clearance in only about half.
Problem
Existing approaches do not explicitly model how security changes under routine code transformations driven by functional rather than security objectives.
Method
WeSCE benchmarks weak-security code editing using continuous risk scores and drift measures spanning average risk, worst-case severity, and vulnerability redistribution.
Results
Stronger models reduce risk in over 80% of samples, while worst-case mitigation exceeds average-case reduction and complete clearance remains limited to roughly half.
Takeaways & Limitations
WeSCE enables transformation-aware evaluation showing that current LLM edits mainly partially mitigate vulnerabilities rather than fully resolve them under weak-security constraints.
Takeaways & Limitations
The benchmark uses programs of approximately 200 LOC, which may limit generalization to larger, highly coupled software systems.
Abstract
from arXiv · showhide
In this work, we introduce WeSCE, a benchmark for quantifying security drift in code editing under weak-security constraints, where tasks specify only functional objectives without explicit security requirements. WeSCE consists of 400 executable programs derived from real-world code, covering feature addition, feature removal, bug fixing, and refactoring. To quantify security drift, we propose a continuous risk representation that aggregates heterogeneous vulnerability signals through a unified formulation, and define drift measures capturing changes in overall risk, worst-case severity, and vulnerability distribution under code transformations, providing a multi-scale view of security spanning average-case behavior to worst-case emphasis.
1 Introduction
WeSCE addresses the previously uncharacterized problem of security drift caused by functionally driven code edits without explicit security requirements. It models security continuously and evaluates changes in risk magnitude, worst-case severity, and vulnerability distribution across realistic editing tasks.
- Motivation: LLM-driven code editing is increasingly used for feature development, bug fixing, and refactoring, making its effects on code security a pressing concern.Prior studies show that LLM-generated code can introduce unsafe patterns.
- Limitations of Existing Benchmarks: Existing benchmarks emphasize functional correctness, execution quality, or security objectives, typically evaluating static snapshots rather than implicit security changes during ordinary edits.These evaluations do not characterize security drift under functional-only instructions.
- Continuous Security Modeling: Binary vulnerability labels obscure how editing redistributes risk when one vulnerability is removed and another is introduced, as well as differences in relative severity.WeSCE instead treats security as a continuous program attribute.
- WeSCE Framework: WeSCE aggregates heterogeneous vulnerability signals with LogSumExp and defines drift measures for risk magnitude, worst-case severity, and vulnerability distribution.The LogSumExp representation interpolates between average-case and worst-case sensitivity; the framework compares pre- and post-edit representations.
- Benchmark Construction: The benchmark contains 400 executable programs derived from real-world code and covers feature addition, feature removal, bug fixing, and refactoring.Programs were sourced from GitHub and Real-Vuln-Benchmark and validated for executability.
- Evaluation Results: Across eight evaluated LLMs, stronger models achieve risk reduction in over 80% of samples, whereas weaker models fall below 60%.Worst-case mitigation exceeds average-case reduction across models, while the best model achieves complete clearance in only about half of samples.
2 Related Work
Prior benchmarks evaluate code correctness, efficiency, contamination, vulnerability detection, or security patches, but generally treat security as a discrete outcome or assume explicit security objectives. WeSCE addresses this gap by continuously modeling security drift under non-security-driven code transformations through risk magnitude, worst-case severity, and vulnerability-distribution measures.
- Benchmarking Code Generation: Code-generation benchmarks span unit-test correctness, expanded test coverage, contamination, and performance profiling.HumanEval and MBPP focus on unit-test correctness; EvalPlus improves coverage, dynamic evaluation addresses contamination, and other benchmarks incorporate profiling.
- Security Evaluation: Security benchmarks verify vulnerabilities but treat security as discrete and do not characterize vulnerability-signal changes under transformations.SecCodeBench uses CWE-level verification, while MT-Sec shows security and correctness degradation over multi-turn interactions.
- Realistic Coding Scenarios: Program-repair and security-patch approaches use execution validation or fix known vulnerabilities, typically assuming explicit security or bug-fix objectives.PatchEval provides sandboxed fix-testing environments, and PatchLM generates patches from CVE-associated commit hunks.
- Unified Security Metrics: Unified functionality-security metrics still produce coarse pass/fail outcomes and do not model security evolution during code transformations.CWEval combines functional correctness with vulnerability checks, while RealSec-bench introduces the composite SecurePass metric.
- Our Approach: WeSCE models security continuously and tracks risk magnitude, worst-case severity, and vulnerability-distribution changes after each edit.Its aggregation framework can incorporate additional vulnerability signals as new vulnerability types emerge without modification.
3 Framework and Method of WeSCE
WeSCE constructs executable, function-only code-editing tasks and quantifies security drift by combining static and dynamic vulnerability signals into continuous risk measures. Its framework tracks changes in risk magnitude, worst-case severity, vulnerability distribution, and complete risk elimination.
- Dataset construction: WeSCE collects 477 vulnerability seeds from GitHub and Real-Vuln-Benchmark, covering static issues and dynamic vulnerabilities.The collection includes 334 GitHub samples and 143 Real-Vuln-Benchmark samples, with near-duplicates removed using CodeBERT similarity and manual inspection.
- Dataset construction: The final dataset contains 400 executable samples evenly distributed across ADD, REMOVE, FIX, and REFACTOR tasks with function-only instructions.Programs are filtered for syntax, executability, and length, then validated through static analysis, dynamic execution, and manual inspection of 40 samples.
- Risk representation: L normalization produces the most stable vulnerability-density estimates, with CV = 0.41 versus 0.82 without normalization, 0.74 with linear scaling, and 0.63 with logarithmic scaling.The normalization reduces program-size bias across programs with varying code surface.
- Risk representation: LogSumExp aggregates heterogeneous vulnerability signals while interpolating between average-case and worst-case risk according to sensitivity parameter b.Small b approximates the arithmetic mean, whereas large b approaches the maximum signal and emphasizes the most severe vulnerability.
- Security drift measures: Security drift is measured through ∆E0, ∆E∞, DTV, and Rc, capturing average-risk change, worst-case change, vulnerability redistribution, and complete clearance.Negative ∆E indicates risk reduction, while positive ∆E indicates risk amplification; batch metrics additionally report risk-reduction rates R0 and R∞.
4 Experiments
Experiments evaluate eight LLMs on WeSCE using unified analysis and aggregation settings, finding that security drift varies substantially by model and editing task. Stronger models and broader transformations generally achieve greater risk reduction and vulnerability-profile shifts, but complete clearance remains limited.
- Experimental setup: Eight LLMs spanning capability tiers are evaluated under unified top-p = 0.9 and top-k = 20 decoding settings.Security analysis combines CodeQL, Bandit, and Atheris with a fixed 90-second fuzzing budget per program, cross-tool checks, and manual validation.
- Model stratification: Opus 4.6, GPT-5.4, and Sonnet 4 form the top tier, while Haiku 4.5, Doubao-1.5-pro, and GLM-4-32B fall below 66% in R∞.Opus 4.6 achieves R∞=82.25% and Rc=51.25%, whereas GLM-4-32B reaches Rc as low as 31.50%.
- Model stratification: 23.50 percentage points in R∞ and 19.75 points in Rc separate Opus 4.6 from GLM-4-32B.The result indicates substantial capability-related differences in incidental security improvement.
- Limitations and structural effects: Only 51.25% of Opus 4.6’s edited programs achieve complete clearance, so nearly half retain non-trivial vulnerabilities.Edits primarily provide partial mitigation rather than full resolution under weak-security constraints; stronger models also induce larger distributional shifts, with Opus 4.6 reaching µTV = 0.629 versus 0.455 for GLM-4-32B.
- Task-level stratification: REFACTOR and FIX consistently outperform ADD and REMOVE across metrics; for Opus 4.6, REFACTOR reaches R∞=98% and Rc=80% versus R∞=71% and Rc=34% for ADD.Broader structural reorganization and bug correction alter vulnerability-related code paths more than localized functionality changes.
5 Limitations and Future Work
WeSCE is limited by its small program scale, execution-focused dynamic analysis, and weak-security setting. Future work should expand program scope, strengthen security control, and incorporate more comprehensive analysis tools.
- Limitations: WeSCE evaluates programs of approximately 200 LOC, which may limit generalization to larger, highly coupled software systems.The small scale enables controlled execution and systematic tracking of security signals under code transformations.
- Limitations: Atheris captures input-driven crashes and certain runtime vulnerabilities but may miss logic flaws and environment-dependent exploits.Consequently, the dynamic component E(d) mainly reflects execution-level failure modes.
- Limitations: Weak-security tasks leave models’ reliance on pretrained security knowledge difficult to separate from task-driven improvements.Future work may explore stronger adversarial prompting and finer-grained control of security signals.
- Future Work: Future work may extend WeSCE to larger, multi-module codebases and incorporate more comprehensive analysis tools.
6 Conclusion
WeSCE benchmarks security drift in LLM-driven code transformations under weak-security settings by continuously representing program security and tracking changes in static and dynamic vulnerability signals. Experiments on eight LLMs reveal consistent stratification in security improvement capability and a bias toward worst-case mitigation.
- 6 Conclusion: WeSCE studies security drift during LLM-driven code transformations in weak-security settings.The benchmark represents program security as a continuous attribute and measures drift over static and dynamic vulnerability signals.
- 6 Conclusion: Continuous security representation and drift measures enable fine-grained tracking of risk evolution during functional code editing.
- 6 Conclusion: Eight LLM experiments reveal consistent model stratification in security improvement capability and a bias toward worst-case mitigation.