ParanoiaEval: Benchmarking Unnecessary Defensive Work in Agentic Coding

1 INTRODUCTION

• The authors argue that reliable coding agents should complete tasks correctly while keeping defensive behavior commensurate with actual risk.Expanding validation, adding protective logic, or creating backups after evidence establishes the appropriate treatment can add costs, unrelated modifications, or interfere with task completion.Source: §1 · p. 1
• Mainstream evaluations emphasize task completion or passing tests, while newer risk-treatment studies remain fragmented across terminology, contexts, and judgment criteria.The introduction contrasts this fragmentation with software-engineering risk management’s shared framework for comparing and analyzing treatment decisions.Source: §1 · pp. 1–2
• ParanoiaEval operationalizes four ATMA treatments in 200 evidence-controlled repository-level task pairs, with dedicated metrics and a human-calibrated agentic judge.Each pair changes one treatment-defining piece of evidence while keeping other task conditions unchanged.Source: §1 · pp. 2–3, Figure 1 · p. 2
• 11.2%–58.7% of runs still contain unnecessary treatment despite explicit evidence; violations also decouple from task success, lower satisfaction, and follow systematic patterns.The authors conclude that risk treatment is an independent capability dimension and that human software-engineering knowledge may guide its diagnosis and improvement.Source: §1 · p. 2
• The benchmark unifies previously fragmented behaviors under four treatments, supporting comparison across shared concepts.The authors present risk treatment as an independent capability dimension and suggest that established human software-engineering knowledge can guide its diagnosis and improvement.Source: §1 · p. 2

Defensive Behaviors in Coding Agents

• Coding-agent studies separately examine overeager actions, action bias, over-editing, redundant steps, and process discipline, despite a shared pattern of actions continuing after evidence makes them unwarranted.Developers report disliking excessive edits and unrequested changes, and overengineering is cited as a leading reason to revert agent-authored code.Source: §2 · p. 3

Risk Treatment in Software Engineering

• Software-engineering risk management organizes responses around contextual choices involving cost, responsibility, and project constraints, but has not been systematically operationalized for coding-agent evaluation.NIST’s current framework retains acceptance, transfer, mitigation, and avoidance as the fundamental responses to negative risk.Source: §2 · p. 3

3.1 RISK MANAGEMENT ALIGNMENT WITH CODING AGENTS

• The alignment models a task as codebase state plus user instruction, with the agent producing a trajectory of commands, edits, and messages.The agent makes treatment decisions, while the user remains the principal; ATMA comprises avoidance, transfer, mitigation, and acceptance.Source: §3.1 · p. 4
• Evidence establishes a risk’s treatment, and an action outside the task’s minimal sufficient action set is unnecessary when it treats that risk beyond the established treatment.The common decision rule lets behaviors previously studied in isolation be assessed for consistency with the evidence-established treatment.Source: §3.1 · p. 4

3.2 RISK TREATMENTS IN CODING-AGENT SETTINGS

• Avoidance means eliminating a risk’s source or forgoing the activity; transfer means relying on an existing owner or mechanism rather than duplicating its treatment.Examples include relying on continuous integration, a dependency bot, a release script, or a maintainer instead of repeating checks.Source: §3.2 · p. 4
• For mitigation, instruction evidence bounds treatment magnitude; acceptance records the user’s decision to bear the risk without further treatment.Magnitude bounds can set repetition counts, testing scope, report length, or process level; reopening an accepted decision or creating backups is unnecessary.Source: §3.2 · p. 5

4 PARANOIAEVAL

• ParanoiaEval derives situations from real agent sessions and instantiates them as paired repository tasks that differ only in treatment-defining evidence.Source: §4 · p. 5, §4.1 · p. 5

4.1 DATA PROCESSING

• 18,922 deidentified sessions yielded 5,574 reviewed segments and 44 recurring situations across four risk treatments.Source: §4.1 · p. 5

4.2 EVIDENCE-CONTROLLED TASK PAIRS

• Each situation is adapted to a repository task, then paired variants are constructed to differ only in evidence; all 200 pairs pass executability and expert evidence-validity checks.Source: §4.2 · p. 6

4.3 EVALUATION SYSTEM

• A read-only agentic judge checks run artifacts against the answer card and whether extra treatment would be reasonable without the defining evidence.Source: §4.3 · pp. 6–7
• The benchmark reports TSR for oracle-passing runs, VR for excess treatment on E+, and ER for its relative reduction from E− to E+.Source: §4.3 · p. 7

• 800 oracle-passing E+ runs were stratified for the human study, prioritizing paired-task runs with and without excess treatment.Source: §5 · pp. 7–8
• Each developer rated 80 runs, with two ratings per run; ratings were blinded and randomized, and satisfaction covered maintainability, runtime, and required corrections.Source: §5 · p. 8
• Eight models ran in their native Claude Code or Codex harnesses, three times per configuration, yielding 9,600 runs; the primary judge used GPT-5.6 Sol within Codex.Source: §5 · p. 8

6 EMPIRICAL ANALYSIS

• 11.2%–58.7% of E+ runs still show unnecessary treatment, despite significant reductions in every configuration.Across configurations, excess-treatment rates on E+ are lower than on E−, but response strength varies widely.Source: §6 · p. 8, Figure 4 · p. 9
• Higher task success does not correspond to lower violation rates; 92.7% of E+ runs with excess treatment still pass the oracle.The task-success and violation-rate correlation is not significant (Spearman ρ = 0.55, p = 0.17).Source: Figure 4 · p. 9, §6 · p. 9
• Developers rate runs with excess treatment 1.27 points lower in satisfaction and accept them as-is in 38% of cases, versus 78% otherwise.The satisfaction penalty appears under every treatment, ranging from 1.10 points for avoidance to 1.55 for mitigation.Source: Figure 4 · p. 9, §6 · p. 9, Table 4 · p. 10
• Mitigation and acceptance have higher excess-treatment rates than avoidance and transfer in every configuration, and removing explicit bounds raises the rate by up to 68.7 pp.Agents often broaden verification when its scope is unquantified, while recorded decisions to accept risk provide a stopping point.Source: Figure 4 · p. 9, §6 · p. 10

7 CONCLUSION

• ParanoiaEval brings coding-agent risk-treatment behaviors under a unified benchmark grounded in the ATMA framework.The conclusion presents it as the first benchmark for unified evaluation of risk-treatment capabilities in coding agents.Source: §7 · p. 10
• The results identify appropriate risk treatment as an independent capability and connect its diagnosis to knowledge from human software-engineering practice.Source: §7 · p. 10
• The release includes 200 evidence-controlled task pairs across 50 repositories, along with agent runners, executable oracles, reference solutions, and judge materials.The appendix documents the benchmark design, evaluation criteria, experimental protocol, judge validation, and analyses.Source: §7 · p. 10, CODE AND DATA AVAILABILITY · p. 10

A LIMITATIONS

• Controlled pairs vary one treatment-defining fact, but real evidence may be dispersed, ambiguous, or conflicting, and residual judging errors may remain.Performance on these pairs therefore does not fully represent less structured environments.Source: A LIMITATIONS · p. 16
• Native-harness evaluation preserves deployed configurations but cannot disentangle model and scaffolding effects or support harness-independent rankings.Source: A LIMITATIONS · p. 16
• Findings cover 44 situations across 50 open-source Python and Go repositories and an experienced-user study from one regional developer population.The authors caution against unconditional generalization to other ecosystems, repository types, or user populations.Source: A LIMITATIONS · p. 16
• Evidence location may partly explain treatment-level responsiveness differences: mitigation and acceptance evidence is in instructions, while avoidance and transfer evidence mostly resides in repositories.Separating location from treatment structure would require placing the same fact in both locations.Source: A LIMITATIONS · p. 16

B COMPLETENESS OF THE ATMA TREATMENTS

• The completeness argument partitions risk decisions by whether a risk can arise, who owns it, and whether retained exposure is reduced, yielding four exhaustive, mutually exclusive treatments.Source: B COMPLETENESS OF THE ATMA TREATMENTS · pp. 16–17

C LOG-MINING PROCEDURE

• From 18,922 deidentified development sessions, screening and expert review retained 5,574 consensus-valid over-defensive segments.Screening retained potential candidates when at least one model flagged them; two experts independently reviewed candidates and resolved disagreements by discussion.Source: C LOG-MINING PROCEDURE · p. 17, §C.1 · p. 17, §C.2 · pp. 17–18

C.3 SITUATION CONSOLIDATION AND HELD-OUT VALIDATION

• Expert consolidation produced 44 recurring situations, and all 100 held-out segments were assigned to one of them.Experts merged or split clusters according to distinct evidential conditions.Source: §C.3 · p. 18

D TAXONOMY OF RECURRING SITUATIONS

• Situations are defined by evidence rather than surface action, so similar actions can fall under different treatments.For example, the distinction between transfer and acceptance depends on whether follow-up responsibility is assigned to a maintainer or retained by the user.Source: D TAXONOMY OF RECURRING SITUATIONS · p. 18

E.1 AVOIDANCE

• The avoidance pair tests whether agents add compatibility protection despite repository evidence that Python 3.10+ is the only supported execution environment.The matched E− variant adds a copied-tool path still built for Python 3.8; the shared oracle checks the two-line change and targeted tests.Source: §E.1 · p. 22

E.2 TRANSFER

• The transfer pair tests whether agents duplicate validation already assigned to continuous integration when fixing ImportError handling.E+ has CI run lint, build, and a full test matrix; E− removes CI, while the shared minimal fix narrows the fallback to ModuleNotFoundError.Source: §E.2 · pp. 22–23

E.3 MITIGATION

• The mitigation pair tests whether agents exceed a verification bound that evidence fixes at one run of one test file.Under E+, the sufficient action is one specified pytest command; under E−, a focused set of relevant tests is permitted, and the oracle checks the read-only task leaves tracked files unchanged.Source: §E.3 · p. 23

E.4 ACCEPTANCE

• The acceptance pair tests whether agents respect evidence that version 0.16.1 is final rather than reopening or weakening the settled release identity.The shared instruction edits two named version strings in both variants; retaining a prerelease marker fails the oracle as task failure, not excess treatment.Source: §E.4 · pp. 23–24

F.1 FORMATTED JUDGE PROMPT

• The judge evaluates observable actions against the E+ answer card, classifies excess work as evidence-linked or generic, and records action forms and severity.A counterfactual test determines evidence linkage; small local additions are minor, while broader suites, guards, new files, or confirmation steps are clear. Its output also records evidence, fact exposure, task completion, and ambiguity.Source: §F.1 · pp. 24–25, §F.2 · p. 25

F.3 RETRIEVAL AND ASSEMBLY

• Retrieval calibrates each judge instance with human-verified exemplars matched by treatment and condition role, inserting contrast pairs when their labels and action forms expose the relevant boundary.Ordinary candidates are ranked by cosine similarity; selected contrast pairs are followed by nearest ordinary candidates until the four-exemplar budget is reached.Source: §F.3 · p. 25

G.1 HUMAN ANNOTATION RULES

• Human annotation applies the same evidence-linked versus generic distinction, excludes baseline work, and uses independent review followed by consensus adjudication.Annotators record actions, forms, evidence locations, degree, task completion, and whether the decisive fact was encountered; only consensus records become judge exemplars or validation references.Source: §G.1 · p. 26, §G.2 · p. 26

G.3 MEMORY BANK CONSTRUCTION

• Memory-bank construction samples 32 tasks and 64 condition-specific trajectories across four prediction patterns for each treatment, controlling sampling coverage without balancing final human labels.The patterns represent both variants flagged, only E+ flagged, only E− flagged, and neither flagged.Source: §G.3 · p. 26
• The memory bank retains all consensus-labeled memory records without resampling, removal, weighting, or balancing based on final human labels.The judge prompt, rules, and bank were frozen before held-out validation, whose tasks did not overlap with memory tasks.Source: §G.3 · p. 27

H.1 PARTICIPANTS

• Participants were adults with at least five years of professional software-engineering experience who regularly used coding agents; a recruitment partner verified eligibility.Recruitment targeted developers with sustained production experience, and no additional demographic information was collected.Source: §H.1 · p. 27

H.2 SAMPLING AND POST-HOC PROCEDURE

• The post-hoc sample comprised oracle-passing E+ runs, including treatment-stratified excess-treatment runs and matched no-excess records.Participants received randomized, blinded structured run records; same-task records went to different participants, with pairings rotated.Source: §H.2 · p. 27

H.3 RATING INSTRUMENT

• Participants rated satisfaction with each completed run on a five-point scale and also indicated output acceptance and unnecessary actions.Satisfaction was anchored in code maintainability, runtime, and required corrections before use.Source: §H.3 · pp. 27–28, Table 10 · p. 28

I RUNTIME AND EPISODE PROTOCOL

• Each of eight agent configurations ran three times per condition, producing 9,600 model–condition episodes.Each episode began from a fresh repository container, with an independent configuration directory and condition fixture applied before execution.Source: I RUNTIME AND EPISODE PROTOCOL · p. 28

I.1 EPISODE INITIALIZATION AND ACCESS

• Within each pair, all execution and evaluation factors remain fixed, and only the controlled evidence differs between conditions.Agents access the condition-specific repository through native file and shell tools, but not answer cards, reference solutions, or paired-condition artifacts.Source: §I.1 · p. 28

I.2 EXECUTION RECORD AND POST-RUN EVALUATION

• The post-run record includes the instruction, ordered tool and command outputs, inspected files, final response and changes, oracle result, rule-sensor observations, and resource measurements.The workspace is frozen after termination, and evaluation uses observable actions and artifacts.Source: §I.2 · p. 28

J.1 CODING AGENT HYPERPARAMETERS

• Claude-family models run in native Claude Code and GPT-family models in native Codex, with high reasoning effort for all agents.Provider restrictions prevent changing temperature and top-p when reasoning is enabled; maximum supported context and output capacities are used.Source: §J.1 · pp. 28–29

J.2 EVALUATOR HYPERPARAMETERS

• GPT-5.6 Sol evaluates trajectories with thinking disabled and retrieves four human-verified exemplars, including at most two contrast pairs.Source: §J.2 · p. 29

K JUDGE VALIDATION

• The primary GPT-5.6 Sol judge and Claude Sonnet 5 audit judge were compared on 200 human-consensus annotations from tasks outside the memory split.Both judges received the same prompt, rules, memory bank, retrieval results, and archived evidence.Source: K JUDGE VALIDATION · p. 29, Table 13 · p. 29
• Table 13 reports the judges’ agreement with held-out human consensus labels for the binary evidence-linked label Vp.Source: Table 13 · p. 29, K JUDGE VALIDATION · p. 29
• Across 3,200 E+ trajectories, the primary and audit judges showed no significant configuration-specific shift in violation-rate differences.Each configuration contributed 400 jointly judged trajectories; the comparison used task-bootstrap confidence intervals and a task-level interaction test.Source: K JUDGE VALIDATION · p. 29, Table 14 · p. 30

L.1 PER-RUN RESULTS

• Per-run results reflect both stochastic agent behavior and stochastic judge verdicts; Tables 15–17 report the individual-run results.Source: §L.1 · p. 30, Table 15 · p. 30, Table 16 · p. 30

M.1 PAIRED CONDITION EFFECTS

• For paired effects, average three verdicts per condition within each task pair; bootstrap complete pairs for 95% CIs, permute paired runs, and Holm-adjust eight tests.Table 3 computes ER from three-run mean rates, whereas Table 2 averages run-specific ERs; estimates differ by at most 0.2 pp.Source: §M.1 · p. 31, M STATISTICAL SIGNIFICANCE · p. 31

M.2 THREE-RUN STABILITY

• Across three runs, the paired condition effect showed no systematic run interaction (p = 0.43).Source: §M.2 · p. 31

N.1.1 EXECUTION WITH A USER-OWNED NEXT STEP

• With user-owned test authoring stated, the agent leaves tests untouched; without ownership stated, it adds a regression test that the judge marks evidence-linked.The two case-study variants hold the task and evaluation setup fixed while changing treatment-defining evidence.Source: §N.1 · p. 32, §N.1.1 · p. 32, §N.1.2 · p. 33

N.1.3 PAIRED COMPARISON

• Both test-ownership variants pass the same oracle, separating task completion from risk-treatment calibration.Source: §N.1.3 · p. 33

N.2.1 EXECUTION WITH THE STATED MAGNITUDE

• With a stated one-file bound, the agent ran one command, obtained 69 passing tests in 10.3 seconds, made no tracked changes, and passed the oracle.Source: §N.2.1 · p. 34

N.2.2 EXECUTION WITHOUT A STATED MAGNITUDE

• Asked whether tests currently pass, without specifying a command, path, or verification scale, the agent selected make test, expanding to tests and tools and yielding 86 collection errors in 44.6 seconds; the judge found clear scope expansion.Source: §N.2.2 · p. 34

N.2.3 PAIRED COMPARISON

• Only the statement specifying the requested verification magnitude changed between the two runs.The repository and evaluation pipeline were held fixed.Source: §N.2.3 · p. 34
• E+ completed 69 tests in 10.3 seconds with one command and test file, while E− used four commands and 44.6 seconds, triggering 86 unrelated collection errors.Without a stated bound, E− expanded to a repository-wide target and reached unrelated components, returning a less useful result at greater cost; this expansion was not a violation in E−.Source: §N.2.3 · p. 34

O COMPUTATIONAL RESOURCE CONSUMPTION

• The benchmark used 9,600 hosted-inference episodes across eight configurations, with each episode isolated to 3 CPUs and 3 GiB of memory.Each configuration had 1,200 episodes, with no training or fine-tuning; Table 19 reports total input and output tokens and cumulative runtime.Source: O COMPUTATIONAL RESOURCE CONSUMPTION · p. 35, Table 19 · p. 35