PaperlayerView exact revision on arXiv

Source-linked AI summary

Impact Is Not Invalidation: Ask About the Claim, Not the Diff

Atul Anand

arXiv:2609.25130v1cs.CLcs.AIcs.LG

TL;DR

Coding-agent memory needs to know which stored claims a repository change falsifies, but judging the diff’s behavior does not answer that claim-specific question. The paper evaluates execution-defined claims and finds that asking whether the claim still holds is more effective than judging behavior change; coverage-based impact information also leaves a gap.

  • Problem

    Behavior-preservation judgments assess a diff without consulting the stored claim, while invalidation requires predicting whether that particular claim is false after the change.

    Source: §I · p. 1, §II · p. 1

  • Method

    The benchmark treats passing test functions as claims and checks whether the same assertion text fails after a commit, then compares claim-level and diff-level judgments.

    Source: §I · p. 1, A. Claims and oracle · p. 2

  • Results

    Claim-level invalidation beats diff-level judgment across model scales, while a coverage-based selector reaches 0.412 precision despite perfect recall on the claims it tracks.

    Source: §IX · p. 9

  • Takeaways & Limitations

    The paper’s practical reading is to store claims in a form that can be queried individually rather than classifying changes.

    Source: §IX · p. 9

    Executable assertions are an exact, self-contained subset of memory claims, so the benchmark covers the checkable subset rather than vague or partly checkable notes.

    Source: §VI · p. 7

Abstract

from arXiv · show

Memory systems for coding agents must decide, when a repository changes, which of their stored claims have become false. Content anchoring invalidates a claim whenever the artifact it came from changes, which fires constantly. Semantic-equivalence classification asks whether a diff preserves behavior, a question about the diff rather than about any stored claim. We show the second signal fails for a reason unrelated to model capability: asked whether a commit preserves behavior, five models spanning a 40x price range fire on 59-72% of real commits and reach precisions of only 0.291 to 0.329 against a 0.25 base rate. Asked instead whether one specific claim still holds, the same models on the same diffs reach 0.705 to 0.974. A control that hands the behavior-preservation judge the claim text, changing only the question, moves precision by 0.010 and 0.016; changing the question moves it by 0.49 and 0.65. We also compare against pytest-testmon, a deployed regression-test selector with coverage-derived dependency data: it reaches 0.868 recall at 0.415 precision, so near-complete knowledge of what a change can reach does not identify what it falsifies. Ground truth is execution, not annotation: a claim is a test function passing at commit t, and it has flipped if that same assertion text fails at t+1. Building this required an observation we did not find in prior work. On a CI-gated mainline a commit that leaves a pre-existing test failing cannot merge, so the naive construction has an empty positive class by design. We report 10,369 claims with 184 execution-verified flips mined from 23 Python libraries, splits held out by repository, a post-knowledge-cutoff split, a shuffled-diff null, a paraphrase control, and a leave-one-repository-out analysis over 17 repositories.

I. Introduction

Coding-agent memory must decide when repository changes make stored claims false, but artifact changes and behavior changes are not the same as claim invalidation. The paper argues that asking whether a specific claim still holds captures most of the available accuracy.

I. Introduction

  • Content anchoring withdraws claims when their source artifact changes, although most file changes do not affect most claims about that file.The introduction notes that false supersession destroys information and cannot be recovered, making precision important.Source: §I · p. 1
  • Semantic-equivalence judges whether a diff changes behavior, not whether a stored claim remains true; a behavior-changing diff may leave an unrelated claim intact.Conversely, a behavior-preserving rename can falsify a claim that names the old symbol.Source: §I · p. 1, §II · p. 1
  • The paper finds that changing the question from behavior preservation to claim validity accounts for most of the available accuracy.Its contributions include an execution-grounded benchmark and a control separating the question asked from the evidence shown.Source: §I · p. 1

II. Problem

  • Claim-relative invalidation asks whether a specific claim is false after a diff, conditioning on both the claim and the change.A predictor should avoid re-verifying every stored claim by execution, which would require O(|store|) test runs per commit.Source: §II · p. 1
  • Figure 1 illustrates the mismatch: removing an argument check changes nth_combination, while the untouched claim about all_equal still holds.The semantic-equivalence judge is right about the behavior change but does not answer whether that stored claim is false.Source: Figure 1 · p. 2

III. Benchmark

The benchmark treats each claim as an assertion that passes at one commit and checks whether the same assertion fails after a change. CI gating makes the naive way of finding flips empty, so the construction restores parent-version tests against child source.

A. Claims and oracle

  • A claim is a test function with assertions that passes deterministically at commit t and flips when the same assertion text fails at t + 1.The oracle is execution, with no human judgment or judge model.Source: A. Claims and oracle · p. 2

B. CI-gating makes the naive construction empty

  • The naive search found zero flips among 1,005 passing, unmodified tests because CI prevents commits that leave pre-existing tests failing from reaching mainline.This makes the positive class empty by construction on repositories with CI, rather than through a sampling accident.Source: B. CI-gating makes the naive… · p. 2
  • To recover flips, the benchmark restores each touched test file to its parent version and runs that assertion against the child source.Because the restored assertion also appears in the test diff, prompts exclude test files and contain no test content.Source: B. CI-gating makes the naive… · p. 2

C. Mining

The benchmark mines commits using deleted test assertions as a signal of retired expectations, then separates repositories to reduce claim-level leakage. It compares claim-relative prediction with content and change-impact baselines as well as behavior-preservation judgments.

C. Mining

  • Ranking commits by deleted assertion lines raises flip density from 0.36% to 2.0%, outperforming commit-message keyword ranking.Assertion deletion directly signals a retired expectation, whereas keywords such as “fix” can point to documentation-only changes.Source: C. Mining · p. 2

D. Corpus

  • Mining 23 Python libraries yielded 10,369 claims and 184 flips (1.8%), with flips in 10 repositories.Repository-held-out splits contain 604 stratified claims with 151 flips from 17 repositories; prompt development used only boltons and arrow.Source: D. Corpus · pp. 2–3

IV. Arms •

  • The arms compare file anchoring, symbol anchoring, pytest-testmon coverage selection, semantic-equivalence judgment, a claim-text control, and claim-relative judgment.Testmon selects tests whose covered source lines changed; its coverage database requires a parent-commit instrumented run.Source: §IV · p. 3

V. Results

On held-out repositories, claim-relative A6 outperforms symbol-anchoring A4 on both precision and recall. Against pytest-testmon, coverage-based impact selection catches many flips but does not establish that the affected claims are false.

A. Held-out repositories

  • 40.7 points more precision at 18 points more recall: A6 exceeds A4, surpassing the pre-registered criterion of 15 precision points at matched recall.A6 catches 108 of 151 flips versus A4’s 81, while raising 28 false alarms versus A4’s 128; the paired correctness test gives p = 1 × 10−20.Source: A. Held-out repositories · p. 3
  • 0.291 precision against a 0.25 base rate: the strengthened semantic-equivalence arm fires on 59% of held-out claims.The result is close to invalidating everything rather than selectively identifying flipped claims.Source: A. Held-out repositories · p. 3

B. Against a real test-selection tool

  • 0.868 recall at 0.415 precision: pytest-testmon catches 131 of 151 flips, but knowing which tests a change can affect does not show which claims it falsifies.A6 and testmon are separated on paired correctness (p = 3 × 10−20); testmon requires a coverage-instrumented parent run, while A6 reads text and runs nothing.Source: B. Against a real test-selection tool · p. 3

C. The question, not the evidence

Giving the behavior-preservation judge the claim text changes its behavior but not its accuracy; A6 remains more often correct, while the held-out comparison with testmon shows a budget–staleness trade-off.

C. The question, not the evidence

  • A6 is correct where A5C is wrong on 84 claims, versus 9 claims in the reverse direction (p = 2 × 10−16).A5C gives the behavior-preservation judge the claim text; it fires on 70% of held-out claims and reaches recall 0.972, but still sacrifices selectivity.Source: C. The question, not the evidence · p. 3
  • Ten of 23 repositories produced flips, and Table I lists repositories with no positives as part of the mining cost.Source: Table I · p. 4
  • The held-out split contains 604 claims and 151 flips from 17 repositories, stratified to a 25% flip rate with 95% bootstrap intervals.Source: Table II · p. 4
  • On the held-out split, testmon buys lower staleness with a larger verification budget, while A6 buys a smaller budget at higher staleness.Testmon supplied a verdict for 80% of claims; the remaining claims are charged to it as holds.Source: Table III · p. 4

D. Is one repository carrying the result?

The claim-relative advantage persists when any single held-out repository is removed. In practitioner terms, A6 improves on symbol anchoring in both stale claims and re-verification budget, and its advantage over baselines survives when claims are rewritten as prose.

D. Is one repository carrying the result?

  • Removing any one repository leaves the A6–A4 precision gap above the +15-point threshold; the smallest gap is +34.4 points.The smallest gap occurs when cachetools is removed; removing more-itertools widens it to +46.7 points.Source: D. Is one repository carrying the… · p. 4, Table V · p. 6

E. What a practitioner buys

  • A6 serves fewer stale claims than symbol anchoring (28% vs. 46%) while re-verifying less of the store (22% vs. 35%).Testmon achieves the lowest staleness among predictive arms, 13%, but uses 52% of the budget and requires suite execution.Source: E. What a practitioner buys · p. 4
  • At the natural 1.78% flip rate, A6 precision is 0.174 against a 0.018 base rate, while semantic equivalence reaches 0.022.Impact-analysis arms reach 0.033 and 0.037.Source: E. What a practitioner buys · p. 5

F. Do the results hold for prose claims?

  • Rewriting each test claim as prose preserves the oracle but removes exact assertion details; the rewriting pass sees only the parent test.The prose arm's precision drops from 0.794 to 0.711.Source: F. Do the results hold for prose claims? · p. 5
  • 0.711 prose-arm precision remains above testmon (0.415), symbol anchoring (0.388), and semantic equivalence (0.291) on the same claims.The code-form precision is 0.794; the prose rewrite loses some detail that makes a flip decidable.Source: F. Do the results hold for prose claims? · p. 5

G. Model ladder

Across five models spanning roughly a 40-fold price range, the behavior-preservation question remains near always-invalidating. The cheapest tested model reaches 0.794 precision at $0.025/M, within 0.18 of the frontier model at roughly one-fortieth the price.

G. Model ladder

  • Across five models and roughly a 40-fold price range, behavior-preservation precision stays within a 0.04 band near always-invalidating, while the same frontier model reaches 0.974 on the claim-level question.The frontier model is worst by the behavior-preservation measure, firing on 72% of claims.Source: G. Model ladder · p. 6
  • 0.794 precision at $0.025/M puts the cheapest tested model within 0.18 of the frontier model at roughly a fortieth of the price.The paper notes that an invalidation gate costing more than rerunning the test has no reason to exist.Source: G. Model ladder · p. 6

H. Controls

The controls test whether the findings reflect recency, claim wording, or repository composition. The validity discussion bounds the results to executable claims, small Python libraries, and the study’s evaluation setting.

H. Controls

  • With unrelated diffs, A6 fires on 0% of claims at 100% coverage, compared with 15% on the original claims.The shuffled-diff null pairs each claim with a different commit’s diff, preferably from another repository.Source: H. Controls · p. 6
  • On the post-cutoff split, A6 reaches 0.750 precision and 0.955 recall on the budget model, and 0.913 precision and 1.000 recall on the frontier model.The corpus contains 730 claims and 22 flips, so the authors present it as evidence against contamination, not as a second estimate.Source: H. Controls · p. 7
  • When the repository is held fixed to click, every arm reproduces within 0.03 across the cutoff; boltons drives the full-split reordering with a 54% flip rate.On boltons, A4 reaches 0.750 precision while testmon reaches 0.286.Source: H. Controls · p. 7, Table IX · p. 8
  • A6 agrees with syntactically rewritten claims on 430 of 450 verdicts (95.6%), with precision 0.816 versus 0.794 on originals; rename-class recall was not evaluable at n = 1.The held-out split contained zero rename claims in development and one in held-out data.Source: H. Controls · p. 7

VI. Threats to validity

  • The benchmark covers only executable Python assertions, an exact and self-contained subset rather than a representative sample of memory-store claims.The authors describe it as a lower bound on claim diversity.Source: §VI · p. 7
  • Ten of 23 repositories yielded flips, and two supplied 84 of 151 held-out positives, making the effective sample smaller than n suggests.The authors report that the conclusion survives every single-repository deletion.Source: §VI · p. 8
  • The natural flip rate is 2.0%, so deployment precision would be much lower than the stratified estimates.Table VI reweights precision to the natural rate and reports re-verification costs under a stated policy, not an observed integration.Source: §VI · p. 8
  • Evidence is limited to 23 small Python libraries; the authors make no claim about industrial monorepos or other languages.They note that static types may let symbol-level baselines perform better, while longer dependency chains in large codebases may reduce impact-analysis precision.Source: §VI · p. 8

VII. Artifact

The artifact releases the benchmark, mining pipeline, evaluation arms, and per-claim predictions. It also includes lightweight checks for pipeline invariants and a test for a major measurement hazard.

VII. Artifact

  • The released artifact includes the benchmark, mining pipeline, every arm, and per-claim predictions for each model and split.The release also includes two tools aimed at reuse.Source: §VII · p. 8
  • repro_smoke.py runs the pipeline on one repository and three commits in under a minute, checking claim text, parent-pass stability, complete post-change verdicts, and test-file exclusion.It caught a source-resolution defect affecting tests inherited from mixin base classes.Source: §VII · p. 8
  • test_miner.py ensures that a pytest batch aborted by a collection error yields no verdicts rather than apparent failures.This pins the parsing behavior behind the project’s worst measurement error.Source: §VII · p. 8

VIII. Related work

Prior work studies memory staleness, semantic change detection, and regression-test selection, but these approaches answer questions different from whether a particular stored claim remains true. The paper argues that claim-relative invalidation is the relevant precision gate for memory stores.

VIII. Related work

  • Prior memory-staleness work uses content anchoring or similarity and reports precision concerns, while this paper distinguishes its execution-grounded claim labels from annotation or synthetic-oracle ground truth.The cited memory studies include EA-Graph [1], temporal-validity retrieval memory [4], and PrecisionMemBench [5].Source: §VIII · p. 8
  • Semantic-change systems such as ChangeGuard [2] and SemaDiff [3] identify behavior-changing commits, while test selectors choose tests a change can reach,,,,,.The paper compares against pytest-testmon and describes A4 as static symbol-level impact analysis.Source: §VIII · pp. 8–9
  • Testmon has recall 1.000 on the claims it tracks but precision 0.412, because reachability does not mean a test flips.The paper contrasts the selector’s safe-superset objective with a memory gate’s cost of false alarms.Source: §VIII · p. 9
  • FineRTS improves test-selection precision by reasoning about semantics, but still asks which tests to rerun rather than whether each stored claim remains true.Its target is a safe superset for a test runner; the paper’s target is a precision gate for a memory store.Source: §VIII · p. 9

IX. Conclusion

  • A budget model asked the claim-level question beats a frontier model asked the diff-level question, while the latter does no better than always invalidating.The paper concludes that invalidation accuracy depends on the question asked, not what the model sees or its size.Source: §IX · p. 9
  • A coverage-based selector with perfect recall still reaches 0.412 precision, so knowing what a change can reach does not establish what it falsifies.The paper’s practical reading is to store claims in a form that can be interrogated individually; benchmark materials and predictions are released.Source: §IX · p. 9

Appendix A Prompts

The appendix specifies the prompts used to compare diff-level behavior judgments with claim-relative judgments. The control supplies claim text while preserving the behavior-preservation question, isolating the effect of changing the question.

Appendix A Prompts

  • All model-based arms use one system prompt over the same user message, and the four prompts are reproduced in full.A5 receives the diff alone; the other described arms receive claim and diff.Source: Appendix A · p. 9
  • The prompt inputs include the claim and diff for A5C and A6, with diffs truncated at 12,000 characters and test files excluded.No few-shot examples, chain-of-thought instruction, or retries are used.Source: Appendix A · p. 9
  • A5 judges whether a diff changes externally observable behavior for pre-existing code, with rules for behavior-preserving edits and behavior-changing effects.Its prompt excludes behavior changes that affect only code paths no pre-existing caller reaches.Source: A5, semantic equivalence (strict) · p. 9
  • A5C adds the test as context but explicitly asks the model to judge the diff, leaving the rest of A5 verbatim.A5C and A6 receive the same user message and return the same one-key JSON; only the question differs, and precision moves by 40 to 67 points across the 2×2.Source: A5C, the information control · p. 9, A6 risk, the arm we report · p. 10
  • A6 asks whether a test claim’s assertions still pass after the diff, returning a held-or-flipped verdict.The binary prompt precedes risk framing; the risk-framed version was selected on development data and used unchanged for held-out and post-cutoff results.Source: A6 base, the claim-relative question · p. 9, A6 risk, the arm we report · p. 10
Loading 2609.25130v1…