PaperlayerView exact revision on arXiv

Source-linked AI summary

SemVerBench: Benchmarking LLM Comprehension of Version-Constraint Resolution Semantics

Qibai Chen, Zeming Liu

arXiv:2609.11180v1cs.AIcs.SE

TL;DR

LLM coding agents frequently resolve version constraints, but their semantic competence has not been directly measured. SemVerBench directly evaluates this ability across three ecosystems and finds predictable mechanism-specific blind spots, while resolver delegation reaches approximately 100%.

  • Problem

    LLM coding agents routinely make version-membership decisions, yet version-constraint semantics have not been directly benchmarked.

  • Method

    SemVerBench evaluates 240 unique-answer, machine-checkable membership items across npm, PEP 440, and Cargo using author-neutral sources and independent oracle labeling.

  • Results

    Models show predictable mechanism-specific blind spots: Cargo partial-comparator carry brings every model near 60%, while GPT-5.1 fails all 26 tested zero-pad cases and Claude scores 97–100%.

  • Takeaways & Limitations

    Because a free deterministic resolver exists and delegation reaches approximately 100%, coding agents should delegate version resolution rather than perform it in-head.

  • Takeaways & Limitations

    The study cannot fully distinguish activating latent knowledge from supplying missing knowledge, and its conclusion concerns the tested benchmark setting.

Abstract

from arXiv · show

Large language model (LLM) coding agents constantly decide whether a version satisfies a constraint such as ^1.2.3 or >=2.0,<3, yet their grasp of version-constraint semantics has never been measured directly. We introduce SemVerBench, the first benchmark of LLM version-constraint resolution semantics across three ecosystems (npm, PEP 440, Cargo): 240 machine-checkable items with unique answers, built author-neutrally from four balanced sources (each ecosystem's official test suite plus three frontier LLM proposers) and labeled by a non-circular two-implementation oracle. Evaluating six frontier models, we find systematic, predictable per-mechanism blind spots: a partial-comparator carry rule (>1.2 means >=1.3.0) traps every model on Cargo (near 60%), and although standard PEP 440 prefix matching is universal, on zero-pad/post-release corner cases GPT-5.1 collapses (0/26) while Claude stays at 97-100% (verified on a 67-item oracle-validated set). Opus significantly outperforms all other models, and Sonnet outperforms the OpenAI models (McNemar). The failures look more like an activation/application gap than a knowledge gap: injecting the rule or a light correct hint recovers most errors, whereas interval decomposition does not, and models are at ceiling on the basic forms of the same rules. An author-stratified analysis finds no statistically significant self-favoritism. Because the task is verifiable and a free, 100%-correct resolver exists, tool delegation reaches ~100%: coding agents should delegate version resolution to a resolver rather than reason about versions in-head.

I. INTRODUCTION

SemVerBench measures how six frontier models resolve version constraints across three ecosystems, revealing systematic mechanism-specific failures despite strong basic-rule performance.

  • 240 machine-checkable items across npm, PEP 440, and Cargo isolate version-membership semantics with unique answers and author-neutral oracle labels.The benchmark removes ambiguity from open-ended generation and attributes errors to constraint semantics.
  • Models perform at or near ceiling on basic grammar forms, while rule injection and light hints recover most errors but interval decomposition does not.The pattern supports an activation/application gap, although the design cannot fully separate activation from missing knowledge.
  • Every model falls to roughly 60% on Cargo partial-comparator carry, where >1.2 means >=1.3.0 and >1 means >=2.0.0.This is a universal, reproducible blind spot rather than diffuse failure.
  • Tool delegation reaches approximately 100%, motivating resolver use instead of in-head version resolution by coding agents.A free deterministic resolver answers these queries in microseconds.

II. RELATED WORK

SemVerBench isolates version-constraint membership, a verifiable sub-competence that prior software-engineering, dependency, and constraint benchmarks do not directly measure.

  • SemVerBench complements broad software-engineering benchmarks by isolating version membership from end-to-end issue resolution.This makes a specific, high-frequency capability directly testable.
  • Prior dependency benchmarks evaluate inferred manifests or dependency-graph understanding, not whether a version satisfies a single constraint.DI-BENCH assesses repository build and test outcomes, while DependEval probes graph relations.
  • Constraint-following benchmarks study natural-language format, length, or content constraints rather than package-manager version semantics.SemVerBench targets the downstream task of resolving a version against a specifier.

A. Task

The task is a Boolean membership query over ecosystem, version, and constraint triples, spanning three grammars whose subtle semantic differences create targeted mechanisms for evaluation.

  • A. Task: Each item asks whether version V satisfies constraint C in ecosystem E, with exact-match scoring against a deterministic oracle verdict.Parse failures are counted separately from incorrect answers.
  • A. Task: The benchmark covers npm, PEP 440, and Cargo, whose overlapping grammars differ in caret, tilde, prefix, prerelease, and partial-comparator semantics.Cargo partial-comparator handling is the dominant difficulty source in the data.
  • A. Task: Syntax records how a constraint is written, whereas mechanism records the semantic rule it exercises; the axes are orthogonal.One syntax can exercise multiple mechanisms, and one mechanism can appear under different syntax forms.
  • A. Task: Evaluated mechanisms include magic-zero caret/tilde behavior, partial-comparator carry, prerelease admission, ordered-comparison exclusion, epochs, and prefix matching.Examples include >1.2 ≡>=1.3.0 and PEP 440 exclusion of adjacent boundary pre/post releases.

D. Author-Neutral Construction

SemVerBench combines official resolver tests with equally prompted frontier-LLM proposals, independently oracle-labeling all items to diversify difficulty without using proposer answers.

  • D. Author-Neutral Construction: The main benchmark contains 240 items, with 80 per ecosystem and 20 from each of four sources.Sources comprise one official test suite and three frontier LLM proposers per ecosystem.
  • D. Author-Neutral Construction: Claude, GPT, and Gemini proposers received identical prompts and fixed topic areas, while their suggested answers were discarded.This design avoids vendor-favored prompting while acknowledging that the prompt is not content-free.
  • D. Author-Neutral Construction: A two-implementation oracle independently labels every item, and proposer identities enable post-hoc selection-bias and self-favoritism analysis.A separate 67-item author-constructed prefix-match set supports focused validation but is outside the main 240 items.

E. Ground Truth: A Non-Circular Two-Implementation Oracle

SemVerBench labels each membership query only when two independently authored implementations agree, keeping ground truth separate from both item proposers and evaluated models.

  • E. Ground Truth: A Non-Circular Two-Implementation Oracle: Each item’s answer is retained only when two independent reference implementations per ecosystem agree.The implementations are node-semver × semantic_version for npm, packaging × pep440_rs for PEP 440, and the Rust semver crate × semantic_version for Cargo.
  • E. Ground Truth: A Non-Circular Two-Implementation Oracle: The oracle is non-circular because its implementations have independent codebases and no panel model labels items.For PEP 440, the implementations are maintained in different languages and item labels require agreement; panel models act only as proposers.
  • E. Ground Truth: A Non-Circular Two-Implementation Oracle: Evaluation uses three temperature-0 runs over 240 items, Wilson 95% confidence intervals, and exact McNemar tests for paired comparisons.The majority-class predictor scores 57.1%, providing the stated floor for interpreting accuracies.

IV. FINDINGS

Across the benchmark, models show substantial aggregate competence but sharply uneven mechanism-level performance, with Claude models leading the paired comparisons.

  • IV. FINDINGS: 90.6% overall accuracy places Claude-Opus first, followed by Claude-Sonnet at 86.8%, while OpenAI and Gemini cluster between 80% and 83%.McNemar testing finds Opus significantly above every other model and Sonnet significantly above each OpenAI model.
  • IV. FINDINGS: 80–91% overall accuracy exceeds the 57.1% majority-class floor, confirming genuine competence on most items.Several individual blind-spot buckets nevertheless do not clear the majority-class floor.
  • IV. FINDINGS: Plain membership, epochs, and basic ranges are near 100% for all models, while difficulty concentrates in a few mechanisms.Cargo partial-carry is approximately 60%, and PEP 440 ordered exclusion is 60–70% for the weaker models.
  • IV. FINDINGS: GPT-5.1’s 80.4% accuracy is numerically below GPT-4.1’s 82.2%, but the within-vendor difference is not statistically significant.The authors interpret this only as evidence that recency does not predict competence on this task.

B. Finding 2: Partial-comparator carry is a universal trap

Partial-comparator carry is a mechanism-level blind spot across ecosystems, especially severe on Cargo, despite being a short documented rule.

  • B. Finding 2: Partial-comparator carry is a universal trap: Cargo partial-carry accuracy falls to roughly 60% for every model, including Opus, making it the benchmark’s clearest cross-vendor blind spot.Opus scores 63%, Sonnet 59%, and the remaining four models fall in the 62–63% band.
  • B. Finding 2: Partial-comparator carry is a universal trap: The carry rule rounds partial comparators upward: >1.2 becomes >=1.3.0 and >1 becomes >=2.0.0.Thus 1.0.5 does not satisfy >1, although models commonly treat the constraint as >1.0.0.
  • B. Finding 2: Partial-comparator carry is a universal trap: All six models score exactly 80% on npm’s partial-carry mechanism, or 16 of 20 items correct under three-run majority.The identical score across models suggests the difficulty is tied to the mechanism rather than a single model.

D. Finding 4: Ordered-comparison exclusion cracks Sonnet

PEP 440 ordered-comparison exclusion is broadly difficult and disproportionately lowers Sonnet, while source stratification finds no significant self-favoritism.

  • D. Finding 4: Ordered-comparison exclusion cracks Sonnet: PEP 440 ordered-comparison exclusion reaches only 60–70% for most models, with Sonnet at 65% and Opus at 84%.The rule excludes >V matching a post-release of V and <V matching a pre-release of V.
  • D. Finding 4: Ordered-comparison exclusion cracks Sonnet: No model shows a statistically significant own-versus-other-LLM item advantage; GPT-4.1’s largest gap is +10 points with p ≈0.10.The only significant author effect is Gemini scoring lower on its own items.
  • D. Finding 4: Ordered-comparison exclusion cracks Sonnet: The blind spots appear on official-suite and cross-family items for every model, so they are not explained by self-authored items.The official suites have no panel-model stake, and the oracle labels items independently of any model.

F. Error analysis: recurring failure modes

The error analysis identifies recurring, mechanism-specific rule misreadings and shows that rule availability or resolver delegation is more effective than extra decomposition steps.

  • Recurring failure modes: Three recurring errors are partial-comparator under-carry, PEP 440 prefix under-match, and boundary pre/post admission.These correspond to M1, M2, and M3, respectively.
  • Recurring failure modes: Cargo accuracy falls to roughly 60% for every model on the partial-comparator carry rule, while other Cargo mechanisms remain near ceiling.Partial comparators such as >1.2 round up to >=1.3.0.
  • By ecosystem: PEP 440 is hardest overall, Cargo isolates the partial-carry trap, and npm shows intermediate partial-carry and prerelease difficulty.PEP 440 concentrates two of the three failure modes, whereas Cargo’s other mechanisms are near ceiling.
  • Mitigation: Rule injection lifts every model substantially, whereas interval decomposition produces essentially no lift.Examples include GPT-5.1 rising from 80.4 to 91 and GPT-4.1 from 82.2 to 95.
  • Mitigation: Tool delegation reaches approximately 100%, with only 2 of 1440 tool-condition responses retaining a wrong verdict after receiving the correct result.The official resolver is wrapped by a check_constraint tool.
  • Implications: Agents should route dependency membership and range decisions through the ecosystem’s official resolver rather than resolve them in-head.When no resolver is reachable, rule injection is the supported second-best mitigation.

A. Controlled false-premise sensitivity

The controlled study tests whether weak hints change performance on a preselected hard subset, finding a large benefit from correct hints and limited influence from false ones.

  • Study design: The study uses a pre-specified hard 61-item subset and compares TRUE, NEUTRAL, FALSE, and no-hint conditions.Items were selected as the lowest-baseline items per mechanism rather than hand-picked.
  • Results: A TRUE hint raises Sonnet from 67% to 97%, correcting roughly 18 of 20 previously failed items.The hint is weak and hedged rather than forceful.
  • Results: Weak FALSE hints typically cause only 0–2 answer flips and often leave accuracy at or above baseline.For the weakest-baseline models, merely presenting a hedged premise may trigger more careful rechecking.

VI. LIMITATIONS AND CONCLUSION

The paper concludes that frontier models systematically misapply version rules despite handling basic forms, and recommends resolver delegation while delimiting the benchmark’s scope and interpretation.

  • Limitations: The benchmark is a single snapshot tied to each ecosystem’s official resolver library, with resolver-specific policy nuances outside scope.PEP 440 prereleases are filtered, and mitigation and false-premise studies are single-run and directional.
  • Limitations: The design cannot fully distinguish activating latent knowledge from supplying missing knowledge because both rule injection and TRUE hints provide rules in context.The author-stratified analysis finds no statistically significant self-favoritism.
  • Conclusion: A universal partial-comparator trap lowers every model to approximately 60% on Cargo, while PEP 440 corner cases collapse GPT-5.1 and leave Claude near-perfect.Ordered-comparison exclusions also challenge strong models, and Claude leads significantly.
  • Conclusion: The failures are more consistent with an activation/application gap than a knowledge gap because basic forms are handled perfectly and rule access recovers errors.The conclusion frames this as a diagnosis rather than a definitive separation of alternatives.
  • Conclusion: Tool delegation reaches approximately 100%, supporting the recommendation that coding agents delegate version-constraint resolution instead of reasoning in-head.The resolver is free and 100%-correct for this task.
Loading 2609.11180v1…