SemVerBench: Benchmarking LLM Comprehension of Version-Constraint Resolution Semantics

• 240 machine-checkable items across npm, PEP 440, and Cargo isolate version-membership semantics with unique answers and author-neutral oracle labels.The benchmark removes ambiguity from open-ended generation and attributes errors to constraint semantics.Source: §I · pp. 1–2
• Models perform at or near ceiling on basic grammar forms, while rule injection and light hints recover most errors but interval decomposition does not.The pattern supports an activation/application gap, although the design cannot fully separate activation from missing knowledge.Source: §I · pp. 1–2
• Every model falls to roughly 60% on Cargo partial-comparator carry, where >1.2 means >=1.3.0 and >1 means >=2.0.0.This is a universal, reproducible blind spot rather than diffuse failure.Source: §I · p. 2
• Tool delegation reaches approximately 100%, motivating resolver use instead of in-head version resolution by coding agents.A free deterministic resolver answers these queries in microseconds.Source: §I · pp. 1–2

• SemVerBench complements broad software-engineering benchmarks by isolating version membership from end-to-end issue resolution.This makes a specific, high-frequency capability directly testable.Source: §II · p. 2
• Prior dependency benchmarks evaluate inferred manifests or dependency-graph understanding, not whether a version satisfies a single constraint.DI-BENCH assesses repository build and test outcomes, while DependEval probes graph relations.Source: §II · p. 2
• Constraint-following benchmarks study natural-language format, length, or content constraints rather than package-manager version semantics.SemVerBench targets the downstream task of resolving a version against a specifier.Source: §II · p. 2

A. Task

• Each item asks whether version V satisfies constraint C in ecosystem E, with exact-match scoring against a deterministic oracle verdict.Parse failures are counted separately from incorrect answers.Source: A. Task · p. 2
• The benchmark covers npm, PEP 440, and Cargo, whose overlapping grammars differ in caret, tilde, prefix, prerelease, and partial-comparator semantics.Cargo partial-comparator handling is the dominant difficulty source in the data.Source: B. Ecosystems and Their Grammars · pp. 2–3
• Syntax records how a constraint is written, whereas mechanism records the semantic rule it exercises; the axes are orthogonal.One syntax can exercise multiple mechanisms, and one mechanism can appear under different syntax forms.Source: C. The Two-Axis Taxonomy · p. 3
• Evaluated mechanisms include magic-zero caret/tilde behavior, partial-comparator carry, prerelease admission, ordered-comparison exclusion, epochs, and prefix matching.Examples include >1.2 ≡>=1.3.0 and PEP 440 exclusion of adjacent boundary pre/post releases.Source: C. The Two-Axis Taxonomy · p. 3

D. Author-Neutral Construction

• The main benchmark contains 240 items, with 80 per ecosystem and 20 from each of four sources.Sources comprise one official test suite and three frontier LLM proposers per ecosystem.Source: D. Author-Neutral Construction · p. 3
• Claude, GPT, and Gemini proposers received identical prompts and fixed topic areas, while their suggested answers were discarded.This design avoids vendor-favored prompting while acknowledging that the prompt is not content-free.Source: D. Author-Neutral Construction · p. 3
• A two-implementation oracle independently labels every item, and proposer identities enable post-hoc selection-bias and self-favoritism analysis.A separate 67-item author-constructed prefix-match set supports focused validation but is outside the main 240 items.Source: D. Author-Neutral Construction · p. 3

E. Ground Truth: A Non-Circular Two-Implementation Oracle

• Each item’s answer is retained only when two independent reference implementations per ecosystem agree.The implementations are node-semver × semantic_version for npm, packaging × pep440_rs for PEP 440, and the Rust semver crate × semantic_version for Cargo.Source: E. Ground Truth: A Non-Circular… · p. 3
• The oracle is non-circular because its implementations have independent codebases and no panel model labels items.For PEP 440, the implementations are maintained in different languages and item labels require agreement; panel models act only as proposers.Source: E. Ground Truth: A Non-Circular… · p. 3
• Evaluation uses three temperature-0 runs over 240 items, Wilson 95% confidence intervals, and exact McNemar tests for paired comparisons.The majority-class predictor scores 57.1%, providing the stated floor for interpreting accuracies.Source: F. Evaluation Protocol · pp. 3–4

IV. FINDINGS

• 90.6% overall accuracy places Claude-Opus first, followed by Claude-Sonnet at 86.8%, while OpenAI and Gemini cluster between 80% and 83%.McNemar testing finds Opus significantly above every other model and Sonnet significantly above each OpenAI model.Source: §IV · p. 4, A. Finding 1: A cross-vendor gap · p. 4
• 80–91% overall accuracy exceeds the 57.1% majority-class floor, confirming genuine competence on most items.Several individual blind-spot buckets nevertheless do not clear the majority-class floor.Source: §IV · p. 4
• Plain membership, epochs, and basic ranges are near 100% for all models, while difficulty concentrates in a few mechanisms.Cargo partial-carry is approximately 60%, and PEP 440 ordered exclusion is 60–70% for the weaker models.Source: §IV · p. 4
• GPT-5.1’s 80.4% accuracy is numerically below GPT-4.1’s 82.2%, but the within-vendor difference is not statistically significant.The authors interpret this only as evidence that recency does not predict competence on this task.Source: A. Finding 1: A cross-vendor gap · p. 4

B. Finding 2: Partial-comparator carry is a universal trap

• Cargo partial-carry accuracy falls to roughly 60% for every model, including Opus, making it the benchmark’s clearest cross-vendor blind spot.Opus scores 63%, Sonnet 59%, and the remaining four models fall in the 62–63% band.Source: B. Finding 2: Partial-comparator carry… · p. 4
• The carry rule rounds partial comparators upward: >1.2 becomes >=1.3.0 and >1 becomes >=2.0.0.Thus 1.0.5 does not satisfy >1, although models commonly treat the constraint as >1.0.0.Source: B. Finding 2: Partial-comparator carry… · p. 4
• All six models score exactly 80% on npm’s partial-carry mechanism, or 16 of 20 items correct under three-run majority.The identical score across models suggests the difficulty is tied to the mechanism rather than a single model.Source: B. Finding 2: Partial-comparator carry… · p. 4

D. Finding 4: Ordered-comparison exclusion cracks Sonnet

• PEP 440 ordered-comparison exclusion reaches only 60–70% for most models, with Sonnet at 65% and Opus at 84%.The rule excludes >V matching a post-release of V and <V matching a pre-release of V.Source: D. Finding 4: Ordered-comparison… · p. 5
• No model shows a statistically significant own-versus-other-LLM item advantage; GPT-4.1’s largest gap is +10 points with p ≈0.10.The only significant author effect is Gemini scoring lower on its own items.Source: E. Finding 5: Conflict of interest: no… · p. 5, Table IV · p. 6
• The blind spots appear on official-suite and cross-family items for every model, so they are not explained by self-authored items.The official suites have no panel-model stake, and the oracle labels items independently of any model.Source: E. Finding 5: Conflict of interest: no… · pp. 5–6

Recurring failure modes

• Three recurring errors are partial-comparator under-carry, PEP 440 prefix under-match, and boundary pre/post admission.These correspond to M1, M2, and M3, respectively.Source: F. Error analysis: recurring failure… · p. 6
• Cargo accuracy falls to roughly 60% for every model on the partial-comparator carry rule, while other Cargo mechanisms remain near ceiling.Partial comparators such as >1.2 round up to >=1.3.0.Source: F. Error analysis: recurring failure… · p. 6

By ecosystem

• PEP 440 is hardest overall, Cargo isolates the partial-carry trap, and npm shows intermediate partial-carry and prerelease difficulty.PEP 440 concentrates two of the three failure modes, whereas Cargo’s other mechanisms are near ceiling.Source: F. Error analysis: recurring failure… · p. 6

Mitigation

• Rule injection lifts every model substantially, whereas interval decomposition produces essentially no lift.Examples include GPT-5.1 rising from 80.4 to 91 and GPT-4.1 from 82.2 to 95.Source: §V · p. 6
• Tool delegation reaches approximately 100%, with only 2 of 1440 tool-condition responses retaining a wrong verdict after receiving the correct result.The official resolver is wrapped by a check_constraint tool.Source: §V · p. 6

Implications

• Agents should route dependency membership and range decisions through the ecosystem’s official resolver rather than resolve them in-head.When no resolver is reachable, rule injection is the supported second-best mitigation.Source: §V · pp. 6–7

Study design

• The study uses a pre-specified hard 61-item subset and compares TRUE, NEUTRAL, FALSE, and no-hint conditions.Items were selected as the lowest-baseline items per mechanism rather than hand-picked.Source: A. Controlled false-premise sensitivity · p. 7

Results

• A TRUE hint raises Sonnet from 67% to 97%, correcting roughly 18 of 20 previously failed items.The hint is weak and hedged rather than forceful.Source: A. Controlled false-premise sensitivity · p. 7
• Weak FALSE hints typically cause only 0–2 answer flips and often leave accuracy at or above baseline.For the weakest-baseline models, merely presenting a hedged premise may trigger more careful rechecking.Source: A. Controlled false-premise sensitivity · p. 7

Limitations

• The benchmark is a single snapshot tied to each ecosystem’s official resolver library, with resolver-specific policy nuances outside scope.PEP 440 prereleases are filtered, and mitigation and false-premise studies are single-run and directional.Source: §VI · p. 7
• The design cannot fully distinguish activating latent knowledge from supplying missing knowledge because both rule injection and TRUE hints provide rules in context.The author-stratified analysis finds no statistically significant self-favoritism.Source: §VI · p. 7

Conclusion

• A universal partial-comparator trap lowers every model to approximately 60% on Cargo, while PEP 440 corner cases collapse GPT-5.1 and leave Claude near-perfect.Ordered-comparison exclusions also challenge strong models, and Claude leads significantly.Source: §VI · pp. 7–8
• The failures are more consistent with an activation/application gap than a knowledge gap because basic forms are handled perfectly and rule access recovers errors.The conclusion frames this as a diagnosis rather than a definitive separation of alternatives.Source: §VI · pp. 7–8
• Tool delegation reaches approximately 100%, supporting the recommendation that coding agents delegate version-constraint resolution instead of reasoning in-head.The resolver is free and 100%-correct for this task.Source: §VI · p. 8