Source-linked AI summary
What Does an Agentic Software Engineering Benchmark Measure? Profiling Task Demands and Agent Behaviour Beyond What Category Labels Reveal
Radin Shayanfar, Keheliya Gallaba, Ahmed E. Hassan
TL;DR
Nominal benchmark labels conceal differences in the engineering work demanded by repository-level coding tasks. The paper introduces the SNC profile and finds that task demands correlate with success across families and scales, while behavioural signatures differ by family.
Problem
Nominal labels and aggregate statistics do not adequately characterize the engineering work demanded by agentic software engineering benchmarks.
Method
The paper applies the Spread–Novelty–Centrality (SNC) profile to five benchmarks and the runs of six agent configurations.
Results
Benchmarks sharing a label demand measurably different work; task correlates of success hold across families and scales, while behavioural signatures are family-specific.
Takeaways & Limitations
Agent behaviour exposes task demands that gold patches cannot, with Claude resolving at gold-patch parity, Qwen over-producing, and under-editing marking failure for both.
Takeaways & Limitations
All five benchmarks are Python-only and cover only issue resolution and feature implementation, limiting direct extension to other languages and task types.
Abstract
from arXiv · showhide
Agentic software engineering benchmarks are typically summarized by nominal category labels such as "bug fix" or "feature implementation," yet benchmarks carrying the same label are built through very different curation pipelines. A label thus reveals little about the engineering work a benchmark demands. We introduce the Spread--Novelty--Centrality (SNC) profile, a three-axis characterization of the demands of repository-level coding tasks, grounded in empirical software engineering research. We apply the profile to five widely used benchmarks and 14,922 trajectories of two model families at three scales, and report three findings. (1) A label is an unreliable proxy for task demands, as every pair of benchmarks is statistically separated on at least two SNC axes, and the separations trace back to specific curation decisions. (2) Agent behaviour reveals demands that the human-written gold solution cannot. Agents produce larger solutions than the gold where problem statements withhold hints and smaller ones where curation inflates the gold. How a task is phrased shapes what an agent produces. (3) Task demands correlate with success uniformly, with resolved runs concentrating in the low-SNC region for every family and scale, whereas the behavioural signatures of success are family-specific. Claude succeeds by matching the scope of the gold solution, and its parity share on files rises from $0.17$ at the smallest scale to $0.54$ at the largest. Qwen succeeds by exceeding the gold scope at every scale, and editing too little marks failure for both families.
1 Introduction
The paper argues that nominal benchmark labels conceal meaningful differences in repository-level engineering demands. It introduces the SNC profile and studies agent behaviour and resolution to expose those differences.
- Motivation: Benchmark labels conceal differences in change scope and required engineering knowledge because curation pipelines make different decisions.Aggregate lengths describe surface and running cost, not which benchmark suits an evaluation goal.
- Approach: The SNC profile measures task demands through Spread, Novelty, and Centrality, computed over each task’s gold patch.The dimensions capture distribution across the codebase, new versus removed code, and architectural significance.
- Findings: Five benchmarks occupy distinct SNC regions, including three feature-implementation benchmarks that share a stated goal but impose different engineering demands.This shows that category agreement does not imply task-demand agreement.
- Approach: Agent trajectories complement the gold patch because agents may resolve the same task through different files, line counts, or restructuring choices.The study therefore compares agent patches and trajectories with gold-patch scope and exploration.
- Findings: Resolved runs concentrate in low-SNC bins across families and scales, while behavioural success signatures differ between Claude and Qwen.Claude resolves near gold-patch parity, whereas Qwen resolves by over-producing and under-editing marks failure for both.
2 Background and Related Work
Agentic software-engineering benchmarks commonly present a natural-language task and repository, but their task sources and curation pipelines differ substantially. Category labels can obscure these differences, motivating a demand-oriented comparison.
- Benchmark landscape: Most agentic software-engineering benchmarks give an agent a natural-language problem statement and repository, requiring a tested code change.Tasks range from repairing localized defects to implementing features across files and modules.
- Issue-resolution benchmarks: SWE-bench selects GitHub issues whose merged pull requests introduce tests that change from failing to passing, while SWE-bench Verified screens for specification and test reliability.SWE-Gym uses the same formulation but draws from repositories disjoint from SWE-bench and serves as a training environment.
- Feature-implementation benchmarks: FEA-Bench supplies feature signatures and docstrings as hints, FeatBench uses hint-free statements for rewrites of existing functions, and FeatureBench carves tested source lines into targets.These construction choices create different task surfaces despite their shared feature-implementation framing.
- Taxonomy and limitations: A Conventional Commits taxonomy broadly matches benchmark scopes, but FeatBench and FeatureBench have near-identical label profiles despite very different construction pipelines.The contrast motivates measuring benchmark differences with the SNC profile rather than labels alone.
- Benchmark validity: Prior work raises concerns about contamination, realism of issue descriptions, weak test suites, and randomness in agentic evaluations.These concerns frame benchmark validity as a broader research issue.
3 Study Design
The study evaluates five benchmarks, two model families at three scales, and their agent trajectories under a unified harness. Its design supports comparisons of task profiles, agent footprints, and resolution outcomes.
- Study overview: Figure 1 organizes the study around SNC task demands, agent behavioural footprints relative to gold patches, and resolved-versus-unresolved comparisons.These correspond to RQ1, RQ2, and RQ3.
- Benchmarks: The study covers five benchmarks totalling 2,487 instances, including SWE-bench Verified and SWE-Gym Lite.SWE-Gym is included because the SNC profile applies regardless of whether a benchmark is intended for training or evaluation.
- Agents and models: Two model families are evaluated at three scales each: Claude under Claude Code and Qwen under Qwen Code.The scale points support cross-family comparisons of behaviour and resolution.
- Evaluation: All runs use Harbor as a unified evaluation harness, with FEA-Bench converted because Harbor does not support it natively.The full study comprises 30 runs and 14,922 agent trajectories using default decoding parameters.
4 RQ1: Do Benchmarks with the Same Nominal Label Demand the Same Kind of Engineering Work?
The SNC profile shows that benchmarks sharing a nominal label demand measurably different engineering work, with separations linked to distinct curation choices.
- The SNC profile characterizes engineering demands along Spread, Novelty, and Centrality, computed from benchmark gold patches.Spread measures distribution across the codebase, Novelty balances additions against removals, and Centrality captures architectural significance.
- Spread combines entropy-based volume concentration with directory-tree radius, while Centrality combines dependency position, churn, and callable complexity.The complementary Spread indicators capture how evenly changes are distributed and how structurally distant touched files are.
- Every pair of the five benchmarks is separated on at least two SNC axes, so shared nominal labels do not imply equivalent engineering demands.The two fix-dominated benchmarks separate on every axis, while the three feature-implementation benchmarks never all share a cluster.
- FeatBench’s testability filter compresses Novelty by selecting in-place rewrites without additions or deletions, placing it near SWE-Gym Lite.Its Novelty cluster lies well below those of the other two feature-implementation benchmarks.
- FeatureBench’s test-first carve-out saturates Novelty near 1.0 by adding back feature-related lines after removing them from the selected changes.This construction displaces its tasks from natural feature work, which typically mixes additions, edits, and removals.
- FEA-Bench has the lowest Centrality among feature-implementation benchmarks because newly added components lack incoming imports, history, and accumulated complexity.Its selection process biases tasks toward low FanIn, low churn, and modest Mass.
5 RQ2: What Does a State-of-the-Art Agent’s Behaviour Reveal About the Demands of Each Benchmark?
Comparing Claude’s resolved patches and trajectories with gold patches reveals benchmark-specific verbosity but broadly uniform exploration breadth. Hint-free wording prompts over-production, while inflated gold patches prompt under-production.
- Patch verbosity: Claude’s successful patches are most verbose relative to gold on FeatBench, most compact on FeatureBench, and near parity on the other benchmarks.The comparison holds across files, lines, and callables.
- Patch verbosity: FeatBench’s hint-free statements and unpinned target surface lead Claude to produce larger patches than the gold across all three verbosity metrics.
- Patch verbosity: FeatureBench’s compact agent patches suggest curation inflates the gold, with 33.8% of gold-patch lines consisting of comments or docstrings.The agent can omit those lines while still resolving the task.
- Patch verbosity: Hint-rich FEA-Bench keeps solutions near gold parity, contrasting with hint-free FeatBench and showing that problem wording shapes produced solutions.
- Exploration breadth: All five benchmarks fall into one Scott-Knott cluster on exploration breadth, indicating broad reading followed by narrow editing is primarily an agent property.FeatureBench ranks highest descriptively because blanked stubs force wider code surveys.
6 RQ3: What Separates Resolved from Unresolved Runs Across Model Families and Scales?
Across six configurations, resolved runs concentrate in low-SNC regions for both model families and all scales, while successful behavioural footprints differ by family. Claude moves toward gold-scope parity with scale, whereas Qwen consistently over-produces; under-editing marks failure for both.
- SNC correlates of resolution: The SNC differences between resolved and unresolved runs are statistically significant across all models and dimensions except Qwen S and M on ρexplore.
- SNC correlates of resolution: Resolved runs concentrate in low bins for nearly every SNC indicator across both families and all scales, while unresolved runs shift toward high bins.Novelty is the exception: unresolved runs are more concentrated in the mid-range associated with in-place rewrites.
- Behavioural signatures: Claude’s resolved runs peak at parity with the gold patch, whereas Qwen’s resolved runs concentrate above 2× on files, with parity near 0.2 across scales.Under-editing marks failure for both families.
- Behavioural signatures: Claude’s resolved parity share on files rises from 0.17 at S to 0.41 at M and 0.54 at L, while its exploration parity share rises from 0.09 to 0.34.
- Behavioural signatures: Qwen shows no comparable scale shift: resolved runs remain above 2× on verbosity and exploration at every scale, with exploration near 0.8 throughout.Scale makes Claude more targeted but leaves Qwen’s strategy fixed.
7 Implications
The SNC profile helps distinguish benchmark demands, while agent-specific behavioural signatures inform model, tool, benchmark, and engineering decisions.
- Model trainers: Training-corpus SNC profiles reveal which task demands are covered beyond labels such as bug fix or feature implementation.A benchmark score confined to a narrow SNC region is evidence about that region alone.
- Tool and harness builders: Configurable scope guidance should replace fixed minimal-diff prompts because Claude resolves at gold-patch parity whereas Qwen over-produces.The appropriate scope policy also shifts with model size.
- Tool and harness builders: Runtime checks can flag patches far below expected scope and route them to a second pass or stronger model.Under-editing marks failure across every family and scale.
- Benchmark authors: Publishing per-instance SNC profiles and excluded regions would let users subset benchmarks and compare demands rather than labels.High-Spread, high-Centrality work is underrepresented and unresolved runs concentrate in its high bins.
- Software engineers: Engineers can delegate low-SNC changes to smaller agents and reserve high-Centrality work for stronger models with closer oversight.Qwen’s overproduced patches require more review effort than Claude’s gold-scope patches, even when both resolve tasks.
8 Conclusion
The paper introduces SNC profiling and behavioural footprints to characterize repository-level benchmark demands and agent behaviour beyond nominal labels and gold patches. Across benchmarks and agent configurations, task correlates of resolution are consistent, behavioural signatures are family-specific, and the study is limited to Python issue-resolution and feature-implementation tasks.
- Conclusion: SNC profiling across five benchmarks and six agent configurations shows that shared labels conceal measurably different work linked to curation decisions.The profile characterizes tasks along Spread, Novelty, and Centrality.
- Conclusion: Claude resolves at gold-patch parity and tightens with scale, whereas Qwen over-produces at every scale; under-editing marks failure for both families.Task correlates of resolution remain largely invariant across families and sizes, but behavioural signatures differ.
- Limitations: All five benchmarks are Python-only, and several SNC indicators depend on language-aware analysis.Extending the profile to other languages therefore requires per-language tooling; the studied tasks cover only issue resolution and feature implementation.
A Appendix
The appendix documents the studied benchmark set and visualizes SNC distributions and agent behavioural footprints, with captions defining each display’s encoding and comparison basis.
- A Appendix: Figure 6 radar plots show seven SNC indicators per benchmark, with faint lines for instances and bold polygons for medians.Compare the bold median polygons across the five benchmarks.
- A Appendix: Figure 7 plots per-instance log10 ratios with Scott-Knott ESD clusters, using 0 as parity between agent and gold-patch measures.Panels (a)–(c) compare resolved patches with gold patches, while panel (d) compares trajectories with final patches and places all benchmarks in one cluster.
- A Appendix: Figure 8 conditions resolved-versus-unresolved gold-patch SNC distributions on outcome across indicator quantile bins and model scales.Columns represent SNC indicators and rows represent scales.
- A Appendix: Table 2 reports corrected χ2-test p-values and Cramér’s V for resolved-versus-unresolved separation per model and feature.Benjamini–Hochberg FDR correction is applied within each sub-table.
- A Appendix: Figure 9 conditions resolved-versus-unresolved behavioural footprints on outcome across five multiplicative bands relative to gold-patch parity.The shaded parity band spans 0.8×–1.25×.