PaperlayerView exact revision on arXiv

Source-linked AI summary

Agentic Rubrics as Contextual Verifiers for SWE Agents

Mohit Raghavendra, Anisha Gunjal, Bing Liu, Yunzhong He

arXiv:2601.04171v1cs.LG

TL;DR

SWE-agent verification must be scalable while remaining grounded and interpretable, but execution-based methods incur setup costs and simpler execution-free methods can miss repository context. The paper introduces Agentic Rubrics, in which an expert agent gathers repository context to create weighted criteria for execution-free patch scoring. On SWE-Bench Verified, the method reaches 54.2% with Qwen3-Coder-30B-A3B and 40.6% with Qwen3-32B, while rubric scores align with tests and expose issues tests may miss.

  • Problem

    Execution-based SWE verification is costly to scale, while execution-free alternatives can be less reliable, interpretable, and codebase-specific.

  • Method

    An expert rubric agent explores the repository to create context-grounded weighted criteria, which then score candidate patches without executing code.

  • Results

    54.2% BEST@16 is achieved on Qwen3-Coder-30B-A3B and 40.6% on Qwen3-32B, with Agentic Rubrics outperforming strong verifier baselines.

  • Takeaways & Limitations

    Agentic Rubrics provide an interpretable, codebase-grounded verification signal that aligns with ground-truth tests and surfaces failure modes tests may not capture.

  • Takeaways & Limitations

    Applying rubric signals to post-training remains future work because reward hacking, policy nonstationarity, and credit assignment introduce additional challenges.

Abstract

from arXiv · show

Verification is critical for improving agents: it provides the reward signal for Reinforcement Learning and enables inference-time gains through Test-Time Scaling (TTS). Despite its importance, verification in software engineering (SWE) agent settings often relies on code execution, which can be difficult to scale due to environment setup overhead. Scalable alternatives such as patch classifiers and heuristic methods exist, but they are less grounded in codebase context and harder to interpret. To this end, we explore Agentic Rubrics: an expert agent interacts with the repository to create a context-grounded rubric checklist, and candidate patches are then scored against it without requiring test execution. On SWE-Bench Verified under parallel TTS evaluation, Agentic Rubrics achieve a score of 54.2% on Qwen3-Coder-30B-A3B and 40.6% on Qwen3-32B, with at least a +3.5 percentage-point gain over the strongest baseline in our comparison set. We further analyze rubric behavior, showing that rubric scores are consistent with ground-truth tests while also flagging issues that tests do not capture. Our ablations show that agentic context gathering is essential for producing codebase-specific, unambiguous criteria. Together, these results suggest that Agentic Rubrics provide an efficient, scalable, and granular verification signal for SWE agents.

1. Introduction

Verification is a scaling bottleneck for SWE agents because execution-based methods are costly and execution-free methods can lack context and interpretability. Agentic Rubrics address this by grounding execution-free scoring in repository context and outperforming strong baselines.

  • Motivation: Code-execution verifiers are environment-aware but costly to scale and can produce sparse or brittle signals.Execution-free alternatives are lighter operationally but may be less reliable, interpretable, and codebase-specific.
  • Approach: Agentic Rubrics have an expert agent inspect the repository before generating criteria for scoring candidate patches without code execution.The criteria are grounded in relevant code paths, interfaces, and project conventions.
  • Contribution: Agentic Rubrics are presented as a repository-grounded, execution-free paradigm for patch selection and post-training.The approach decomposes patch quality into interpretable criteria and supports granular feedback.
  • Contribution: Agentic Rubrics consistently outperform strong test-based and execution-free verifier baselines under parallel test-time scaling on SWE-Bench Verified.The paper also analyzes alignment with ground-truth tests and diagnostic concerns that tests may miss.

2. Preliminaries

SWE verification methods assign scores to candidate patches for selection or learning, either by executing code or by using execution-free signals. Rubric-based verification makes scoring explicit and interpretable, while repository grounding improves precision and consistency.

  • Verification: A verifier assigns a score to a candidate patch to select or train toward higher-quality solutions.Execution-based methods run tests, whereas execution-free methods use classifiers, similarity metrics, or LLM judges.
  • Rubric-based Verification: A rubric decomposes patch correctness into explicit weighted criteria whose judgments aggregate into an overall verifier score.Criteria may be grouped by axes, and scores can be binary or graded.
  • Rubric-based Verification: Repository grounding addresses missing interfaces, constraints, and conventions that make problem-statement-only criteria less precise and consistent.The goal is lightweight scoring after task-relevant context has been incorporated into the criteria.

3. Experimental Design

The experiments generate repository-grounded verification artifacts, score 16 independent candidate rollouts, and evaluate verifier-based selection with BEST@K. Comparisons span non-agentic and agentic verifiers, including tests, patch similarity, classifiers, and rubrics.

  • Rubric Generation: The rubric-generation agent explores repositories and produces structured rubrics.yaml files for verification.The workflow uses repository navigation, file inspection or editing, and shell tools to gather task-relevant context.
  • Agentic Rubrics: Rubric items use weights 1, 2, or 3 across File Change, Spec Alignment, Integrity, and Runtime axes.The axes cover minimal edits, issue requirements, hygiene constraints, and intended runtime behavior.
  • Rubric Grading: The verifier aggregates weighted binary criterion scores as S = ∑i w_i s_i / ∑i w_i to rerank candidate patches.The resulting score lies in [0, 1].
  • Experimental Setup: Each problem uses K = 16 independent rollouts whose candidate patches receive verifier scores and are reranked for selection.The study uses Qwen3-32B and Qwen3-Coder-30B-A3B on 500 SWE-Bench Verified problems.
  • Evaluation: BEST@K resolution selects the highest-scoring candidate, with ground-truth tests defining whether the selected patch resolves the problem.ORACLE PASS@K and RANDOM@K provide upper-bound and uniform-selection reference points.
  • Evaluation: Figure 2 reports Best@16 resolution and Best@K scaling, using Claude Sonnet-4.5 for artifact generation and GPT-5 for LLM judging.The scaling curves average numbers over 100 trials.
  • Baselines: The comparison includes direct-scoring non-agentic verifiers and artifact-based agentic verifiers that first interact with the repository.Artifacts include tests, proxy patches, and structured rubrics.
  • Baselines: Agentic Tests execute generated tests, Agentic Patch Similarity compares candidates with a generated proxy patch, and Agentic Rubrics grade explicit criteria.These methods represent distinct artifact-based verification strategies.

4. Results

Agentic Rubrics improve patch selection over test-based and execution-free baselines while providing interpretable, graded signals that generally align with tests and expose under-tested issues.

  • Test-Time Scaling: 54.2% BEST@16 on Qwen3-Coder-30B-A3B and 40.6% on Qwen3-32B make Agentic Rubrics the top-performing verifier in both settings.On Qwen3-32B, this is +3.5 points over Patch Classifier; on Qwen3-Coder-30B-A3B, it is +4.0 points over the best non-agentic baseline.
  • Test-Time Scaling: Rubric-based scoring maintains its advantage as K increases, rather than concentrating its gain at a single operating point.The scaling curve is reported for Qwen3-32B.
  • Verifier Comparison: Repository-grounded rubrics avoid brittle runnable-test and proxy-patch steps by scoring candidates execution-free against file change, specification, integrity, and runtime criteria.Proxy-patch similarity can under-rank semantically correct fixes that differ stylistically.
  • Rubric Score Alignment: Rubric scores distinguish ground-truth passing from failing patches, with GT-pass scores typically at 0.85–1.0 and ROC-AUC 0.886 against test Pass/Fail.GT-failing scores are lower on average and often around 0.4–0.5, producing a graded rather than binary signal.
  • Rubric Score Alignment: Rubrics identify concrete quality dimensions beyond test outcomes, including unnecessary edits, missed requirements, runtime issues, and integrity preservation.For GT-passing patches, specification alignment and integrity are near-saturated, but over-scoped edits and occasional runtime issues still receive penalties.
  • Rubric Utility: 78% of rubric judgments are high-utility when rubric and test outcomes agree, while 54% of rubric rejections of test-passing patches flag plausible missed root causes or missing edges.The remaining cases include low-signal, over-specified, redundant, or rubric–test-mismatched judgments.

5. Ablations

The ablations examine rubric-agent capability, open-weight fine-tuning, repository grounding, and judge capability. Results show that stronger rubric generators and repository interaction improve selection, while rubric generation is a stronger objective than binary patch classification.

  • Rubric-Agent Model Choice: 54% BEST@16 resolution is achieved by Claude Opus-4.5, Claude Sonnet-4.5, and Gemini-3-Pro rubrics on SWE-Bench Verified.Qwen3-Coder-30B-A3B and Code World Model reach approximately 45%, while Qwen3-32B reaches around 43%.
  • Rubric-Agent Model Choice: More capable rubric models generally generate more rubrics per instance, enabling finer-grained differentiation between candidate solutions.Sonnet-4.5 averages over 20 rubrics per instance, twice the count of Qwen3-32B and Code World Model.
  • Training Open-Weight Rubric Agents: Agentic rubric generation substantially outperforms patch-classifier verification and non-finetuned base models.The result indicates that structured, context-grounded rubric generation is trainable and a stronger objective than binary classification for execution-free verification.
  • Impact of Repository Grounding: Repository interaction improves rubric quality and downstream selection by grounding criteria in concrete files, classes, and methods.Removing repository access reduces BEST@16 by 4.0 points on Qwen3-32B rollouts and 1.4 points on Qwen3-Coder-30B-A3B rollouts.
  • Judge Model Capability: Judge-model capability has a small but non-trivial effect on rubric-grading performance, while high reasoning effort is unnecessary because rubrics are self-contained and atomic.The analysis uses three increasing reasoning efforts for GPT-5 and also measures grading flakiness.

6. Related Work

Prior SWE-agent work uses test-time scaling and verifiers to improve coding-agent performance, while rubric research has established rubrics as evaluation and reward signals. This paper extends those ideas with context-aware rubrics for execution-free SWE verification.

  • Coding Agents and Test Time Scaling: Test-time scaling uses inference-time compute to improve performance on verifiable SWE-agent tasks.Prior verifier work commonly trains reward or scoring models such as patch classifiers or testing agents.
  • Rubrics as Verifiers for LLMs: Rubrics are widely used to evaluate LLM capabilities and as reward signals during reinforcement learning.This work studies context-aware rubrics as holistic verifiers for SWE candidates through test-time scaling.

7. Conclusion

The paper presents Agentic Rubrics as an execution-free, interpretable verification signal for SWE patches. They improve selection, align with tests and reference patches, expose test-missed failure modes, and motivate future integration into post-training.

  • Conclusion: Agentic Rubrics consistently outperform strong non-agentic and agentic baselines under parallel test-time scaling on SWE-Bench Verified.The verifier is context-grounded while avoiding test execution during scoring.
  • Conclusion: Rubrics provide interpretable feedback, align with human-written tests and reference patches, and surface failure modes unavailable tests may miss.The conclusion also identifies repository interaction, rubric-agent capability, and judge-model choice as key pipeline design choices.

8. Limitations

The paper’s limitations concern extending rubric verification beyond parallel test-time scaling and ensuring rubric quality. Future post-training use faces reward hacking, nonstationarity, credit assignment, and low-utility rubric modes.

  • Scope of Evaluation: Integrating rubric signals into post-training remains future work because reward hacking, nonstationarity, and credit assignment create additional challenges.The current study evaluates Agentic Rubrics in the parallel test-time scaling setting.
  • Rubric Quality: A subset of automatically generated rubrics has low utility through over-specification, redundancy, or rubric–test mismatches.The paper proposes human review, template reuse, and targeted prompts to improve rubric fidelity while preserving auditability.

A.1 Analyzing agentic rubric scores against Ground-Truth patch

Agentic Rubrics score human-written Ground-Truth patches highly, indicating alignment with high-quality fixes across rubric-generating models.

  • Table 3 reports average weighted rubric scores generated by different models over human-written Ground-Truth patches.
  • Frontier coding models produce Agentic Rubrics with higher alignment to human-written Ground-Truth patches than open-weight models.

A.2 Agentic abilities of rubric generation models

Rubric-generation ability varies substantially across models: frontier models reliably produce parseable YAML, while Qwen3-32B has weaker format compliance; finetuning improves its harness use and rubric distribution.

  • 97.8% and 96.8% of instances yield parseable YAML rubric files for Sonnet-4.5 and Gemini-3-Pro, respectively, with zero parse errors.
  • Qwen3-32B produces valid rubrics for only 74.6% of instances, reflecting weaker structured-output compliance.
  • Finetuning Qwen3-32B on Sonnet-4.5 rubric-agent trajectories reduces harness errors and improves rubric distributions to match the teacher model.

A.3 Cost analysis for agentic verification methods

Agentic Rubrics have the lowest reported BEST@16 per-instance cost among the compared verifier methods, while the analysis distinguishes fixed artifact costs from rollout-dependent grading costs.

  • Artifact cost is incurred once per instance, whereas grading cost is paid per rollout and scales with the number of rollouts.
  • $0.293 is the average total cost per instance for Rubrics in BEST@16, versus $0.515 for Test Generation and $0.736 for Patch Similarity.
  • Rubric generation requires fewer tool calls and tokens while achieving higher performance, according to the cost analysis.

A.4 Rubric Flakiness Study

The appendix examines rubric determinism, utility, model behavior, and illustrative failure cases, including how rubrics can reject test-passing patches for under-tested concerns.

  • A.4 Rubric Flakiness Study: In repeated judging, Sonnet-4.5 rubrics show 2% flakiness, while Qwen3-32B rubrics show 9% flakiness.A rubric item is flaky when any of five trial scores differs; atomic, self-contained instructions are associated with low flakiness.
  • A.4 Rubric Flakiness Study: High consistency supports reproducible assessments and reduces gaming opportunities, while future work should refine deterministic, non-prescriptive rubric-writing practices.
  • A.5 Hybrid verifiers using rubrics v/s classifier: A hybrid verifier combining agentic tests and rubrics can outperform either method in isolation.The appendix presents this as a simple aggregation setup and motivates more complex combinations.
  • A.8 Rubric and their grading - Illustrative Examples: In the matplotlib inset-axis example, a candidate passes Ground-Truth tests but receives a low rubric score because a dummy bounding box does not preserve meaningful layout behavior.
  • A.6 Categories of rubric utility classification: The appendix includes a taxonomy for rubric utility and figures concerning rubric distributions, parseability, finetuning, and hybrid verification.
  • A.9 Rubric Examples and grading: The example rubric checks renderer initialization, the full tight-layout render path, backward compatibility, and preserved inset-axis positioning and sizing.
  • Agentic Rubrics: Rubric construction emphasizes exact paths and symbols, standalone criteria, and literal YAML-friendly patterns rather than generic or cross-referenced items.
  • Rubric Utility Analysis Prompt: The utility-analysis prompt distinguishes accepted from failing rubrics according to whether they align with the reasons golden tests accept or reject a candidate patch.
Loading 2601.04171v1…