PaperlayerView exact revision on arXiv

Source-linked AI summary

AutoVerifier: Residual-Guided Non-Parametric Optimization for Reference-Based Answer Verification

Zebei Zhao, Zhihao Shi, Minqi Shi

arXiv:2608.25637v1cs.CL

TL;DR

Reference-based verifiers must decide equivalence when acceptability depends on implicit assumptions tied to the question and scoring criterion. AutoVerifier learns these biases from recurring verifier errors, validates proposed updates through replay, and promotes them into code or prompt guidance. Across four benchmarks, it reaches 93.05% macro accuracy, improves the prompt-only verifier by +1.76 points, and reduces fallback calls by 32.13%.

  • Problem

    Reference-based verification lacks an auditable way to learn implicit answer-equivalence assumptions that depend on the question and scoring criterion.

  • Method

    AutoVerifier clusters recurring verifier errors into rule cards and promotes validated deterministic patterns to code modules or model-dependent patterns to prompt guidance.

  • Results

    93.05% macro accuracy across four benchmarks exceeded reported verifier baselines, while code modules added +1.76 macro points over prompt-only verification and reduced fallback calls by 32.13%.

  • Takeaways & Limitations

    Accepted verifier updates remain auditable, editable, and reusable while learned comparison logic contributes beyond prompt rewriting.

  • Takeaways & Limitations

    The current scope is reference-based binary verification with a fixed construction pool and frozen inference interface.

Abstract

from arXiv · show

Reference-based verifiers are important for evaluating reasoning models and providing accurate outcome rewards in reinforcement learning with verifiable rewards. To improve verification accuracy, prior work has explored rule-based, model-based, and tool-augmented verifiers for checking answer equivalence across diverse answer forms. However, the equivalence of answer forms such as $1+3.14$ and $1+π$ may depend on the question and scoring criterion. We frame such implicit assumptions as verifier inductive biases. To address this challenge, we propose AutoVerifier, a residual-guided non-parametric optimization method that learns these biases from recurring verifier errors. Specifically, AutoVerifier records these biases in rule cards and promotes them to code modules or prompt guidance only after replay validation detects no direct regressions, keeping accepted updates auditable, editable, and reusable. Experiments on four verifier benchmarks demonstrate that AutoVerifier outperforms state-of-the-art verifiers by a large margin.

1 Introduction

Reference-based verification must handle answer equivalence whose validity depends on implicit assumptions about the question and scoring criterion. AutoVerifier learns these verifier inductive biases from recurring errors and promotes validated fixes into auditable code or guidance, improving benchmark performance.

  • Motivation: Answer equivalence can depend on the question and scoring criterion, including rounded probabilities, symbolic forms, and scientific notation with units.For example, 1 + 3.14 may differ from 1 + π depending on whether exact computation or applied numerical approximation is expected.
  • Motivation: Fixed rule-based and tool-augmented logic cannot automatically learn implicit assumptions, while model-based criteria remain embedded in model behavior rather than editable comparison logic.The unresolved need is to learn verifier inductive biases from evidence while keeping updates auditable, editable, and reusable.
  • Method: AutoVerifier groups recurring verifier errors, records biases in rule cards, and promotes supported patterns into code modules or prompt guidance after replay validation.Cards include supporting examples and counterexamples; deterministic logic becomes code, while model-dependent judgments guide the fallback.
  • Results: 93.05% macro accuracy was achieved across four verifier benchmarks, outperforming reported rule-based, model-based, scientific, and tool-augmented baselines.The benchmark experiments report the strongest result across the evaluated verifier categories.
  • Results: +1.76 macro points over the prompt-only setting and 32.13% fewer fallback calls show gains from learned comparison logic and direct decisions.The contribution summary and results passage attribute these improvements to the AutoVerifier construction process and code modules.

2 Related Work

Prior answer verifiers combine predefined rules, model judgments, and executable tools, but their coverage and maintainability remain limited. Non-parametric update research instead studies improvement through edits to external procedures, code, tests, memory, and evaluation records.

  • Answer verifiers: Rule libraries encode extraction, parsing, normalization, tolerances, symbolic checks, and abstention as inspectable procedures, but require manual coverage growth.Scientific verifiers add domain logic such as algebraic equivalence, unit conversion, and notation handling.
  • Answer verifiers: Model-based verifiers broaden coverage beyond fixed rules, while subtle answer equivalence remains failure prone.The passage frames these approaches as complementary but insufficient for all equivalence cases.
  • Non-parametric updates from experience: Learning beyond gradients improves systems by editing external state such as code, tests, memory, and evaluation records rather than model parameters.Related systems evolve reusable procedures, skills, tools, scaffolds, or code from experience.

3 Method

AutoVerifier learns recurring verifier biases from construction residuals, representing them as rule cards and conservatively promoting them to code modules or prompt guidance. Replay validation protects against direct regressions while routing abstained cases to a model-based fallback.

  • Frozen inference: At frozen inference, activated code modules make prioritized direct decisions, and the fallback verifier runs only when all code modules abstain.Later modules cannot override an earlier direct decision, and overlapping conflicts are replay-checked during construction.
  • Rule cards: Each rule card specifies an activation region, comparison procedure, abstention behavior, support examples, counterexamples, and replay evidence.These fields validate whether the proposed bias is appropriate before benchmark evaluation.
  • Update paths: Cards with deterministic comparison logic become code modules after code audit and replay validation, while judgment-dependent cards become prompt guidance for the fallback model.The two paths preserve reusable deterministic logic while retaining model flexibility for cases unsafe to encode as code.
  • Conservative validation: Replay validation accepts a code update only when support, counterexample, overlap, and full-replay checks pass without direct regressions.The promotion check permits expanded supported coverage while protecting construction examples from newly introduced direct errors.
  • Verifier state updates: The construction loop updates verifier state around a fixed model rather than changing model parameters.Accepted code updates modify the module library, while accepted guidance revises the fallback prompt.
  • Residual-guided construction: AutoVerifier replays the current verifier on construction data, accumulates residuals, and clusters recurring error patterns before drafting rule cards.Residuals are grouped by answer object, surface form, source domain, and observed error type.

4 Experiments

The experiments evaluate AutoVerifier under a frozen construction protocol across four verification benchmarks, comparing it with multiple verifier styles and isolating the contribution of learned code modules. Results show consistent benchmark gains, progressive rule coverage, direct corrections, and conservative handling of ambiguous cases.

  • Benchmark comparison: AutoVerifier achieves the best accuracy on all four benchmarks, improving over the best compared baseline by +0.15, +0.53, +0.22, and +0.10.The gains are reported for VerifyBench, VerifyBench-Hard, SCI-VerifyBench, and VerifierBench, respectively.
  • Benchmark comparison: 93.05% macro accuracy is reached across the four benchmarks, exceeding rule-based, model-based, scientific, and tool-augmented verifier baselines.The evaluation uses macro accuracy across benchmarks as its primary metric.
  • Controlled prompt comparison: +1.76 macro points are gained by adding code modules, compared with prompt rewriting alone, with positive gains on every benchmark.Prompt rewriting raises macro accuracy from 91.03 to 91.29, while Prompt + Code reaches 93.05; the gain also holds under micro accuracy.
  • Construction replay trace: Accepted construction rounds progressively expand direct coverage as recurring residuals become validated code modules.The stepwise increases on the construction monitor and four benchmarks support accumulated rule promotion rather than a single late update.
  • Accepted rule groups: 2,109 of 5,000 construction-monitor examples are covered by Round 10, with Rounds 1–5 accounting for 1,840 examples and Rounds 6–10 adding 269.Early rounds contribute most coverage, while later rounds add smaller, more specialized updates.
  • Direct Decision Attribution: 2,665 direct decisions reduce fallback calls by 32.13% and correct 149 prompt-only errors, with direct coverage spanning 26.65%–38.04% across benchmarks.Corrections are largest on SCI-VerifyBench, where 78 prompt errors are fixed.
  • Bias Categories and Failure Boundaries: Generic shortcut controls fail to reproduce the code-module gains, while accepted biases concentrate on recurring answer forms and guard conditions.The generic heuristic covers 502 examples but introduces 15 regressions; accepted groups contribute guarded portions of the improvement.
  • Bias Categories and Failure Boundaries: 53 of 80 remaining disagreement cases involve annotation or ambiguity issues, while 27 are remaining verifier errors.Code modules abstain on ambiguous or underspecified judgments and route those cases to the fallback verifier.

5 Conclusion

AutoVerifier converts recurring verifier errors into auditable comparison logic by promoting supported patterns into code modules or prompt guidance. Across four benchmarks, it reaches 93.05% macro accuracy, improves prompt-only verification by +1.76 points, and reduces fallback calls by 32.13% without changing model parameters.

  • 5 Conclusion: AutoVerifier converts recurring verifier errors into auditable comparison logic through code modules or prompt guidance.The method learns inductive biases from construction residuals and promotes supported patterns.
  • 5 Conclusion: 93.05% macro accuracy, a +1.76-point prompt-only improvement, and 32.13% fewer fallback calls are achieved across four benchmarks without changing model parameters.These are the reported final outcomes of the frozen verifier.

Limitations

The current scope is reference-based binary verification with a fixed construction pool and frozen inference interface.

  • Limitations: AutoVerifier is currently scoped to reference-based binary verification with a fixed construction pool and frozen inference interface.Future work proposes extending the construction loop to richer rubrics, multiple references, and domain-specific evaluation standards.

Ethical Considerations

AutoVerifier is intended for downstream filtering or training, where incorrect reference-based judgments can mislead data selection when references are incomplete or ambiguous. The paper also specifies conservative operational safeguards and responsible handling of construction and diagnostic data.

  • Ethical Considerations: Incorrect verifier outputs can mislead data selection when references are incomplete or ambiguous.This risk matters because AutoVerifier may be used as a reward component in downstream filtering or training pipelines.
  • Ethical Considerations: Deterministic rules retain abstention, and benchmark scores are separated from hard case analysis.These practices keep uncertain rule decisions and difficult diagnostic cases from being conflated with headline benchmark performance.
  • Ethical Considerations: Scientific and medical examples are verification cases rather than expert advice.The paper explicitly limits the interpretation of these examples and cautions against treating them as professional guidance.
  • Ethical Considerations: Diagnostic cases should be anonymized and distributed only under the relevant dataset licenses.Construction and benchmark data should likewise be used only under their licenses and permitted-use terms.
  • Ethical Considerations: The construction appendix documents implementation, audit, and additional evaluation details supporting the main text.Its sections include illustrative cases, construction records, and additional controls and diagnostics.

B Construction Implementation and Records

The construction implementation iteratively audits verifier behavior: it replays the current system, clusters recurring errors, drafts and audits rule cards, and promotes only validated updates.

  • B Construction Implementation and Records: Table 4 summarizes the construction-round pipeline and audit contract for each stage.The table is the compact reference for how construction proceeds from replay to validated promotion.
  • B Construction Implementation and Records: The pipeline initializes an empty module library and base prompt verifier before replaying the current verifier.This establishes the starting state for iterative construction.
  • B Construction Implementation and Records: Recurring verifier errors are clustered and converted into candidate rule cards for audit.The cards are evaluated against support, counterexample, overlap, and replay requirements.
  • B Construction Implementation and Records: Only candidates passing support, counterexample, overlap, and full replay validation are promoted.Existing modules may be repaired or merged, while new code modules are added only for uncovered deterministic behavior.
  • B Construction Implementation and Records: Cards requiring model judgment are incorporated into prompt guidance rather than deterministic code modules.The pipeline therefore separates deterministic comparison logic from behaviors needing model judgment.

B.2 Construction Data Details

The construction system uses a fixed, selected pool and promotes deterministic behaviors through support, counterexample, overlap, and replay checks. Its final records document broad rule coverage, clean support, and auditable module revisions.

  • B.2 Construction Data Details: Construction examples are normalized into a shared verifier schema and decontaminated against evaluation questions and references.WebInstruct-verified supplies questions and references, while candidate responses and binary labels are constructed for verifier training evidence.
  • B.2 Construction Data Details: Accepted updates require positive support, counterexamples, overlap checks, and clean construction replay before entering the frozen runtime.Candidates failing these checks remain rejected rather than being counted as accepted updates.
  • B.2 Construction Data Details: 5,000 examples form a selected construction pool combining source records with constructed diagnostic examples.The pool balances domain coverage with answer-object and error-type coverage rather than randomly sampling available records.
  • B.2 Construction Data Details: 2,109 examples are covered in the construction monitor: 2,096 live source supports plus 13 constructed boundary or format records.These are construction records over the 5,000-example pool, not a separate data-size convention.
  • B.2 Construction Data Details: 63 delivered code-module behaviors have live runtime support, with 0 missing support entries and 0 runtime errors.Supported groups include object extraction, verdict matching, numeric and unit normalization, mismatch detection, and invalid-answer guards.
  • B.2 Construction Data Details: The Round 8 identifier revision preserves case for mixed-case scientific identifiers and fires only when whole terminal identifiers are extracted from both sides.The revision fixes an unsafe compact surface match while retaining a narrowly scoped fail rule.
  • B.2 Construction Data Details: The construction–evaluation leakage audit finds zero decisive exact and fuzzy verifier-triple matches, with only low 5-gram surface overlap nonzero.Question–reference and reference-only repeats are reported separately because short answers and templates can repeat without copying full verifier examples.
  • B.2 Construction Data Details: All four code-module gains over the prompt component are positive under paired comparisons.The reported uncertainty analysis uses Wilson intervals and paired normal approximations over per-example correctness differences.

C.2 Additional Controls and Diagnostics

Additional controls attribute the main gains to learned code modules under fixed prompts and examine robustness across module groups and a public fallback model. The diagnostics also clarify that improvements are not claimed to be fallback-model invariant.

  • C.2 Additional Controls and Diagnostics: 93.05% four-benchmark average accuracy follows from adding code modules to the same prompt verifier, a +1.76-point improvement.Per-benchmark gains are positive, and the fixed prompt and fallback isolate the contribution of code modules from another prompt change.
  • C.2 Additional Controls and Diagnostics: Removal ablations replace one module group’s decisions with prompt verdicts while leaving other modules and prompt calls unchanged.This isolates each group’s marginal accuracy contribution rather than rewarding high firing frequency alone.
  • C.2 Additional Controls and Diagnostics: 2,665 direct decisions, 32.13% fewer fallback calls, and 149 corrected prompt errors result after all promoted code modules are enabled.Module groups contribute differently across benchmarks, with invalid-output guards especially relevant on SCI-VerifyBench and closed-choice mapping on VerifyBench-Hard and VerifierBench.
  • C.2 Additional Controls and Diagnostics: Generic frozen heuristics cover fewer examples, have nonzero direct errors, and slightly reduce final micro accuracy relative to the prompt verifier.This negative control contrasts recurring-error-induced modules with simple hand-designed matching rules.
  • C.2 Additional Controls and Diagnostics: 91.62% macro accuracy is achieved by the Qwen3-4B fallback versus 89.31% without the same code-module protocol, while 32.13% of fallback calls are avoided.The authors treat this as a fallback-replacement diagnostic, not evidence that the rule library is invariant across all fallback models.
  • C.2 Additional Controls and Diagnostics: The archived candidate ablation supports protected construction-slice replay, precision checks, counterexamples, and full replay for promotion.These checks are used to preserve precision rather than merely increase accepted updates.
  • C.2 Additional Controls and Diagnostics: 96.30%, 90.29%, 86.76%, and 90.77% are the official prompt baseline accuracies on VerifyBench, VerifyBench-Hard, SCI-VerifyBench, and VerifierBench.These values use the public benchmark prompt format with GPT-5.4-Mini and the same binary-label parser.

D.1 Manual Error Analysis

The manual analysis examines sampled disagreements after direct code-module decisions are removed, using author-assigned diagnostic categories rather than changing benchmark evaluation or verifier construction.

  • Authors inspect each case using the question, reference answer, candidate answer, benchmark label, and model verdict, without seeing rule reasons or construction history.
  • Cases are assigned to four categories: true verifier error, ambiguous or underspecified reference, multiple valid answers, or likely label/reference issue.
  • The accompanying report also tracks reference-sensitive disagreements, annotation assumptions, sparse reasoning failures, domain-external equivalence, and mixed or overlong responses.
  • The analysis samples 80 remaining disagreement cases, with 20 cases contributed by each benchmark.
  • The diagnostic counts and labels are not used for rule updates, prompt revision, rule promotion, benchmark relabeling, or changing benchmark scores.

D.3 Responsible Artifact and Checklist Notes

The artifact policy releases derived materials for auditing while restricting raw third-party data and screening shared text fields. The records are intended for verifier research and audit replication, not high-stakes deployment.

  • The supplementary package includes an artifact README, requirements file, prompt templates, anonymized synthetic examples, and alignment notes.
  • Released records should contain derived schemas, hashes, code modules, prompts, routing ledgers, and aggregate audit records rather than raw third-party examples unless redistribution is permitted.
  • Construction calls are separated from inference calls, and inference efficiency is reported as fallback-call reduction after the verifier is frozen.
  • Shared text fields are screened for personally identifying strings, contact information, author or local-path metadata, and unnecessary offensive content.
  • Flagged fields are removed, redacted, or replaced by hashes when raw text is unnecessary for recomputing a reported result.
  • The records support reference-based verifier research and audit replication, but are not intended for high-stakes grading, medical advice, or expert-system deployment.
Loading 2608.25637v1…