PaperlayerView exact revision on arXiv

Source-linked AI summary

Correct but Unselectable: The Hidden Interface Tax in Multi-Candidate Reasoning

Qianli Ma, Zhiqing Tang, Fanshuai Meng, Zhi Yao, Hanshuai Cui, Weijia Jia

arXiv:2610.32235v1cs.LG

TL;DR

Small-model generate-then-rerank systems can miss correct answers at selection even when those answers appear in the candidate pool. The paper formalizes this coverage–conversion gap and proposes CACC, which repairs candidate interfaces before fixed selection. CACC raises numeric-reasoning accuracy and improves GPQA Diamond final accuracy by 13.6 pp alone.

  • Problem

    Oracle coverage records whether a correct candidate appears, but does not distinguish an absent answer from one the fixed selector cannot consume.

    Source: §1 · p. 2

  • Method

    CACC applies offline-calibrated rules to repair fragments, align answer modes, and clean candidates before they reach the fixed selector.

    Source: §1 · p. 3, §4 · p. 5

  • Results

    +13.6 pp: CACC alone raises GPQA Diamond final accuracy from 5.6% to 19.2%, while CACC raises both oracle and final accuracy on numeric reasoning.

    Source: §1 · pp. 2–3

  • Takeaways & Limitations

    The coverage–conversion decomposition and selectable coverage distinguish raw candidate presence from whether a correct candidate can be selected.

    Source: §1 · p. 3, §7 · p. 10

    The study targets numeric and MCQ answer modes with benchmark-locked parsers, and its offline rulebook is manually calibrated per benchmark.

    Source: §6 · p. 10

Abstract

from arXiv · show

Small language models deployed under cost, latency, or privacy constraints cannot simply switch to larger models when reasoning fails. Multi-candidate generate-then-rerank offers a natural test-time scaling route. However, more candidates do not always yield higher accuracy. We identify a previously overlooked bottleneck between generation and selection: a correct answer may appear in the candidate pool yet fail to be selected because it is incomplete, unparseable, or misaligned with the answer mode the fixed selector expects. We term this loss the hidden interface tax and formalize a coverage-conversion decomposition that distinguishes raw oracle coverage from selectable coverage, the subset of problems where a correct candidate can actually be consumed by the selector. We propose Compatibility-Aware Candidate Construction (CACC), an interface layer between the candidate source and the fixed selector that repairs fragments, aligns answer modes, and removes selector-confusing noise without retraining either component. On numeric reasoning (GSM8K, competition_math), CACC raises both oracle coverage and final accuracy. On GPQA Diamond, CACC alone improves final accuracy from 5.6% to 19.2% (+13.6 pp); combined with proposer-side strengthening it reaches 25.8% (+20.2 pp). On MMLU-Pro, CACC exposes a coverage-conversion gap: answer-mode-compatible construction raises oracle coverage, but final accuracy decreases by 0.85 pp under the original selector; adding proposer-side strengthening improves final accuracy.

1 Introduction

Multi-candidate reasoning can fail even when a correct answer appears, because the fixed selector may not be able to consume it. The paper separates candidate coverage from conversion and proposes CACC to repair the interface without changing the generator or selector.

1 Introduction

  • More candidates may be incomplete, unparseable, or in the wrong answer mode, while oracle coverage alone cannot distinguish absent correct answers from present but unusable ones.This distinction helps diagnose whether to add candidates or fix interface alignment.Source: §1 · p. 2
  • +13.6 pp: CACC alone raises GPQA Diamond final accuracy from 5.6% to 19.2%.Source: §1 · p. 2
  • Selectable coverage distinguishes problems with a correct candidate in the pool from those where the fixed selector can consume and select it.Source: §1 · p. 3
  • CACC applies fragment repair, answer-mode alignment, and candidate hygiene between the candidate source and the unchanged selector.Its offline stage freezes rules from benchmark schemas, parser contracts, and audited failures; online construction uses those rules.Source: §1 · p. 3
  • On numeric reasoning, CACC raises both oracle and final accuracy; on MCQ, CACC with proposer-side strengthening reaches +20.2 pp on GPQA Diamond and narrows the MMLU-Pro coverage–conversion gap.Source: §1 · p. 3

2 Related Work

Prior work expands candidate sources, refines trajectories, or improves selection, but the paper distinguishes these from repairing the interface between existing candidates and a fixed selector. It also identifies candidate-interface evaluation as a gap in the benchmark and model ecosystem it reviews.

2 Related Work

  • Source-expansion methods raise the chance that a correct trajectory appears, but do not ensure candidates are complete, parseable, or selectable by a fixed verifier.Distribution shaping can raise raw oracle coverage yet hurt final accuracy when misaligned with the verifier.Source: §2 · p. 3
  • Trajectory-refinement methods alter generation or model state, whereas CACC repairs and cleans already-generated candidates before fixed selection.Source: §2 · p. 3
  • Verifiers and rerankers select from a pool, but format, duplication, answer mode, and wrappers can still alter their selection behavior.CACC changes the selector’s input distribution without replacing the verifier.Source: §2 · p. 3
  • The reviewed benchmarks and models lack explicit candidate-interface evaluation, the gap CACC addresses.Source: §2 · pp. 3–4

3 Problem Formulation

The section frames generate-then-rerank as a fixed-selector interface problem: a correct answer must not only appear, but be consumable at readout. It defines selectable coverage and separates missing answers from failures to select available ones.

3 Problem Formulation

  • The pipeline generates a raw candidate pool, constructs and retains candidates, then applies the unchanged selector and benchmark parser.The section asks whether the interface can turn a correct candidate's appearance into its selection under a fixed selector.Source: §3 · p. 4
  • Final accuracy separates into oracle coverage and conditional selector conversion, distinguishing absent correct answers from present answers the fixed selector fails to select.For an extractive selector, the section states this decomposition exactly.Source: §3 · p. 4
  • Selectable coverage requires a correct candidate that the fixed selector can consume, including benchmark parseability and answer-mode compatibility.The described answer modes include numeric spans and choice letters; selector-confusing residue and noise are also excluded.Source: §3 · pp. 4–5
  • When residual r is nonzero, the compatibility diagnostic may reject a candidate the selector would answer correctly, so h need not upper-bound raw oracle coverage.The section guarantees that upper-bound relation only when r = 0.Source: §3 · p. 5
  • In the MMLU-Pro example, a correct J candidate appears in the eight-candidate pool, but majority readout selects A.The example illustrates how oracle coverage can hold while final correctness fails.Source: §3 · p. 5

4 Method

CACC places a candidate-construction layer between generation and the fixed selector. It calibrates rules offline, then diagnoses and constructs candidates online without changing the selector.

4 Method

  • CACC freezes an offline diagnostic rulebook and routing policy, then repairs, aligns, and cleans candidates online before the fixed selector.The selector remains unchanged throughout this interface pipeline.Source: §4 · p. 5
  • Offline calibration uses schemas, parser configurations, candidate traces, scoring manifests, and audited cases to build a frozen rulebook and routing policy.Dev or audit labels may be used offline; test-set labels do not modify the rules online.Source: §4 · p. 5
  • Online inference computes gold-free diagnostics, routes candidates to repair actions, then deduplicates and parser-checks the retained pool before selection.The procedure retains the candidate cap and passes the resulting pool to the unchanged selector.Source: §4 · p. 6
  • CCR targets missing or incomplete answers, AMC targets answer-mode incompatibility, and SCH targets noise and duplication that can distract the selector.The three modules intervene on coverage, conversion, and selector-input distribution, respectively.Source: §4 · p. 6
  • The reference implementation uses deterministic rules for CCR, AMC, and SCH, while completion also calls the generator and requires rescoring.CACC keeps the retained cap fixed but does not match total inference cost.Source: §4 · p. 6

5 Experiments and Mechanistic Analysis

With the generator and selector held fixed, CACC improves numeric-task coverage and final accuracy, while MCQ results show that additional coverage does not always convert into higher final accuracy. Proposer-side strengthening and other controls provide evidence that outcomes depend on how candidate construction, answer-mode readout, and selection interact.

5 Experiments and Mechanistic Analysis

  • GSM8K oracle coverage rose from 0.4882 to 0.6391 and final accuracy from 0.2873 to 0.3632; competition_math:numeric oracle rose from 0.3210 to 0.4436 and final from 0.1519 to 0.2579.The results are reported for CACC against Base; the numeric block of Table 1 and Figure 3 Panels A–B show the comparisons.Source: §5 · p. 7, Table 1 · p. 8
  • The MMLU-Pro decomposition shows that oracle gains need not yield final gains: answer-mode alignment and proposer-side strengthening are still needed for stable conversion.Table 2 reports paired deltas across frozen audit rows; its MMLU-Pro CACC result is the key mechanism point.Source: Table 2 · p. 8, §5 · p. 8
  • On GPQA Diamond, CACC raises final accuracy from 0.0556 to 0.1919, and CACC+SP reaches 0.2576; on MMLU-Pro, CACC raises oracle to 0.4767 but final is 0.2277.The GPQA CACC-only gain is +13.6 percentage points; MMLU-Pro’s same-family CCR-only baseline has oracle/final of 0.4072/0.2362.Source: §5 · p. 8, Table 1 · p. 8
  • CACC+SP raises final from 0.2277 to 0.2936 on MMLU-Pro and from 0.1919 to 0.2576 on GPQA; conversion-aware rescue further raises selection-side conversion on both tasks.The reported rescue changes are 0.2936 → 0.3320 on MMLU-Pro and 0.2576 → 0.2929 on GPQA; the Power Sampling×CACC diagnostic shows that distribution shaping at the wrong layer can widen the gap.Source: §5 · p. 9

5.1 Additional Controls and Prospective Validation

  • MMLU-Pro majority voting converts limited newly added coverage and loses some previously available correct candidates; on ARC-Challenge, unchanged rules raise oracle coverage with little final-accuracy change.Qualitative audits link interface failures to fragments, answer-mode mismatch, and selector-confusing residue.Source: §5.1 · pp. 9–10
  • Additional controls report gains across GPQA selector comparisons and all five Mistral-generation seeds, while constrained decoding exceeds CACC by 6.57 pp on GPQA and 0.98 pp on MMLU confirm-512.The controls distinguish frozen-pool, target-verifier, Mistral, and fresh paired-generation settings.Source: §5.1 · p. 9, Table 4 · p. 10

6 Discussion

CACC’s frozen offline rulebook can compose with proposer strengthening or conversion-aware rescue without changing the verifier. The study’s scope remains limited to benchmark-locked numeric and MCQ answer modes, with effects depending on the selector, readout, and evaluation scope.

6 Discussion

  • A frozen offline rulebook composes with proposer-side strengthening or conversion-aware rescue without changing the verifier.The authors recommend an interface audit before investing in further sampling or feedback rounds.Source: §6 · p. 10
  • CACC’s effects depend on the selector, readout, and evaluation scope; the original MMLU-Pro setting illustrates this with an accuracy decline.The current study covers numeric and MCQ answer modes with benchmark-locked parsers; broader task types require generalizing the answer-mode registry.Source: §6 · p. 10

7 Conclusion

The paper identifies a hidden interface tax in generate-then-rerank reasoning and proposes CACC to address it. Numeric-task results and GPQA Diamond results show accuracy gains under a fixed verifier, with a larger reported gain when CACC is combined with proposer-side strengthening.

7 Conclusion

  • +20.2 pp on GPQA Diamond when CACC is composed with proposer-side strengthening.CACC alone yields +13.6 pp on GPQA Diamond.Source: §7 · p. 10
  • CACC raises both oracle and final accuracy on numeric reasoning under a fixed verifier.The conclusion describes fragment repair, answer-mode alignment, and candidate hygiene as the mechanisms used to close the gap.Source: §7 · p. 10
  • CACC targets correct candidates that appear in the pool but cannot be consumed by the fixed selector.The paper formalizes this with a coverage–conversion decomposition and selectable coverage as the operative upper bound.Source: §7 · p. 10

A Experimental Protocol, Data Scope, and Reproducibility

The experiments use a fixed generate-then-rerank protocol and document the model stack, inference operations, data scope, and records supporting reproducibility. The appendix also describes candidate-budget and same-pool verifier diagnostics, while delimiting the work to low-cost, local, or intranet small-model reasoning.

A Experimental Protocol, Data Scope, and Reproducibility

  • Comparisons hold the benchmark split, retained cap, parser, scoring endpoint, and downstream selector fixed apart from candidate construction.Numeric tasks use final numeric exact match, while MCQ tasks are hard-locked to choice_letter.Source: A Experimental Protocol, Data Scope,… · p. 15
  • The main generator and fixed selector are Qwen3-1.7B; CACC+SP uses a Qwen3-8B proposer, and FTTT is a high-compute external reference.The verifier was not trained on MMLU-Pro, GPQA, or any evaluation split used in the paper.Source: A Experimental Protocol, Data Scope,… · p. 15
  • Run manifests record dataset and split, filtering, candidate source, cap, parser, scoring, selector, seed, predictions, and generation scripts for traceability.The appendix identifies GSM8K, competition_math:numeric, MMLU-Pro, and GPQA evidence units and describes paired closure statistics.Source: A Experimental Protocol, Data Scope,… · pp. 15–16
  • CACC adds one batched generation of four completion sequences per example and rescoring of the modified candidate pool.The SP rows use a Qwen3-8B proposer.Source: A Experimental Protocol, Data Scope,… · p. 15
  • Table 5 reports sample size, answer mode, oracle, final, and V|O for the main evidence units.Source: Table 5 · p. 16
  • The application boundary is low-cost, local, or intranet small-model reasoning; the study does not involve personal-data decisions or high-risk automated deployment.Source: A Experimental Protocol, Data Scope,… · p. 16
  • The appendix includes GSM8K candidate-budget diagnostics and a same-pool verifier-sensitivity table.Their captions identify the diagnostic scopes but do not state comparative outcomes.Source: Table 7 · p. 16, Table 8 · p. 16

B Coverage Decomposition and Parser Provenance

The appendix details the MMLU-Pro coverage–conversion and parser-readout evidence, second-family recovery and layer interactions, and CACC’s offline rulebook flow. Together, these passages distinguish raw coverage from successful conversion while documenting recovery evidence and implementation scope.

B Coverage Decomposition and Parser Provenance

  • MMLU-Pro CACC raises oracle coverage, but answer-mode alignment and proposer-side strengthening are still needed to convert it stably into final gain.The appendix presents this as the key mechanism point in the decomposition.Source: B Coverage Decomposition and Parser… · p. 17
  • Under archived numeric scoring, legacy/no-AMC has oracle 444/12032 = 0.0369 and verifier 115/12032 = 0.0096; MCQ reparse gives 4900/12032 and 2842/12032.The reparse uses the same legacy/no-AMC candidate interface; the comparison is a scoring/readout-path ablation, not two independent pools.Source: B Coverage Decomposition and Parser… · p. 17, Table 9 · p. 17
  • MMLU-Pro and GPQA are hard-locked to choice_letter, making the legacy/no-AMC comparison a readout-path ablation.Source: B Coverage Decomposition and Parser… · p. 17

C Recovery, Composition, and Interaction Evidence

  • CACC occupies the candidate-interface layer, distinct from trajectory improvement, candidate-source expansion, reasoning-space expansion, feedback training, and selection.Table 3 reports interaction evidence; Table 11 distinguishes frozen statistical results from summary-only diagnostics.Source: C Recovery, Composition, and… · p. 17
  • FTTT’s archived GSM8K full log reports 1152/1319 = 0.8734 with an average of 6.67 test-time trials.It is presented as a high-compute external reference in the feedback/test-time-training regime.Source: C Recovery, Composition, and… · p. 17
  • Both second-family paired recovery closure intervals lie above zero, supporting recoverable compatibility.The recovery evidence is in Table 3 and Figure 4 Panel D; Figure 5 visualizes the paired bootstrap intervals.Source: C Recovery, Composition, and… · p. 17, Figure 5 · p. 18, Figure 4 · p. 18

D Offline Rulebook Components

  • CACC’s offline rulebook is frozen before online inference, when gold-free routing and candidate construction are executed; its method composition is CCR+AMC+SCH.The replacement policy is treated as a swappable component.Source: D Offline Rulebook Components · p. 20

E Interface Error Taxonomy and Rule Interfaces

The section groups candidate-interface failures into parser failure, answer-mode mismatch, and selector confusion. It also documents the interfaces and evidence artifacts used to make candidate construction and scoring reproducible.

E Interface Error Taxonomy and Rule Interfaces

F Notation Summary and Per-Benchmark Setup

The setup tables collect notation and benchmark-specific reproduction details, while the reporting conventions define the experimental unit and interpretation of verifier means.

F Notation Summary and Per-Benchmark Setup

G.1 Experimental units and reporting conventions

  • Accuracies are measured per question; three-seed verifier means are descriptive, not independent question samples.Original-pool controls reuse historical candidate contents, while fresh-generation controls use separate baselines.Source: §G.1 · p. 22

G.2 Dataset-specific verifier supervision

Across these studies, outcomes vary with verifier readout, candidate selection, and answer format: coverage and final accuracy do not always move together, and prospective ARC results are inconclusive.

G.2 Dataset-specific verifier supervision

  • GPQA canonical means were 16.16% and 24.92%, while MMLU held-out means were 26.40% and 27.84%; mean raw-readout gains were 9.93 pp and 0.37 pp, respectively.The study uses the same verifier within each target and seed; MMLU-Pro supervision is restricted to the official validation set, and GPQA supervision uses a cleaned 346-question set.Source: §G.2 · pp. 22–23, Table 20 · p. 23

G.3 Paired constrained decoding

  • The interaction on F was −7.07 pp on GPQA and −8.79 pp on MMLU confirm-512; direct CD-minus-CACC contrasts were +6.57 pp and +0.98 pp, respectively.The primary contrasts use Holm-adjusted p-values; the direct CD-minus-CACC contrasts are outside that primary family.Source: §G.3 · pp. 23–24, Table 21 · p. 23

G.4 Selector families

  • Candidate coverage was identical across selector families within each pool and scope, while final accuracy changed with selection.Strong-judge paired intervals were [+10.10, +23.23] pp on GPQA and [−2.73, +5.47] pp on MMLU confirm-512.Source: §G.4 · p. 24, Table 22 · p. 24

G.5 Five-seed generator study

  • Mean final accuracy rose from 14.22% to 15.94% in the five-seed MMLU confirm-512 generator study.Mean oracle coverage rose from 48.20% to 58.36%, with all five final-accuracy deltas positive.Source: §G.5 · p. 24

G.6 Format-only intervention and pool diagnostics

  • The answer-marker intervention shifted signed margins by −0.3899, while all-question accuracy changed by −0.065 pp.Only the final-answer marker changed; reasoning text and decoded answer content were fixed, isolating format sensitivity rather than aggregate accuracy gain.Source: §G.6 · p. 24
  • Of 1,620 newly covered examples in the full MMLU majority path, 261 converted; conversion on jointly covered examples fell from 57.31% to 47.67%.These comparisons describe recorded candidate pools, not a traced generation lineage.Source: §G.6 · p. 24

G.7 Prospective ARC-Challenge validation

  • On the prospective 256-question ARC-Challenge study, final accuracy changed by +0.78 pp, a difference that was not statistically significant.The schema and label wrapper were added without changing the operative payload.Source: §G.7 · p. 25, Table 23 · p. 25

G.8 Release provenance and adaptation measurement

  • The original MMLU Base pair is unavailable for a new paired test of the historical −0.85 pp contrast, and the framework provides no correctness guarantees.Some stronger-proposer rows use reconstructed reruns rather than the exact original candidate streams; application-specific validation remains necessary.Source: §G.8 · p. 25

Related papers

Loading 2610.32235v1…