PaperlayerView exact revision on arXiv

Source-linked AI summary

Code-Guided Reasoning for Small Language Models: Evaluating Executable MCQA Scaffolds

Prateek Biswas, Dhaval Patel, Vedant Khandelwal, Shuxin Lin, Amit Sheth

arXiv:2605.18827v1cs.IRcs.LGcs.PL

TL;DR

Direct MCQA scores do not capture small language models operating inside executable scaffolds that decompose questions, call models repeatedly, and select answers. CGR provides an auditable protocol and trace package for comparing these conditions, with 66.21% assisted versus 38.11% direct macro accuracy in the observed non-zero-baseline partition. The estimates remain descriptive because assisted inference has a larger call budget and the retained runs lack controls separating repeated calls, natural-language deliberation, and executable state.

  • Problem

    Direct MCQA benchmarks provide limited evidence about how small language models behave in deployed workflows using tools, code, and repeated model calls.

  • Method

    CGR evaluates the same solver directly and inside generated Python scaffolds, recording direct, assisted, generator-side, and diagnostic trace channels.

  • Results

    66.21% assisted macro accuracy versus 38.11% direct macro accuracy was observed in the non-zero-baseline partition.

  • Takeaways & Limitations

    CGR supplies the paired records and audits needed to interpret when executable assistance changes solver behavior across datasets and solver models.

  • Takeaways & Limitations

    The retained evaluation is descriptive rather than a controlled causal study, and assisted inference uses a larger solver-call budget than direct inference.

Abstract

from arXiv · show

Multiple-choice QA benchmarks usually evaluate small language models (SLMs) as direct answerers, but deployed language-model systems increasingly rely on external scaffolds such as tools, code, and repeated model calls. We introduce Code-Guided Reasoning (CGR), an evaluation protocol and generated-program resource for measuring when executable reasoning scaffolds improve SLM performance on MCQA tasks. CGR standardizes six components: a normalized item interface, a direct solver prompt, a generator prompt, a Python scaffold, solver-call and extraction helpers, and a three-channel result record. On 20,498 retained result rows from a locally prepared MCQA bundle and six metadata-registered solver models, the observed non-zero-baseline partition shows 66.21% macro assisted accuracy versus 38.11% direct accuracy, a +28.10 percentage-point difference with a pair-bootstrap interval of [20.32, 36.43]. Under a stricter Ab > 30% direct-signal gate, the macro difference is +14.11 points. These estimates are descriptive. Assisted inference uses a larger solver-call budget, answer extraction is brittle, Time-MQA contains the observed regressions, and some generated programs violate the no-hard-coding instruction. CGR provides the trace package needed to interpret these results, including direct, assisted, and generator-side answers, partition definitions, generated programs, response metadata, and audits.

1 Introduction

CGR evaluates small language models not only as direct MCQA answerers but also inside generated executable scaffolds. It reports separate answer channels and trace evidence to measure and audit this interface shift.

  • Motivation: Direct MCQA accuracy does not measure deployed workflows that decompose questions, make repeated model calls, run computations, and select an answer.Such workflows matter when small models are cheaper, easier to host locally, or subject to data and latency constraints.
  • Motivation: CGR asks whether the same small solver behaves differently when moved from direct option selection into a generated executable action space.The scaffold can store variables, branch, call the solver, extract letters, and use a tiebreaker.
  • Protocol: CGR scores direct solver, assisted solver, and generator-side answers separately to distinguish decomposition, format repair, extra calls, and generator knowledge.The target solver answers directly and also through a generated Python scaffold.
  • Validity choice: The primary comparison uses dataset–model pairs with at least one correct direct answer, while zero-baseline rows remain diagnostics rather than deployment evidence.This non-zero-baseline partition reduces ambiguity from prompt-format failure or option-extraction mismatch.
  • Resource: 20,498 retained result rows across nine dataset configurations and six solver labels form the CGR trace package.The package records generated programs, answer channels, response metadata, and source-provenance fields.
  • Result: 66.21% assisted macro accuracy versus 38.11% direct macro accuracy is the observed result for the non-zero-baseline partition.Audits accompany this comparison because assisted inference uses a larger call budget and can involve extraction failures, literal-answer patterns, and regressions.

2 Background and Related Work

CGR builds on work showing that measured reasoning depends on inference procedure, executable operations, and evaluation traces. It positions executable scaffolds as an auditable measurement condition rather than a new solver architecture.

  • Inference procedures: Chain-of-thought changes the requested reasoning trace, while self-consistency changes decoding and selection; both motivate describing the inference procedure.CGR applies this principle to executable MCQA scaffolds.
  • Executable reasoning: Program-of-Thoughts, PAL, and CodeAct show how generated code can provide computation, control flow, and inspectable state.CGR instead uses generated code as a controller that queries a separate solver model.
  • Scope: CGR does not maintain template memory, train a navigator, or optimize template trajectories; it evaluates freshly generated executable scaffolds.This distinguishes the protocol from reasoning-scaffold methods based on stored or scaled thought templates.
  • Auditing: Tool-use evaluations require external calls, execution rules, APIs, and failure modes to appear in the evaluation record.CGR therefore reports scaffold contracts, extraction behavior, call budgets, and possible answer leakage alongside accuracy.
  • Contribution: CGR contributes a measurement setting for executable assistance, not a new solver model or public source-question corpus.Its reported evidence includes outputs, generated code, provenance, and partition definitions.

3 Datasets

CGR evaluates a locally prepared, normalized bundle spanning multiple reasoning domains and records both dataset provenance and retained solver-result coverage. The package measures executable assistance around existing MCQA content rather than introducing a new question corpus.

  • Normalization: Each normalized source item has a common item id, question text, option list, option ids, and correctness flags.The retained analysis excludes solver results for the CureBenchPhase2QA configuration.
  • Dataset accounting: Table 1 separates source item counts from retained registered result rows because solver coverage differs across datasets and models.Its provenance column identifies source papers, official pages, or author-prepared local subsets.
  • Coverage: The bundle spans reasoning regimes including expert-reviewed multiple-choice knowledge, elementary science, graduate knowledge, temporal analysis, medicine, physics, and contest mathematics.The listed configurations include MMLU-Pro, OpenBookQA, SuperGPQA, Time-MQA, MedQA, PhysicsQA, and AIME.
  • Scope boundary: FailureSensorIQ targets Industry 4.0 reasoning over failure modes, sensor data, and relationships across industrial assets.Its results do not establish safety for operational diagnosis without expert validation.
  • Object of study: CGR defines the executable-assistance measurement setting around existing datasets rather than claiming a new public source-question corpus.The measurement package includes direct and assisted outputs, generator prompts, generated Python, extraction, metadata, and partition definitions.

4 Methodology

CGR compares a direct solver path with an item-specific generated Python scaffold and records solver-side, generator-side, and difficulty outputs separately. Its audits expose how execution and answer selection affect interpretation.

  • Evaluation flow: The direct path requests one option letter, while the assisted path executes a generated Python scaffold with a fixed three-value return contract.The contract stores the scaffold-selected solver answer, generator-side answer, and generator-estimated difficulty.
  • Formalization: Each evaluation unit combines a question, dataset configuration, target solver, and generator label before producing a generated program.The direct path observes the solver answer, while the assisted path executes the generated program.
  • Scoring: Correctness is evaluated separately for direct, assisted, and generator-side channels against the gold option.This preserves distinctions among solver behavior, scaffold selection, and generator-side judgments.
  • Scaffold execution: Generated programs may branch, compute, query the solver repeatedly, and select answers through agreement, verification, or tiebreaking logic.The executor provides solver-call and answer-extraction helpers, but programs are not assumed-correct explanations.
  • Audits: Assisted calls average 7.18 versus 1.01 direct calls, while the ten-call limit and no-hard-coding rule are prompt instructions rather than runtime guarantees.The analysis treats interface violations, extraction failures, and metadata coverage as audit findings.

5 Experimental Setup

The experiment filters metadata-registered solver outputs, analyzes dataset–solver coverage, and separates primary and diagnostic partitions with audit records. Accuracy is macro-averaged over dataset–solver pairs for partitioned claims, while uncertainty uses pair-level bootstrap intervals.

  • The analysis retains registered solver names and summarizes accuracy by dataset–solver pair, so aggregate rows reflect uneven evaluated coverage rather than balanced benchmark averages.
  • The retained solver roster combines six models, including four earlier local solvers, Gemma 4 E2B, and Nemotron-3-Nano-4B.Provider-specific model identifiers remain in local metadata and logs.
  • The roster is not a balanced architecture or parameter-size study, and complete immutable main-run configuration manifests were not retained.Notebook and response metadata support deterministic calls with 2000-token solver and 8192-token generator caps.
  • A dataset–solver pair enters the primary partition when it has at least one correct direct answer; otherwise it enters the zero-baseline diagnostic partition.Partitioned claims use macro accuracy over dataset–solver pairs so larger datasets do not dominate.
  • Uncertainty for partition-level macro quantities uses percentile bootstrap resampling over dataset–solver pairs.The intervals do not capture repeated-generation or repeated-rerun variation because only one retained result per item is available.
  • CGR reports response-metadata coverage, call imbalance, extraction failures, literal-answer scans, and threshold sensitivity alongside accuracy.These audits document assumptions for reading retained scores but do not establish artifact safety or causality.

6 Results

CGR reports higher assisted than direct accuracy in the primary non-zero-baseline partition, but the descriptive result is bounded by threshold sensitivity, unequal call budgets, channel separation, and dataset-specific regressions. Time-MQA provides the clearest negative boundary condition, while audits make these qualifications visible.

  • Primary results: 66.21% assisted macro accuracy versus 38.11% direct accuracy yields a +28.10-point difference in the observed non-zero-baseline partition.The comparison closes 64.7% of the generator gap and is descriptive rather than a matched-budget causal estimate.
  • Primary results: The all-item summary is micro-averaged, whereas partitioned rows are macro-averaged over dataset–solver pairs and answer different evaluation questions.The zero-baseline row isolates assisted successes despite no direct successes.
  • Validity checks: The pair-bootstrap interval for the observed non-zero-baseline improvement is [20.32, 36.43] percentage points.Dataset- and solver-cluster resampling remains positive but widens uncertainty because solver identity and dataset domain affect direct baselines.
  • Channel separation: Assisted and generator-side answers are not nested: 180 primary-partition rows favor assisted answers while 2,217 favor generator-side answers.Generator-side accuracy is therefore diagnostic and should not be collapsed into assisted-solver accuracy.
  • Validity checks: The stricter Ab > 30% gate retains a positive but smaller gain, indicating the headline is not a universal 28-point improvement.This comparison does not isolate executable structure from extra inference because matched-budget and no-code baselines are outside the experiment set.
  • Dataset patterns: The largest broad-partition gains occur for low-but-nonzero direct baselines, including MedQA, AIME, MMLU-Pro, and SuperGPQA.For example, MedQA Llama 3.2 11B rises from 1.20% to 84.57%.
  • Dataset patterns: Time-MQA contains both large gains for low-baseline solvers and regressions for three more capable direct solvers.The paper treats the regression observation as a boundary condition rather than establishing its proposed inconsistency mechanism.
  • Dataset patterns: Zero-baseline pairs reach 62.19% macro assisted accuracy but remain diagnostics rather than primary evidence of solver deployability.Several pairs show high assisted scores despite no correct direct answers.

7 Limitations

The evaluation is descriptive rather than a controlled causal study, and several design and implementation issues constrain how assisted gains should be interpreted.

  • Assisted inference uses more solver calls, so the observed gains are not equal-budget improvements in the underlying solver.The retained evaluation does not match inference budgets between direct and assisted conditions.
  • A matched-budget direct self-consistency baseline is needed to test whether repeated solver calls alone explain assisted gains.
  • The current results do not isolate executable structure from extra inference, natural-language deliberation, repeated no-code control, or generator-only answering.
  • Answer extraction is brittle, and generated-code validity is audited rather than enforced.

8 Conclusion

CGR measures what happens when the same small solver moves from direct option selection into an executable scaffold, while preserving traces needed to interpret the result. The retained evidence reports positive gains under selected partitions but also documents important boundaries and next-step controls.

  • 66.21% assisted versus 38.11% direct macro accuracy is observed on the primary non-zero-baseline partition, while the stricter Ab > 30% gate shows a +14.11-point gain.
  • CGR’s paired records connect direct, assisted, and generator-side answers with partition definitions, generated programs, metadata, and audits.
  • A positive score under the Ab > 30% gate is stronger evidence of assistance changing a solver with direct task signal, whereas zero-baseline gains are diagnostic.
  • The claim is bounded because assistance is not equal-budget, extraction is brittle, Time-MQA contains regressions, and some programs violate no-hard-coding.
  • Future versions should add enforced extraction, call-limit, sandbox, code-validation, repeated-sampling, and matched-budget controls.
  • The artifact is a trace package around existing MCQA datasets, with source-question redistribution constrained by upstream terms.

C Methodology Details

The methodology combines generated Python scaffolds with saved answer-channel records and execution metadata. An OpenBookQA example illustrates repeated solver calls, tiebreaking, and the need to separate generator-side answers.

  • The OpenBookQA scaffold asks the solver twice, extracts option letters, and uses a tiebreaker when calls disagree.
  • In the retained Granite 4H Small run, direct answered E while assisted and generator-side answers were A, matching the gold label.
  • The generated program returns both solverLLM_answer and genLLM_answer, keeping generator-side output as a separate diagnostic channel.
  • Direct-call metadata covers 20,490 of 20,498 rows and assisted-call metadata covers 20,492, with no joined generator code-generation metadata.
  • No-hard-coding is a design intent rather than enforcement, and static audit finds literal solverLLM_answer = "A"-style patterns in some programs.

D Experimental Setup Details

The experimental setup uses a non-zero-baseline and thresholded partition framework to compare direct, assisted, and generator-side accuracy. The study explicitly leaves several controls outside the experiment set.

  • The solver roster is not a balanced architecture sweep or parameter-size study, with solver calls requested at temperature 0.0 and a 2000-token cap.
  • The non-zero-baseline split is a minimum interpretability gate, excluding dataset–model pairs with no direct correct answers from the primary comparison.
  • Generator-gap closure is reported only as a descriptive diagnostic when its denominator is positive, not as an independently deployed baseline.
  • The experiment does not isolate executable structure from extra inference because matched-budget direct, chain-of-thought, no-code, and generator-only controls are absent.

E Additional Result Figures

The additional figures partition CGR results by evaluation slice, solver, dataset, and estimated difficulty. They emphasize that zero-baseline pairs are diagnostic and that Time-MQA contains regressions.

  • Three evaluation slices distinguish all retained records, observed non-zero-baseline pairs, and zero-baseline diagnostic pairs.
  • Dataset–solver matrices report assisted-minus-direct accuracy, marking zero-baseline cells as diagnostic rather than primary evidence.
  • Negative bars identify the Time-MQA regressions among observed non-zero-baseline dataset–solver pairs.
  • Macro profile plots average results across solver settings by dataset and across datasets by solver, with accuracy shown radially.
  • Difficulty figures pair generator-estimated difficulty with representative scaffold text, but the signal is not externally validated.

F Expanded Results Narrative

Expanded analyses show that assisted gains vary across datasets and solvers, with large improvements often appearing at low but nonzero direct baselines. The results remain descriptive because assisted inference uses more calls, generated-code validity is imperfect, and uncertainty depends strongly on solver identity.

  • Expanded results: The assisted solver closes 70.0% of the all-item generator gap and 64.7% in the observed non-zero-baseline macro partition.
  • Expanded results: Large gains occur on low-but-nonzero direct baselines, including Granite 8B Code’s +47.78 points on SuperGPQA and +45.78 points on MMLU-Pro.
  • Expanded results: Assistance also improves capable pairs, such as Gemma 4 E2B on MedQA from 52.91% to 91.58%.
  • Interpretation: The stricter Ab > 30% gate retains a positive but smaller gain, limiting interpretation of the headline as universal.
  • Expanded results: The Humanity’s Last Exam pilot reports 12.57% direct and 34.03% assisted accuracy on 573 rows, separately from the primary bundle.
  • Limitations: Assisted accuracy reflects a larger inference budget because generated programs may make multiple calls and retry invalid extractions without a matched-budget direct baseline.
  • Limitations: Generated-code validity is not guaranteed: audits found hard-coding-like assignments, while the execution environment lacks several proven safety controls.
  • Quantitative result: The primary observed non-zero-baseline comparison is +28.10 percentage points, with a pair-bootstrap interval of [20.32, 36.43].
Loading 2605.18827v1…