PaperlayerView exact revision on arXiv

Source-linked AI summary

Two Heads Are Better Than One: Dual-Model Verbal Reflection at Inference-Time

Jiazheng Li, Yuxiang Zhou, Junru Lu, Gladys Tyen, Lin Gui, Cesare Aloisi, Yulan He

arXiv:2502.19230v2cs.CL

TL;DR

Automated Student Answer Scoring needs explainable rationales, but preference optimization can obscure why an assessment is preferred, and LLMs may fail to locate reasoning errors. The paper synthesizes critiques by comparing reasoning paths and trains separate Reasoner and Critic models for reflective scoring. It reports that DARS outperforms ASAS baselines across metrics and evaluation settings.

  • Problem

    Preference optimization can improve reasoning while leaving why one assessment is preferred opaque, and LLMs struggle to detect and locate reasoning errors in ASAS.

    Source: §1 · p. 1

  • Method

    DARS compares preference-based reasoning paths to synthesize targeted critiques, then trains dedicated Reasoner and Critic models for iterative reflection and scoring.

    Source: §1 · p. 2

  • Results

    DARS consistently outperforms baselines, including in scarce-data settings, scales with model size, and generalizes across different LLM base models.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The framework supports reasoning refinement without oracle labels, with the Critic providing feedback and signaling when refinement converges.

    Source: §1 · p. 2, §6 · p. 9

    Training both models requires higher training FLOPs than a single-Reasoner approach, and generalization beyond ASAS remains unexplored.

    Source: Limitations · p. 9

Abstract

from arXiv · show

Although preference optimization methods have improved reasoning performance in Large Language Models (LLMs), they often lack transparency regarding why one reasoning outcome is preferred over another. This limitation is especially critical in Automated Student Answer Scoring (ASAS), where explainability is essential to justify assessment outcomes. Verbal reinforcement learning offers the potential to generate explicit reflection, but it tends to produce superficial critiques that can harm assessment performance. Existing LLMs also struggle to reliably detect subtle reasoning errors in ASAS tasks. Moreover, manually identifying intermediate reasoning errors is expensive and difficult to scale. To address these challenges, we introduce a contrastive reflection synthesis pipeline that generates precise verbal feedback by identifying discrepancies in structure reasoning graph paths. Leveraging these synthetic reflection data, we propose DARS, a Dual-model Reflective Scoring framework featuring a dedicated Critic model trained for effective reflection. DARS achieves strong performance and consistently outperforms existing ASAS baselines across all evaluation metrics. Extensive experiments further provide novel insights into the value of reflection data, framework design, and the scaling behavior of DARS. We release the DARS code at https://github.com/lijiazheng99/DARS.

1 Introduction

ASAS rationales can contain subtle errors, while preference optimization may leave the reasons for a preferred assessment opaque and verbal reflection can be superficial. DARS synthesizes targeted critiques from divergent reasoning paths and trains separate Reasoner and Critic models; experiments report consistent gains across baselines and settings.

1 Introduction

  • ASAS automates human graders' reasoning, and generating free-form rationales alongside scores is one approach to making automated decisions more transparent.ASAS also offers the potential for faster and more consistent assessment at scale.Source: §1 · p. 1
  • Generated rationales can mix valid logic with subtle but impactful errors, while advanced models may overlook flawed steps and produce vague reflections.The introduction also notes a lack of high-quality annotations and argues that assessment involves graph-like conceptual structures.Source: §1 · pp. 1–2
  • Preference optimization captures which assessment is preferred but not why, while verbal reinforcement learning critiques and revises reasoning that LLMs may struggle to localize accurately.The paper describes this as leaving reasoning steps opaque and limiting reliable self-correction.Source: §1 · p. 1
  • The contrastive pipeline compares preference-based reasoning paths over a shared tree to locate divergences and generate targeted critiques without human annotation.Each tree decision reflects whether a key answer element is present.Source: §1 · p. 2
  • DARS trains dedicated Reasoner and Critic models: the Reasoner gives an initial score and rationale, while the Critic reflects and signals convergence.The framework is designed to support verbal reinforcement learning without oracle labels or manually defined thresholds.Source: §1 · p. 2
  • DARS consistently outperforms baselines, including in scarce-data settings, scales with model size, and generalizes across different LLM base models.These are the paper's reported experimental findings.Source: §1 · p. 2

2 Preliminary

The preliminary formulation represents a student's answer through the key answer elements it covers and maps that coverage to a score. It motivates structured thought trees as a way to represent intermediate assessment decisions that existing datasets do not provide.

2 Preliminary

  • ASAS systems use question prompts, key answer elements, marking rubrics, and student responses to predict scores, while existing datasets provide annotated answer-score pairs.The task automates teachers' complex reasoning on short-answer assessment.Source: §2 · p. 2
  • A student's response and key answer elements define a multi-hot coverage vector, which a question-specific scoring function maps to the final score.The paper notes that these intermediate assessment states are unavailable in current datasets because of reasoning complexity and annotation costs.Source: §2 · p. 2
  • Following Li et al. (2024a), a structured thought tree represents assessment decisions, with each path encoding binary decisions over the answer's key elements.The preliminary section describes the tree as a way to mimic human assessment.Source: §2 · p. 2
  • A path yielding a correct score is the human-preferred path, while one yielding an incorrect score is human-rejected; each rationale summarizes its path's intermediate decisions.The two path types provide the preference pair used in the formulation.Source: §2 · p. 2

3 DARS: Dual-Model Reflective Scoring

DARS uses a Reasoner and Critic to produce iterative scoring rationales, with the Critic providing reflections or stopping the process. Its reflection data are synthesized by comparing structured reasoning paths to locate divergences and generate targeted critiques.

3 DARS: Dual-Model Reflective Scoring

  • DARS pairs a Reasoner with a Critic that supplies free-form reflections to steer reasoning; at inference, no thought tree is constructed and no reinforcement learning is used in training.The critique-and-revise behavior arises from SFT-trained models interacting on-policy at test time.Source: §3 · pp. 2–3

3.1 Contrastive Reflection Synthesis

  • Contrasting structured reasoning paths turns each preference pair into a fine-grained critique that identifies the minimal subgraph behind the discrepancy.The pipeline addresses how linearized LLM reasoning can obscure the source of errors and make self-critiques vague or superficial.Source: §3.1 · p. 3

Step 1: Identify Discrepancy in Reasoning Paths

  • A signed difference vector flags rubric-linked key-element decisions that differ between the chosen and rejected paths, localizing the subgraph responsible for diverging scores.Each non-zero component is converted into a natural-language structural hint about the intermediate assessment decisions.Source: Step 1: Identify Discrepancy in… · pp. 3–4

Step 2: Generate Synthetic Reflections

  • An LLM uses the structural hint and rejected and chosen rationales to generate a reflection that can pinpoint a missed answer element.The hint anchors concise, node-level critiques in the concept graph.Source: Step 2: Generate Synthetic Reflections · p. 4

3.2 Dual-Model Training & Inference

  • At inference, the Reasoner drafts a scoring rationale and the Critic either gives targeted feedback for revision or emits [STOP] when reasoning has converged.The dialogue continues until the Critic determines that the reasoning has converged.Source: §3.2 · p. 4

Training Reasoner and Critic Models

  • The Reasoner is trained for initial assessment and reflection-guided refinement, while the Critic learns to diagnose errors and decide when to stop without oracle labels or preset iteration limits.The Critic predicts [STOP] when the assessment is correct.Source: Training Reasoner and Critic Models · p. 4

Inference-Time Iterative Refinement

  • During iterative refinement, the Critic checks each Reasoner assessment, provides a reflection when revision is needed, and otherwise triggers [STOP] so the final assessment is output.The inference loop uses the trained Reasoner and Critic to refine the assessment rationale.Source: Inference-Time Iterative Refinement · p. 4

4 Experiments

Experiments compare DARS with scoring and reasoning baselines, then examine reflection quality, scaling, ablations, and generalization. DARS improves assessment performance, while results also show that its effectiveness depends on the Reasoner and Critic roles and their training.

4.1 Experimental Setup

  • Experiments use six datasets from ASAP and a proprietary biology-exam source, evaluating assessment with ACC, macro F1, and QWK.All methods, including DARS, use the same LLaMA 3B base model.Source: §4.1 · p. 5, §4.2 · p. 5

4.2 Overall Comparison

  • Under data scarcity, DARS improves ACC by 5%, F1 by 11%, and QWK by 2%; Reasoner+Critic gains 3%-7% over Reasoner-only across datasets and metrics.Compared with GPT-4 as Critic, DARS is 18%-34% better in metrics and significantly outperforms SFT and DPO.Source: §4.2 · pp. 5–6
  • DARS outperforms GPT-4 as Critic with fewer iterations: it improves by iteration 2, while GPT-4 takes nearly four iterations and then degrades.The comparison visualizes performance trends and completion rates; the reported degradation is across all metrics.Source: Figure 4 · p. 6, §4.2 · p. 6

4.3 Quality Evaluation for Reflection

  • The Critic correctly localizes assessment errors in 64% of cases, but inaccurate reflections mislead the Reasoner in 34% of cases.Reflections were inaccurate in 36% of cases, often from misinterpreting the answer or key-element scope; the Reasoner followed Critic guidance in 97% of cases.Source: §4.3 · pp. 6–7

4.4 Scaling Experiment for DARS Framework

  • Larger Critics produce greater ACC and QWK gains than larger Reasoners; the F1 trend is less pronounced amid dataset-size and label-distribution imbalances.Performance improvements are measured relative to the Reasoner-only result.Source: §4.4 · p. 7, Figure 6 · p. 7

4.5 Ablation Studies on DARS

  • Weak Reasoners lose more than 10% across metrics despite strong Critics and high-quality reflection data, while excluding refinement data lowers Reasoner-only results nearly 5% across metrics.The refinement-data comparison attributes the gains to training on error-correction examples.Source: §4.5 · p. 7
  • A single model trained for both reasoning and reflection shows negative iterative-refinement improvement and struggles to terminate appropriately or provide useful instructions.The unified-model experiments use Qwen 3B and LLaMA 3B.Source: §4.5 · p. 8, Figure 9 · p. 8

4.6 Generalization Studies

  • An unseen-data Critic still improves Reasoner QWK by 1%, though it is 3% less effective in QWK than the seen-data Critic; performance also varies minimally across tested model sizes, architectures, and pairings.The cross-model pairings include a Qwen Reasoner with a LLaMA Critic, with consistent performance attributed to text-based interaction.Source: §4.6 · p. 8, Figure 10 · p. 8, Figure 11 · p. 8

5 Related Work

Prior work uses verbal reflection to improve LLM reasoning and rationale generation to make automated scoring more transparent. The paper positions its approach against limits in self-correction and existing ASAS explainability methods.

5 Related Work

  • VRL has emerged as a promising approach for enhancing LLM reasoning at inference time.Source: §5 · p. 8
  • Early self-reflection methods refined outputs using contextual cues, while trained critics later supplied verbal feedback, primarily at the single-step level.More complex reasoning tasks have typically relied on oracle labels for correction; this work instead introduces trace-level reflections without oracle labels in verification.Source: §5 · p. 9
  • ASAS research has pursued transparency through feature analysis, attention visualization, and rationale generation, but rationale-generating approaches often underperform classification-based methods.The paper builds on thought trees to improve transparency of the assessment-refinement process through iterative LLM reasoning.Source: §5 · p. 9

6 Conclusion and Discussion

The paper concludes that its dual-model framework and contrastive reflection synthesis support reasoning refinement without oracle labels. Its Reasoner and Critic are trained to refine reasoning and identify errors and convergence, respectively.

6 Conclusion and Discussion

  • The synthesis pipeline generates more targeted verbal reflections as part of the dual-model approach to reasoning improvement.Source: §6 · p. 9
  • DARS combines a dedicated Reasoner and Critic with contrastive reflection synthesis to refine reasoning without oracle labels.Source: §6 · p. 9
  • The Reasoner learns to refine its reasoning from feedback, while the Critic learns both to identify errors and determine when to stop.Source: §6 · p. 9

Limitations

The study identifies higher training costs, untested generalization beyond ASAS, and non-exhaustively optimized prompts as limitations. It recommends further investigation of broader task applicability and prompting strategies.

Limitations

  • Training both models requires more FLOPs than single-Reasoner approaches because their SFT uses additional data to enhance model capabilities.Source: Limitations · p. 9
  • Generalizability beyond ASAS remains unexplored, despite evaluation across six datasets focused predominantly on ASAS.Source: Limitations · p. 9
  • Prompt design was not exhaustively optimized; future work could examine in-context learning and chain-of-thought prompting.Source: Limitations · p. 9
  • Math and code reasoning may not need binary structured thought trees, but could use predefined rules to verify intermediate steps and identify path discrepancies.Source: Limitations · p. 9

Ethics Statement

The study used anonymized student responses from public and proprietary datasets and reports no observed harmful content or personal-information exposure. Before high-stakes examination deployment, experts should evaluate the assessment decisions and rationales for reliability and fairness.

Ethics Statement

  • The study used anonymized student responses from public and proprietary datasets, none containing sensitive or personally identifiable information.Source: Ethics Statement · p. 9
  • Review of the LLM outputs found no instances of harmful content or exposure of personal information.Source: Ethics Statement · p. 9
  • For high-stakes examination deployment, experts must evaluate assessment decisions and rationales for reliability and fairness.Source: Ethics Statement · p. 9

A Further Experiment Setup

The appendix reports additional experimental setup details and case studies of iterative assessment refinement, including a limitation of GPT-4-turbo as a Critic.

A Further Experiment Setup

  • The authors manually reviewed 20 instances per dataset, classifying Critic reflection errors and Reasoner refinement outcomes.The categories distinguish correct reflections from incorrect ones, and correct refinements from errors that obey or ignore the reflection.Source: A Further Experiment Setup · p. 14

B.1 Explanation for Main Example

  • In the example, the Reasoner initially over-credits details absent from the student response; the Critic flags them, and the revised assessment corrects the score before [STOP].The Critic's termination token signals the end of the reasoning loop.Source: §B.1 · p. 15

B.2 Case Studies on GPT-4-turbo as Critic

  • GPT-4-turbo gives vague, poorly aligned feedback and fails to terminate refinement even when the Reasoner’s assessment is correct.The authors argue that effective reflection requires a more tailored Critic with domain-specific feedback.Source: §B.2 · p. 15

B.3 Detailed Error Analysis

The detailed analysis finds that Critic guidance shifts predictions toward ground truth, while DARS with two LLaMA 3B models outperforms a single LLaMA 8B DPO model.

B.3 Detailed Error Analysis

  • After the second Critic-guided iteration, label distributions move closer to ground truth and confusion matrices show stronger diagonal patterns.First-iteration predictions show a bias toward scores 0 and 1.Source: §B.3 · p. 15, Figure A5 · p. 18
  • Most label transitions move from incorrect to correct, often by one point between scores 1 and 3, though some correct labels are also changed incorrectly.The analysis uses a single run with LLaMA 3B Reasoner and Critic.Source: §B.3 · p. 15, Figure A5 · p. 18

B.4 Two Smaller Models May Better Than a Larger One

  • DARS with LLaMA 3B Reasoner and Critic outperforms a single LLaMA 8B DPO model.The authors suggest the result may relate to LLaMA 3B being distilled from the 8B version.Source: §B.4 · p. 15, Figure A6 · p. 18

B.5 Can Refinement Data Enhance Preference Optimization for the Reasoner?

The section examines whether reflection data can regularize DPO training, reporting slight gains in QWK and F1 over vanilla DPO.

B.5 Can Refinement Data Enhance Preference Optimization for the Reasoner?

  • The baseline adds an SFT loss on synthetic reflection data to regularize DPO training.The approach is presented as a preference-optimization baseline inspired by prior work.Source: §B.5 · p. 18
  • QWK and F1 scores improve slightly with reflection-data regularization compared with vanilla DPO.The comparison is shown in Figure A7.Source: §B.5 · p. 18, Figure A7 · p. 18
  • The authors report that refinement data can also act as an effective regularizer for single-reasoner training, enhancing performance and stability.Source: §B.5 · p. 18

B.6 Case Studies on Our Framework

The case studies illustrate both Critic errors and the ability to detect flawed reasoning despite a correct score; a separate comparison reports more actionable feedback from larger Critics.

B.6 Case Studies on Our Framework

  • In the vinegar example, only the vinegar amount supplies new information, but the Reasoner awards 2 or 3 points instead of the correct 1.The other listed details are vague or already covered by the procedure.Source: §B.6 · p. 18
  • The Critic identifies an unsupported student conclusion even though the Reasoner’s final score of 2 is correct for two valid design improvements.Source: §B.6 · p. 19

B.7 Case Study: Comparing Critic’s Output with Different Sizes

  • In the size comparison, Qwen 14B and 32B Critics provide more constructive, detailed revision guidance than the vague 3B or abrupt 7B responses.The 32B Critic gives the most comprehensive feedback in the example.Source: §B.7 · p. 19, Figure A10 · p. 22

Related papers

Loading 2502.19230v2…