PaperlayerView exact revision on arXiv

Source-linked AI summary

Edit, Don't Imitate: Minimal-Edit On-Policy Distillation for Multimodal Reasoning

Wen Ye, Ailin Deng, Zirui Wang, Shixing Yu, Han Zhang, Tian Yun, Nikhil Mehta, Tarun Kalluri, Xuefei Cao, Xuezhe Ma, Hengduo Li

arXiv:2610.30231v1cs.LGcs.AI

TL;DR

Visual mathematical reasoning can fail when a VLM misreads a diagram, and obtaining accurate descriptions at inference requires additional resources. The paper uses descriptions and verified solutions during training to selectively correct incorrect on-policy rollouts. Across scales, it reports gains without test-time descriptions, including improved perception-oriented performance and transfer to external mathematics benchmarks.

  • Problem

    Diagram misperception can derail visual mathematical reasoning, especially for smaller VLMs, while supplying descriptions at inference requires additional annotation or model calls.

    Source: §1 · p. 1

  • Method

    Minimal-edit corrective on-policy distillation uses reference descriptions and verified solutions to select teacher-disfavored tokens in incorrect rollouts, while correct rollouts receive task-reward reinforcement learning.

    Source: §1 · p. 1, §3.2 · pp. 4–5

  • Results

    Without test-time descriptions, the method meets or exceeds the corresponding base model supplied with a reference figure description, improves perception-oriented performance, and transfers to external multimodal mathematics benchmarks.

    Source: §1 · p. 2, §4.1 · p. 7, §5 · p. 9

  • Takeaways & Limitations

    The results support using recoverable privileged information as a training-time diagnostic rather than an additional inference input or generation target.

    Source: §5 · p. 9

    The reverse-KL sign mask is not guaranteed to select every token occurrence reflecting an underlying error.

    Source: §5 · p. 10

Abstract

from arXiv · show

Visual mathematical reasoning can fail before reasoning begins: when a vision-language model (VLM) misreads a diagram, subsequent reasoning proceeds from false premises. This perceptual bottleneck is especially pronounced in smaller models, while accurate textual descriptions can substantially narrow the gap to larger models. Because obtaining such descriptions at inference requires additional annotation or model calls, we propose minimal-edit corrective on-policy distillation. Using reference figure descriptions and verified reference solutions as training-only information, a teacher scores incorrect student rollouts under a minimal-edit objective, and only teacher-disfavored tokens contribute to the corrective loss. Correct rollouts receive task-reward reinforcement learning. This preserves on-policy supervision while avoiding full-response imitation. Across model scales, the method requires no descriptions at test time and meets or exceeds the corresponding base model supplied with a reference figure description at inference. It also improves perception-oriented performance, particularly low-level geometric perception, and transfers to external mathematics and perception benchmarks. Because the minimal-edit teacher evaluates corrections rather than generating its own solution, the method remains effective with a frozen same-size teacher, although its gains are smaller than with a larger teacher on most metrics.

1 Introduction

Diagram misperception can derail visual mathematical reasoning, particularly for smaller VLMs, while obtaining accurate descriptions at inference requires additional resources. The paper uses reference descriptions and verified solutions only during training to correct selected tokens in incorrect rollouts, and reports gains without test-time descriptions.

1 Introduction

  • A faithful diagram description raises a 2B VLM’s Geometry3K pass@1 from 0.32 to 0.46, approaching the 0.49 attained by an 8B model.The paper identifies perception as a major source of smaller models’ disadvantage, while noting that reasoning capacity still matters.Source: §1 · p. 1
  • Reference descriptions and verified solutions provide training-only perceptual information, avoiding the need to obtain descriptions at inference.The introduction motivates using the signal during training so the model ultimately solves from the image and question.Source: §1 · pp. 1–2
  • For incorrect rollouts, a minimal-edit teacher scores the student’s existing response and only teacher-disfavored tokens contribute to corrective loss; correct rollouts receive task-reward RL.The teacher is instructed to correct errors while preserving already-correct response portions; it does not generate a revised solution.Source: §1 · pp. 1–2, Figure 1 · p. 2
  • Without test-time descriptions, minimal-edit distillation meets or exceeds the corresponding base model supplied with a reference figure description, improves perception-oriented performance, and transfers to external multimodal mathematics.A frozen same-size teacher remains effective, though its gains are smaller than those of a larger teacher on most metrics.Source: §1 · p. 2

2 Related Work

Related work addresses selective on-policy supervision, reward routing, privileged visual information, and perception-aware training. The paper distinguishes its approach by scoring the student’s unchanged response under a minimal-edit objective and selecting disfavored tokens rather than generating replacement text or an error suffix.

2 Related Work

  • Prior selective approaches target unrecoverable prefixes or token advantages, while ROSD distills after locating an erroneous span and Correction2Mastery transfers a generated correction [Xin et al., 2026; Jia et al., 2026; Zhao et al., 2026b; Lyu et al., 2026].The paper instead scores the unchanged response under a minimal-edit instruction and selects tokens by the sign of the sampled-token log-ratio.Source: §2 · pp. 2–3
  • Reward-and-distillation methods mix signals or route failed samples to self-distillation; this method routes incorrect rollouts to minimal-edit token correction and correct ones to task reward [Xu et al., 2025; Tan and Hong, 2026; Li et al., 2026].This routing avoids discarding correct samples while applying correction only to incorrect ones.Source: §2 · p. 3
  • Other privileged-visual methods use crops, structured hints, recoverable cues, or explicit description and reasoning spans; this method conditions only a frozen teacher on a recoverable figure description [Yuan et al., 2026; Xiang et al., 2026; Tian et al., 2026; Wang et al., 2026a].The student needs no description, cue module, or structured perception stage at inference.Source: §2 · p. 3
  • Perception-aware training has used image ablation, mined visual annotations, perturbed-image rollouts, or a preliminary perception stage; this work uses a teacher’s textual reading of the diagram [Wang et al., 2026c; Xiao et al., 2026; Liu et al., 2025; Chen et al., 2025].The paper compares against released checkpoints from this line on a shared backbone.Source: §2 · p. 3

3 Method

Reference figure descriptions improve Geometry3K performance across model scales, motivating training that transfers their perceptual benefit without using them at inference. Minimal-edit distillation scores incorrect student rollouts and selects teacher-disfavored tokens for correction, while correct rollouts receive task-reward reinforcement learning.

3.1 How much do reference figure descriptions improve visual reasoning?

  • With descriptions, Qwen3-VL-2B reaches 0.462 pass@1, matching Qwen3-VL-4B without descriptions and approaching Qwen3-VL-8B’s 0.491.Qwen3-VL-4B reaches 0.672 with a description, exceeding Qwen3-VL-8B without one.Source: §3.1 · p. 3, Figure 2 · p. 4
  • Privileged OPD transfers a description-based teacher signal to a student that samples from image-and-question input alone.The teacher scores the same student-sampled token sequence with the reference description; the per-token signal is the student–teacher log-probability difference.Source: §3.1 · p. 4

3.2 Minimal-edit corrective on-policy distillation

  • For incorrect rollouts, the minimal-edit teacher scores the unchanged response conditioned on the figure description, verified solution, and an instruction to preserve correct content and style.The teacher scores the response rather than generating a corrected solution.Source: §3.2 · p. 4
  • The corrective mask retains tokens the teacher disfavors, targeting errors such as the student’s leg–hypotenuse misreading rather than all response positions.The mask selects tokens with k_t > 0; the illustrated conventional privileged teacher marks 224 of 515 tokens, including correct passages.Source: §3.2 · p. 5, Figure 3 · p. 5
  • Correct rollouts receive GRPO task reward and incorrect rollouts receive minimal-edit distillation; the corrective mask retains roughly 2% of tokens in incorrect rollouts.The method computes group-relative advantages over the full rollout group before routing the rollouts.Source: §3.2 · p. 5

4 Experiments

Experiments show that minimal-edit distillation improves multimodal mathematics and perception without test-time figure descriptions, including when a frozen same-size teacher is used. Ablations indicate that the corrective weight, verified reference solution, corrective token selection, and reward for correct rollouts all matter for performance and training stability.

4 Experiments

  • Training uses Geometry3K problems filtered for a verified reference solution, with descriptions supplied to the teacher; evaluation covers Geometry3K, MathVerse, figure-description recall, and Geoperception.The student receives only the image and question, and the reference description is not provided at inference.Source: §4 · pp. 6–7

4.1 Main results

  • Minimal-edit distillation consistently improves Geometry3K and MathVerse, exceeds the reference-description base models, and leads all multimodal-math metrics on Qwen2.5-VL-7B.The Qwen2.5-VL-7B result uses one of the smallest training sets.Source: Table 2 · p. 7, §4.1 · p. 7, Table 3 · p. 8
  • Minimal-edit distillation improves both perception-oriented evaluations over the 2B and 8B base models; on Qwen2.5-VL-7B, it is competitive with published models.GRPO with reference figure descriptions does not reliably improve perception over the base model.Source: §4.1 · p. 7

4.2 Ablation studies

  • Removing task reward from correct rollouts causes entropy to collapse and responses to reach the generation limit, while the full method maintains stable training dynamics.Both incomplete configurations lead to declining validation accuracy and optimization collapse.Source: Figure 4 · p. 8, §4.2 · p. 9

4.1 Main results

  • A frozen same-size teacher remains effective: its results are close to those of a larger teacher, with a larger gap at 2B than at 8B.At 2B, minimal-edit self-distillation still outperforms naive and privileged distillation with the larger teacher.Source: §4.1 · pp. 7–8

4.2 Ablation studies

  • Every tested corrective weight improves on the untrained base, but an intermediate weight performs best; removing the verified reference solution degrades all six measures.The sweep varies λ from 0.001 to 0.1, and the reference-solution ablation removes it from the λ = 0.01 configuration.Source: §4.2 · p. 8, Table 5 · p. 9
  • Randomly masking tokens at the corrective mask’s roughly 2% keep rate does not prevent instability; the selected corrective tokens carry meaningful information.The random-mask control and full method differ only in token selection.Source: Table 4 · p. 8, §4.2 · p. 9

5 Conclusion and Future Perspectives

Minimal-edit corrective on-policy distillation uses privileged visual information for response-specific correction during training, without requiring figure descriptions at inference. Across model scales, it improves reasoning and perception-oriented performance, while a same-size frozen teacher remains effective.

5 Conclusion and Future Perspectives

  • Across model scales, minimal-edit distillation improves reasoning and perception-oriented performance, meets or exceeds the corresponding base model with an inference-time reference description, and works with a frozen same-size teacher.Gains from the same-size teacher are smaller than those from a larger teacher on most metrics.Source: §5 · p. 9
  • Incorrect rollouts receive response-specific correction from teacher-disfavored tokens, while correct rollouts receive task-reward reinforcement, using privileged descriptions and verified solutions only during training.The teacher scores the student's existing response rather than generating a replacement; sampled rollouts are routed through mutually exclusive branches.Source: §5 · p. 9, A Minimal-Edit Corrective OPD Pseudocode · p. 14
  • The reverse-KL mask is not guaranteed to select every token occurrence that reflects an underlying error, although partial localization provides an effective corrective signal in the experiments.The conclusion identifies faithful, task-aligned, informative privileged views as a requirement for extending the approach to other visual domains.Source: §5 · p. 10

A Minimal-Edit Corrective OPD Pseudocode

  • For each example, the algorithm samples K student rollouts and uses verifier rewards to route correct responses to the task-reward branch and incorrect responses to teacher scoring.TeacherForce scores a second occurrence of the same rollout; at inference, the student receives only x.Source: A Minimal-Edit Corrective OPD Pseudocode · pp. 14–15

B Computational Cost Analysis

The cost analysis compares minimal-edit distillation with naive on-policy distillation while holding student behavior fixed. Its compute advantage depends on the incorrect-rollout fraction and the teacher sequence lengths.

B Computational Cost Analysis

  • The comparison holds student behavior fixed using observed sequence lengths and incorrect-rollout fractions; D is Ps + R, and the teacher has rN parameters for an N-parameter student.The shared student computation includes rollout generation, a post-rollout behavior-policy pass, and a policy update; activation recomputation raises update cost to 8ND.Source: B Computational Cost Analysis · p. 15
  • Minimal-edit teacher scoring processes only the incorrect-rollout fraction w, at Pt + 2R tokens per scored rollout, while naive OPD scores every rollout over D tokens.The extra response copy lets the teacher condition on the complete student response while evaluating its teacher-forced suffix.Source: B Computational Cost Analysis · p. 15
  • At w = w*, expected teacher-processed tokens are equal; as the correct-rollout fraction rises during training, minimal-edit teacher calls become less frequent relative to naive OPD.The threshold is w* = D / (Pt + 2R).Source: B Computational Cost Analysis · pp. 15–16
  • 7.1% fewer FLOPs for the 8B student, but 13.3% more FLOPs for the 2B student, relative to naive OPD.For 8B, the expected teacher-processed tokens are 781.67 versus 945.94; for 2B, they are 1314.89 versus 992.81.Source: Table 7 · p. 16, B Computational Cost Analysis · p. 16

C Training Dynamics of the Minimal-Edit Run

During the 2B training run, the corrective mask remains sparse while correct rollouts become more frequent. The supervision mix consequently shifts toward task reward without broadening token-level correction.

C Training Dynamics of the Minimal-Edit Run

  • About 2% of response tokens from incorrect rollouts remain selected by the corrective mask, with no sustained upward trend over 250 steps.The teacher continues to identify a small set of candidate corrections as the on-policy distribution changes.Source: C Training Dynamics of the… · p. 16
  • The correct-rollout fraction rises from roughly one third early in training to more than one half near the end.Because the correct and incorrect branches are mutually exclusive, more sampled responses receive task-reward reinforcement as this fraction rises.Source: C Training Dynamics of the… · p. 16
  • As successful rollouts become more common, sparse token-level correction remains directed at the remaining failures rather than spreading across responses.The observed combination is consistent with the intended division of labor between task reward and corrective supervision.Source: C Training Dynamics of the… · p. 16

D Baseline Training Details

The appendix specifies supervised and reinforcement-learning baselines, describes how training-only figure descriptions are collected, and details the fact-based recall evaluation used for generated descriptions.

D Baseline Training Details

  • Supervised baselines use full-parameter fine-tuning, while RL baselines use GRPO with eight rollouts per prompt and a rule-based Geometry3K correctness reward.SFT uses AdamW, bfloat16, and up to three epochs; RL uses one strictly on-policy update per rollout batch and runs for 30 epochs.Source: D Baseline Training Details · pp. 16–17

E.1 Figure-description collection

  • Collected figure descriptions preserve visually shown facts and use the verified answer only to check completeness, not to reveal the answer or derived values.The question remains verbatim in the final prompt, so descriptions should supply visual facts absent from the question.Source: §E.1 · pp. 17–18

E.2 Recall calculation

  • Figure-description recall is computed from covered atomic reference facts per diagram, then averaged so each test diagram contributes equally regardless of its fact count.A candidate receives coverage for a fact only when it conveys matching information; equivalent wording counts.Source: §E.2 · p. 18

E.3 Gemini 3.1 Pro judge prompt

  • The Gemini judge decomposes reference descriptions into independently checkable facts and marks each covered only when the candidate conveys the same value or relationship.Omissions and contradictory values are uncovered; Tables 8 and 9 report recall means and standard deviations across three sampled descriptions per diagram.Source: §E.3 · p. 19, Table 8 · p. 20, Table 9 · p. 20

F.1 Figure-description recall standard deviations

The main tables report mean figure-description recall, while Tables 8 and 9 add standard deviations across three independently sampled description draws.

F.1 Figure-description recall standard deviations

  • Main-table figure-description recall is reported as the mean over three candidate descriptions sampled independently for each test diagram.Sampling uses temperature 0.8 and seeds 1001, 1002, and 1003.Source: §F.1 · p. 20
  • Each candidate description is scored against the same reference figure description using the judge described in Section E.Source: §F.1 · p. 20
  • Tables 8 and 9 repeat the main-table means with standard deviations across the three sampled-description draws.Source: §F.1 · p. 20

F.2 Geoperception breakdown by predicate

The appendix reports predicate-level Geoperception pass@1 tables and specifies the prompts used for conventional privileged-teacher and minimal-edit-teacher scoring.

F.2 Geoperception breakdown by predicate

  • Tables 10–12 decompose aggregate Geoperception pass@1 by predicate, reporting means with eight rollouts per example.The tables cover Qwen3-VL-2B, Qwen3-VL-8B, and Qwen2.5-VL-7B backbones.Source: §F.2 · p. 20, Table 10 · p. 21, Table 11 · p. 21
  • The predicate columns cover angle classification, line comparison, point on line, equality, point on circle, parallelism, and perpendicularity.Source: §F.2 · p. 21

G.1 Privileged teacher prompt

  • The conventional privileged teacher receives the diagram image, collected figure description, and question, then scores the appended student rollout token by token.Source: §G.1 · p. 21

G.2 Minimal-edit teacher prompt

  • For incorrect rollouts, the minimal-edit teacher is conditioned on the figure description, reference solution, and student solution, but performs teacher-forced scoring rather than generating a rewrite.The prompt asks for the smallest correction while preserving correct text verbatim.Source: §G.2 · p. 22

H Qualitative Examples

The examples show minimal-edit conditioning concentrating corrective signals near errors while preserving correct response content, across perceptual misreads and downstream arithmetic mistakes.

H Qualitative Examples

  • In the triangle example, the edit identifies 32 as the hypotenuse, revises dependent reasoning, and changes the answer from 32 3 to 16 while retaining applicable text.The hypothetical rollout illustrates the edit; training scores the unchanged student response instead.Source: H Qualitative Examples · p. 22, Figure 7 · p. 23
  • For the arithmetic example, minimal-edit conditioning preserves the correct relation AB = 56 cos(42◦) and concentrates correction near the product error, changing 41.2 to 41.6 feet.The conventional privileged teacher assigns positive estimates across otherwise usable derivation text.Source: H Qualitative Examples · p. 24, Figure 8 · p. 24, Figure 9 · p. 25
  • In the diameter–radius example, the student treats a 10 cm diameter as the radius; the edit instead sets r = 5 and updates the arc length to 65π/18 ≈11.34.Minimal-edit conditioning focuses its strongest corrective signal on the initial radius misidentification, preserving most of the otherwise coherent calculation.Source: H Qualitative Examples · p. 24, Figure 10 · p. 26, Figure 11 · p. 27
  • Across the three examples, minimal-edit conditioning better preserves tokens that need not change and converts response-specific preferences into a sparse corrective mask.The examples span a triangle-side misidentification, an arithmetic error after correct setup, and a diameter–radius reading error.Source: H Qualitative Examples · pp. 22, 28

Related papers

Loading 2610.30231v1…