OPD-Aha: From Linguistic Momentum to Visual Reflection in Multimodal On-Policy Distillation

1 INTRODUCTION

• Erroneous student prefixes can pull the privileged teacher toward a hallucinated continuation, collapsing corrective supervision when the student most needs intervention.The failure arises as accumulated text supporting a mistaken image interpretation marginalizes the teacher’s visual evidence.Source: §1 · pp. 1–2
• OPD-Aha contrasts the identical teacher’s predictions with privileged visual evidence and a visual null, then reconstructs a target that suppresses image-inconsistent continuations.This isolates a corrective visual preference that persists even when the teacher’s overall distribution aligns with the student’s erroneous reasoning.Source: §1 · p. 2
• The reconstructed target suppresses erroneous continuations more strongly, giving reflection tokens such as wait and actually a relative advantage when explanations conflict with the image.After reflection, students rely less on earlier erroneous explanations and more on visual evidence.Source: §1 · p. 2
• OPD-Aha yields consistent accuracy improvements across fine-grained perception benchmarks and positive transfer to unseen multimodal reasoning tasks.The paper argues that robust multimodal distillation requires preserving visual correction against the linguistic momentum of student trajectories.Source: §1 · p. 2

2.1 PRIVILEGED MULTIMODAL ON-POLICY DISTILLATION

• Privileged OPD trains a student on its own rollout while a frozen teacher evaluates those states using additional training-only visual evidence unavailable to the student.At each decoding step, both models share the student-generated linguistic prefix.Source: §2.1 · pp. 2–3

2.2 ERRONEOUS STUDENT PREFIXES MASK PRIVILEGED SUPERVISION

• The analysis retains progressively longer portions of failed Vision-OPD trajectories and measures the teacher’s length-normalized log-likelihood margin between the correct answer and the student’s realized wrong answer.The margin indicates whether the teacher favors a visually grounded correction or the erroneous answer.Source: §2.2 · p. 3
• As failed reasoning accumulates, the teacher’s preference flips toward the student’s wrong answer, and the teacher–student discrepancy collapses.The decision margin changes from positive to negative near the response midpoint.Source: Figure 1 · p. 3, §2.2 · p. 3
• Even when cross-model discrepancy collapses, the teacher’s real–null contrast retains a preference for the correct answer; shuffling the visual change across tokens destroys this signal.The surviving response is a token-specific visual preference rather than undirected sensitivity to the input.Source: §2.2 · p. 3, Figure 1 · p. 3

3 RECONSTRUCTING VISUAL SUPERVISION

• The student learns from the reconstructed target along its unchanged rollout, without requiring auxiliary visual inputs during inference.The adjusted target probabilities carry the visual correction during training.Source: §3 · p. 4, §3.2 · p. 5

3.1 ISOLATING VISUAL PREFERENCE

• Comparing the identical teacher under a privileged image and a dimension-preserving visual null at the same prefix isolates visual preference that can survive agreement but is not necessarily corrective.The visual null is instantiated by replacing the image with its mean RGB color.Source: §3.1 · pp. 4–5

3.2 TARGET RECONSTRUCTION FROM VISUAL PREFERENCE

• The method balances alignment with visual preference against proximity to the privileged teacher, selectively adjusting token probabilities while retaining a valid language distribution.The correction strength β controls the adjustment; it can favor interruption when visual evidence sufficiently suppresses an image-inconsistent continuation.Source: §3.2 · p. 5

4.1 RECONSTRUCTION SUSTAINS CORRECTIVE SUPERVISION

• At late-prefix states, the reconstructed target maintains over an order of magnitude higher correct-answer probability than standard privileged supervision.The targets assign similar correct-answer probabilities under short prefixes, but standard privileged supervision sharply abandons the correct answer as erroneous text accumulates.Source: §4.1 · p. 6
• Overall accuracy across evaluated visual benchmarks improves consistently as reconstruction strength increases, alongside a larger transient response-length increase before stabilization.The response-length increase is transient; accuracy improvements are reported across the same range of reconstruction strengths.Source: §4.1 · p. 6, Figure 3 · p. 6

4.2 SUPPRESSING ERRONEOUS CONTINUATIONS ELICITS REFLECTION

• Before reflection, reconstruction gives reflection tokens a relative advantage by suppressing erroneous continuations more strongly, even as their total probability decreases.Matched positions without reflection do not show the same relative-probability increase; reflection frequency rises during training and remains elevated for stronger settings after stabilization.Source: §4.2 · pp. 6–7, Figure 4 · p. 7

4.3 REFLECTION RESTORES VISUAL RELIANCE

• After reflection, support from the accumulated student prefix decreases, followed after a short delay by increased support from visual evidence compared with matched non-reflection positions.The balance shifts toward visual evidence after self-interruption.Source: §4.3 · p. 7, Figure 5 · p. 7

5.1 PERFORMANCE ON FINE-GRAINED VISUAL REASONING

• OPD-Aha increases accuracy over Vision-OPD by 4.0% at 4B and 3.2% at 9B across six fine-grained, high-resolution, and real-world benchmarks.All OPD-Aha results use β = 4.Source: §5.1 · p. 7, Table 1 · p. 8

5.2 COMPARISON OF RECONSTRUCTION SIGNALS

• At β = 1, real–null reconstruction reaches 78.8% average accuracy, versus 76.9% for amplified teacher–student disagreement and 76.4% for the standard privileged target.The real–null signal improves results across all six benchmarks.Source: Table 2 · p. 8, §5.2 · p. 8

5.3 GENERALIZATION TO MULTIMODAL REASONING

• OPD-Aha improves over Base on all evaluated zero-shot reasoning metrics, reversing Vision-OPD’s regression below Base for the 4B student.The tasks include MathVerse, MathVista, MathVision, WeMath, and DynaMath; OPD-Aha’s positive transfer holds at both scales.Source: §5.3 · pp. 8–9, Table 3 · p. 9

• Privileged on-policy distillation combines supervision on student-generated states with training-only privileged context, extending knowledge distillation and learning with privileged information [Hinton et al., 2015; Vapnik & Vashist, 2009; Lopez-Paz et al., 2016; Chen et al., 2026a; Agarwal et al., 2024; Gu et al., 2024; Zhao et al., 2026; Jin et al., 2026; Li et al., 2026c].Related approaches include sequence-level and student-output distillation [Kim & Rush, 2016; Ko et al., 2024], behavior transfer from privileged teachers [Penaloza et al., 2026], gradient decomposition or projection [Yoon et al., 2026; Xue et al., 2026], and multimodal extensions [Bousselham et al., 2025; Yuan et al., 2026; Tian et al., 2026; Li et al., 2026b].Source: §6 · p. 9
• Reflection research spans self-feedback, reinforcement learning, visual verification, tool-based critique, and multimodal self-training, while studies report that self-correction benefits vary and can lack reliable feedback [Madaan et al., 2023; Shinn et al., 2023; Wan et al., 2025; Gou et al., 2024; Chen et al., 2024; Weng et al., 2023; Huang et al., 2024; Kamoi et al., 2024; Cheng et al., 2025; Zhong et al., 2026; Tie et al., 2025].In this paper’s setting, reflection words are not explicitly supervised; they mark self-interruptions associated with suppressing image-inconsistent continuations and renewed visual reliance.Source: §6 · p. 9

• Erroneous student prefixes can overwhelm the teacher’s visual evidence, collapsing privileged on-policy supervision when correction is most needed.A token-specific visual corrective preference survives this collapse.Source: §7 · p. 10
• OPD-Aha uses an intra-teacher contrast to isolate that surviving visual preference and reconstruct targets that suppress image-inconsistent continuations.The student learns to interrupt flawed reasoning with reflection tokens, followed by renewed visual reliance after a short delay.Source: §7 · p. 10
• OPD-Aha achieves consistent improvements across fine-grained perception and complex multimodal reasoning benchmarks.The conclusion attributes these gains to separating visual preference from confounded language priors under flawed prefixes.Source: §7 · p. 10

A THEORETICAL PROPERTIES OF TARGET RECONSTRUCTION

• With positive teacher probabilities over the vocabulary, the normalized reconstructed distribution is the unique global optimum of the KL-regularized objective.At the optimum, each token balances its visual preference against its log-probability shift from the privileged teacher.Source: A THEORETICAL PROPERTIES OF TARGET… · p. 16
• Reconstruction can reverse an erroneous teacher preference when the surviving visual separation is sufficiently strong; the required threshold rises with erroneous base preference and falls with visual separation.Later prefix states require stronger reconstruction when linguistic momentum intensifies or visual separation weakens.Source: A THEORETICAL PROPERTIES OF TARGET… · p. 16
• Increasing reconstruction strength monotonically raises expected visual alignment and departure from the privileged target, making β a control over how far the target moves along the visual-preference direction.Within the KL neighborhood of the privileged target, the reconstruction has the strongest expected visual alignment.Source: A THEORETICAL PROPERTIES OF TARGET… · p. 16

B.1 LEARNING WHEN TO REFLECT

• During training, broad reflection-token preference recedes at non-reflection states but remains elevated immediately before reflection, indicating state-specific rather than general vocabulary learning.The analysis compares trajectories that lead to reflection with comparable non-reflection trajectories.Source: §B.1 · p. 17, Figure 6 · p. 17

B.2 RECONSTRUCTED SUPERVISION DRIVES EVIDENCE-SPECIFIC RECOVERY

• Token shuffling removes most correct-option margin despite matched target magnitude or uncertainty, while irrelevant crops reduce reversal of late wrong preferences.These tests indicate that recovery depends on the corrective token direction and task-relevant visual evidence, not merely a generic target shift.Source: §B.2 · p. 17, Figure 7 · p. 18, Figure 8 · p. 18
• After reflection, OPD-Aha’s correct-option margin advantage is largest in the first 1–8 tokens, while its visual-support advantage is larger in later windows.The trends connect an early answer-margin advantage with a continuation that increasingly draws on visual evidence.Source: §B.2 · p. 18, Figure 8 · p. 18, Figure 7 · p. 18

C.1 EXPERIMENTAL SETUP

• The evaluation compares Qwen3.5 students at 4B and 9B across six fine-grained visual benchmarks, holding initialization, rollout and update budgets, and decoding fixed.Students generate from the original full image; privileged-supervision methods use the same localized evidence crop.Source: §C.1 · p. 19

C.2 TRAINING DETAILS

• A frozen copy of the initial student serves as teacher for both real-evidence and visual-null predictions, isolating the localized visual evidence’s effect under a shared prefix.The correction is distilled along the unchanged student rollout, while inference retains the student and original full image.Source: §C.2 · p. 19, Table 4 · p. 19

C.3 TARGET RECONSTRUCTION REMAINS ROBUST ACROSS SUPERVISION GEOMETRIES

• JSD achieves the highest Avg6 at both 4B and 9B, although the best per-benchmark results are distributed across divergence choices.Multiple divergences retain reconstruction gains, indicating the benefit is not tied to one supervision geometry.Source: §C.3 · p. 20, Table 5 · p. 20

C.4 LOG-PROBABILITY PRESERVES THE RELATIONAL STRUCTURE OF VISUAL EVIDENCE

• Probability-space and log-probability reconstruction both improve over the standard privileged target and yield similar aggregate gains.The log-probability form preserves the multiplicative relation between real and visual-null predictions while anchoring the privileged teacher distribution.Source: §C.4 · p. 20, Table 6 · p. 20

C.5 RECONSTRUCTED SUPERVISION IS ROBUST TO VISUAL-NULL CONSTRUCTIONS

• Reconstructed visual preference yields similar aggregate improvements with Gaussian noise, a mismatched image, a black image, or mean-color nulls.The mean-color transformation is a simple default, not a critical design choice.Source: Table 7 · p. 20, §C.5 · p. 21

C.6 TRAINING DYNAMICS

• Token-level JSD and actor gradient norm decrease sharply early in training and then stabilize, while generated-rollout accuracy initially decreases before recovering.The early interval overlaps with a transient response-length increase, so generation changes before improved rollout decisions become visible.Source: §C.6 · p. 21, Figure 9 · p. 21

C.7 QUALITATIVE CASE STUDIES

• Across three paired cases, OPD-Aha revisits uncertain interpretations and checks the diagnostic visual evidence before answering, while Vision-OPD carries its initial perceptual mistake into a confident wrong answer.The cases cover fine-grained text, spatial viewpoint, and clock reading.Source: §C.7 · p. 21
• For the poster, OPD-Aha questions its initial correct “ELY DIOCESE” reading, rechecks the lettering after considering “Diocess,” and returns to option A.Vision-OPD instead reads the title as “Ely Diocess” and selects option C.Source: Figure 10 · p. 22
• For the mailbox, OPD-Aha resolves left–right ambiguity by relating the mailbox to the woman’s body and outstretched arm, unlike Vision-OPD’s image-position reading.The reflection rechecks both the woman’s pose and mailbox position.Source: Figure 11 · p. 23
• For the distant clock, OPD-Aha distinguishes the short hour hand near XI from the long minute hand near XII and selects approximately 11:00.Vision-OPD misreads the minute hand near II and selects approximately 11:10.Source: Figure 12 · p. 24