Source-linked AI summary
OPD-V: Visual On-Policy Self-Distillation with Modality Balance
Aniri, Jinhe Bi, Peng Liao, Zengjie Jin, Volker Tresp, Fei Shen, Yunpu Ma, Tat-Seng Chua
TL;DR
Modality Imbalance can cause textual priors to dominate MLLM generation, limiting how effectively privileged information guides on-policy self-distillation. OPD-V treats Modality Balance as privileged information through Positive and Negative Teachers and a trust region, consistently improving reasoning while reducing training cost across six benchmarks and four backbones.
Problem
Existing MLLM OPSD designs overlook Modality Imbalance, raising how privileged information can remain effective when textual context dominates multimodal generation.
Method
OPD-V expresses Modality Balance through Positive and Negative Teachers and uses their logits margins to define a trust region for on-policy self-distillation.
Results
Across six benchmarks and four MLLM backbones, OPD-V consistently improves reasoning performance while reducing training cost.
Takeaways & Limitations
Modality Balance can serve as privileged information for improving visual OPSD across matched methods, architectures, and scales.
Takeaways & Limitations
The verified target is available only during training and is used solely to construct teacher supervision.
Abstract
from arXiv · showhide
On-Policy Self-Distillation (OPSD) has become a standard post-training approach for improving visual reasoning in multimodal large language models (MLLMs). Existing methods draw privileged information from diverse input sources to guide self-distillation. Yet these designs overlook Modality Imbalance, a challenge inherent to MLLM reasoning. When textual information dominates generation, the model cannot fully integrate its multimodal input. Consequently, carefully designed privileged information remains underused, limiting the effectiveness of OPSD. To examine this limitation, we construct a Positive Teacher with the Zoom-In Image and a Negative Teacher with the Mask Image, which exhibit different degrees of Modality Imbalance. Changes in their reasoning correctness and token logits reveal that Modality Balance can itself serve as privileged information. Motivated by this finding, we introduce OPD-V, a visual OPSD paradigm that instantiates such information through the Positive Teacher and Negative Teacher. Positive Modality-Balance Logits Margins define a Modality-Balance Trust Region that selects the on-policy tokens used for self-distillation. Experiments across 6 benchmarks, 4 MLLM backbones, and 5 post-training methods show that OPD-V consistently improves reasoning performance while reducing training cost.
1 Introduction
The paper identifies Modality Imbalance as a limitation of privileged information in MLLM on-policy self-distillation and proposes OPD-V, which treats Modality Balance as privileged information. OPD-V uses Positive and Negative Teachers with transformed images and a Modality-Balance Trust Region to improve reasoning while reducing training cost.
- Problem: Modality Imbalance causes textual priors to dominate generation, limiting integration of multimodal inputs and the effectiveness of privileged information.The paper defines Modality Imbalance as disproportionate reliance on textual context instead of integrating visual and textual information.
- Method: OPD-V represents Modality Balance through a Positive Teacher using the Zoom-In Image, a Negative Teacher using the Mask Image, and a student using the Original Image.The Modality-Balance Attention Ratio is lowest for the Negative Teacher, intermediate for the student, and highest for the Positive Teacher.
- Method: Higher Modality-Balance Logits Margins correspond to wider teacher attention-ratio gaps and increased student correctness, providing directional scoring for OPSD.The Positive and Negative Teachers score the same student response under matched textual context and on-policy prefixes, enabling Modality Balance to guide self-distillation.
- Results: OPD-V consistently improves reasoning performance across 6 benchmarks, 4 MLLM backbones, and 5 post-training methods while reducing training cost.Compared with standard OPSD, OPD-V also reduces step latency on both 4B and 9B backbones.
- Method: Positive-margin tokens form the Modality-Balance Trust Region used by OPD-V for Jensen–Shannon distillation.The trust region selects on-policy tokens based on tokenwise comparisons between Positive Teacher and Negative Teacher log probabilities.
2 Preliminaries
The framework studies supervised multimodal reasoning post-training, where an MLLM samples on-policy responses from an original image and textual query while verified targets provide training-only privileged information. OPSD distills a detached teacher conditioned on the verified target, and modality imbalance is quantified by relative visual attention over the generated trajectory.
- Supervised multimodal reasoning: Each training example contains an original image, textual query, and verified target available only during training.The trainable MLLM receives only the original image and query, while the target constructs teacher supervision.
- Supervised multimodal reasoning: The trainable MLLM samples an on-policy trajectory y = (y1, . . . , yT) from pθ(· | I, x).The sampled response is generated from the original image and textual query.
- On-policy self-distillation: OPSD evaluates each student-generated prefix with a detached teacher conditioned on the verified target and distills its token distribution into the student.The teacher scores the same student prefix, while gradients propagate only through the student distribution along the on-policy trajectory.
- Modality imbalance: Modality imbalance is the tendency to rely disproportionately on textual context, allowing language priors to support predictions even when visual information is required.This imbalance is assessed during token prediction, which conditions on visual input, textual query, and autoregressive prefix.
- Modality imbalance: A larger ρ(c; y) indicates stronger visual attention relative to textual attention over the on-policy trajectory.The Modality-Balance Attention Ratio measures modality imbalance without requiring equal numerical contributions from the two modalities.
3 OPD-V
OPD-V turns Modality Balance into on-policy self-distillation supervision using Positive and Negative Teachers built from Zoom-In and Mask Images. Their tokenwise logits margin defines a trust region whose selected tokens are distilled toward the Positive Teacher distribution.
- Teacher construction: OPD-V constructs a Positive Teacher from the Zoom-In Image and a Negative Teacher from the Mask Image, scoring identical student-generated tokens with matched queries and prefixes.The Zoom-In Image crops and magnifies the task-relevant region; the Mask Image replaces a random rectangular region with black pixels.
- Token selection: A positive Modality-Balance Logits Margin identifies tokens receiving greater normalized-logit support from the Zoom-In view than the Mask view.These tokens form the Modality-Balance Trust Region and are retained for self-distillation.
- Distillation objective: Within the Modality-Balance Trust Region, OPD-V uses the Positive Teacher distribution as the Jensen–Shannon distillation target for the student distribution.The positive margin also scales the selected on-policy tokens during self-distillation.
- Training cost: OPD-V adds one Negative Teacher forward pass but introduces no additional model parameters or student updates because the teachers share EMA parameters.The margin and trust region are computed tokenwise on the same on-policy tokens.
4 Experiments
Experiments across multiple backbones and six visual reasoning benchmarks show that OPD-V improves accuracy over matched baselines while reducing response length and training step time. Ablations further support complementary teacher supervision and the selected Zoom-In Image–Mask Image operation pair.
- Overall Effectiveness: OPD-V raises Qwen3.5-4B average accuracy from 64.30% to 80.01%, exceeding Vision-OPD’s 77.10% by 2.91 points across all six evaluations.The improvement holds across matched baselines, architectures, and scales.
- Overall Effectiveness: 80.01% average accuracy from a 4B model surpasses the listed 397B Qwen3.5 model at 77.44% and 1T-parameter Kimi-K2.6 at 72.81%.The passage attributes this result to using Modality Balance as privileged information.
- Computational Efficiency: OPD-V’s final-ten-step mean response length is 140.9 tokens versus 553.8 tokens for OPSD, a 74.5% reduction.OPD-V remains comparatively short and stable after the early training stage.
- Computational Efficiency: Mean step latency drops from 352 s to 240 s for Qwen3.5-4B and from 451 s to 340 s for Qwen3.5-9B, reductions of 31.8% and 24.7%.Shorter responses and more efficient privileged-input construction account for the lower cost.
- Mechanistic Validity: Combining the Positive and Negative Teachers reaches 80.01%, exceeding the stronger single-teacher variant by 5.39 percentage points and the Negative-Teacher-only variant by 8.03 points.The Negative Teacher alone reaches 71.98%, while the Positive Teacher alone reaches 74.62%.
- Mechanistic Validity: With Mask Image fixed, replacing Zoom-In Image with Repeat Image lowers accuracy from 80.01% to 75.95%; with Zoom-In Image fixed, Mask Image outperforms Blur, Prune, and No Image.The corresponding alternatives reach 74.31% for Blur, 73.74% for Prune, and 72.17% for No Image.
5 Conclusion · Appendix · A Related Work
The paper identifies Modality Balance as privileged information for OPSD and introduces OPD-V, which uses teacher-derived signals to define a Modality-Balance Trust Region. Across six benchmarks and four MLLM backbones, OPD-V improves reasoning performance while reducing training cost.
- 5 Conclusion: The work identifies Modality Balance as privileged information for OPSD.
- 5 Conclusion: OPD-V is introduced as a method built around Modality Balance.
- 5 Conclusion: OPD-V uses a Positive Teacher and a Negative Teacher to operationalize its modality-balance guidance.
- 5 Conclusion: The Positive Teacher and Negative Teacher define a Modality-Balance Trust Region.
- 5 Conclusion: OPD-V is evaluated across six benchmarks and four MLLM backbones.
- 5 Conclusion: OPD-V consistently improves reasoning performance while reducing training cost.
A.1 From SFT to On-Policy Self-Distillation · A.2 Modality Imbalance in MLLMs
The paper traces a shift from fixed-trajectory SFT to on-policy self-distillation, where supervision follows student-generated states. It then identifies modality imbalance as a source of ambiguity and motivates selecting tokens using visual-evidence balance rather than teacher confidence alone.
- A.1 From SFT to On-Policy Self-Distillation: SFT uses teacher-forced reference responses, whereas knowledge distillation replaces single reference tokens with teacher distributions.SFT supervision remains tied to fixed reference trajectories rather than states produced by the current student.
- A.1 From SFT to On-Policy Self-Distillation: On-Policy Distillation applies distributional supervision to trajectories sampled from the student model.The student first samples its own trajectory before receiving teacher-based supervision.
- A.1 From SFT to On-Policy Self-Distillation: OPSD preserves on-policy supervision while replacing the external teacher with a detached copy of the same model.The teacher scores each student-generated prefix, and the loss updates the student distribution at the visited state.
- A.1 From SFT to On-Policy Self-Distillation: OPD-V training samples rollouts from pθ(· | I, x) and constructs a zoomed image plus its masked version for each training example.The algorithm obtains Izoom from the training data and defines Imask as Mask(Izoom).
- A.2 Modality Imbalance in MLLMs: Modality imbalance is the disproportionate reliance on textual context over visual input during MLLM response generation.Prior studies find textual information can dominate prediction even when the query requires visual evidence.
- A.2 Modality Imbalance in MLLMs: Teacher confidence alone can select tokens whose supervision is driven mainly by textual context rather than visual evidence.OPD-V therefore makes modality balance part of self-distillation, selecting tokens according to stronger visual-evidence use.
B OPD-V Algorithm · B.1 Loss and Top-K Distillation Details
OPD-V aligns Positive and Negative Teacher scoring with shared on-policy prefixes, using the Positive Teacher for Jensen–Shannon distillation and the Negative Teacher to define a Modality-Balance Trust Region. B.1 further restricts distillation to positively balanced tokens and reduces memory through tail-adjusted top-K distributions.
- B OPD-V Algorithm: Each prompt produces n on-policy rollouts, with teacher scoring aligned to the same visited prefixes.At token position t, the student and teacher distributions are computed under the shared prefix.
- B OPD-V Algorithm: The Positive Teacher supplies the Jensen–Shannon distillation target, while the Negative Teacher provides the paired comparison for the Modality-Balance Trust Region.Only generated response tokens with r_t = 1 enter the loss, and the detached teacher is updated by EMA after the student update.
- B.1 Loss and Top-K Distillation Details: A positive Logits Margin selects positions where the Positive Teacher supports the student-generated token more than the Negative Teacher.These selected positions constitute the Modality-Balance Trust Region.
- B.1 Loss and Top-K Distillation Details: Within the Modality-Balance Trust Region, δ_MB,t scales the selected on-policy tokens during self-distillation.The distillation target remains distributional, comparing Positive Teacher and student distributions rather than only their probabilities for y_t.
- B.1 Loss and Top-K Distillation Details: To reduce MLLM memory use, implementation first selects the student top-K token set at each response position.Full-vocabulary logit distillation is memory intensive, motivating the top-K restriction.
- B.1 Loss and Top-K Distillation Details: The Positive Teacher is evaluated on the same K_t, retaining top-K probabilities and aggregating the remaining vocabulary mass into one tail bucket.This construction applies to each distribution s ∈ {p_θ,t, q^+_t}.
- B.1 Loss and Top-K Distillation Details: The resulting (K + 1)-dimensional representation preserves probability mass outside K_t without materializing the full vocabulary.The Positive Teacher and student distributions use tail-adjusted top-K representations for the per-token distillation term.
- B.1 Loss and Top-K Distillation Details: Loss reduction normalizes by the valid generated response-token count, while δ_MB,t controls the strength of the selected distillation signal.The Negative Teacher enters only through δ_MB,t, whereas the Positive Teacher remains the distillation target.
B.2 Image Operations
OPD-V varies only the teacher image input while holding the textual query, on-policy prefix, and teacher parameters fixed. The Positive Teacher uses an evidence-centered Zoom-In Image, whereas the Negative Teacher uses a randomly masked version of that crop.
- Image Operations: OPD-V changes only the teacher image input, keeping the textual query, on-policy prefix, and teacher parameters fixed.Both teacher images derive from the same Original Image containing the target-region cue.
- Image Operations: The Positive Teacher receives Izoom, an evidence-centered crop defined by the bounding box around the task-relevant region and magnified before processing.In experiments, Izoom comes from the evidence-centered crop supplied with the Vision-OPD training data.
- Image Operations: The Negative Teacher receives Imask = Mask(Izoom), formed by replacing a randomly selected rectangular region of Izoom with black pixels.This creates a visually weakened condition while preserving most image context.
C Training Prompt Example · D Model Configuration Details
The appendix specifies the training-prompt structure and clarifies the model configurations and data construction used by OPD-V. It also distinguishes training-time image conditions from the inputs used during evaluation.
- C Training Prompt Example: Figure 7 shows a training sample containing the prompt image, image placeholder, question, visual-focus instruction, answer choices, output format, and ground-truth answer.The figure uses the same field structure as the training prompt.
- C Training Prompt Example: The left column identifies which image condition accompanies the shared textual prompt during student and teacher passes.This links the prompt example to the distinct image conditions used in training.
- D Model Configuration Details: Table 3 organizes the four MLLM backbones by model identifier, parameter scale, and VLM/perception stack.The table makes the backbone comparison explicit.
- D Model Configuration Details: The configurations compare Qwen3.5, which uses early vision–text fusion, with Qwen3-VL, which uses Interleaved-MRoPE with DeepStack.These designs are presented as the distinguishing VLM/perception stacks.
- D Model Configuration Details: 6241 visual reasoning samples comprise the training split from the referenced dataset.Each sample includes an Original Image, textual query, verified target, and the dataset field used as Izoom.
- D Model Configuration Details: The Mask Image is generated during training from the evidence-centered cropped Original Image Izoom.This construction defines the training-time masked-image condition.
- D Model Configuration Details: Evaluation uses only the Original Image and textual query.The evaluation input conditions therefore differ from the training sample fields that include Izoom-derived masking.
E Evaluation Protocol Details
Evaluation uses deterministic normalization with a Yes/No judge for unresolved cases, while reporting an unweighted Average across six benchmark accuracies. The benchmark-level mean is distinct from sample-weighted accuracy pooled over 32162 examples.
- Evaluation Protocol: Evaluation runs in non-thinking mode, normalizes responses deterministically, and sends unresolved cases to a Yes/No judge.Multiple-choice responses are checked by direct option matching and first-letter option matching before judging.
- Aggregation: The reported Average is the unweighted mean over the six benchmark accuracies.Table 2 pairs each benchmark with its sample count and scoring rule.
- Aggregation: The benchmark-level mean differs from sample-weighted overall accuracy, which pools correct predictions over all 32162 examples before computing one percentage.These are separate aggregation procedures for summarizing benchmark performance.
F Hyperparameters and Compute
The training configuration uses direct mini-batch actor updates with eight responses per prompt, separates context and rollout limits, and specifies teacher updates, correction, checkpointing, and logging behavior.
- Training scale: Batch size and actor update mini-batch size are both 48, with n = 8 on-policy responses sampled per prompt.Each step uses the sampled mini-batch directly for the actor update, and the loss is reduced by n|B|.
- Sequence and candidate limits: The prompt limit is 8192 tokens, the response limit is 1024 tokens, and the distillation candidate set uses K = 100.These limits bound input context and rollout length, while K fixes the tail-adjusted distillation distribution candidate size.
- Teacher updates and operations: The EMA teacher update uses τ = 0.05, rollout correction uses threshold 2.0, and checkpointing and validation are disabled with −1 settings.The detached teacher update retains 0.95 of previous teacher parameters, while offline W&B logging records run statistics without online synchronization.