PaperlayerView exact revision on arXiv

Source-linked AI summary

LEGO-OPD: Factorized Teacher Composition for Multimodal On-Policy Distillation

Jaeyun Shin, Hangeol Chang, Jong Chul Ye

arXiv:2610.00333v1cs.CVcs.AIcs.CLcs.LG

TL;DR

Multimodal on-policy distillation must improve visual grounding while preserving language reasoning, but complete teacher predictions can entangle grounding with language priors and stronger visual supervision can impair reasoning. LEGO-OPD composes a language prior with a visual likelihood estimate and adaptively calibrates the visual update at each prefix. Across Qwen3 evaluations, it outperforms the evaluated OPD baselines on multimodal and text-only reasoning tasks.

  • Problem

    Combining complete VLM predictions entangles visual grounding with language priors, while stronger visual supervision can improve perception but overemphasize visual evidence and degrade reasoning.

    Source: §1 · p. 1

  • Method

    LEGO-OPD composes an LLM language prior with a VLM visual likelihood estimate through generalized Bayesian updating and calibrates the visual update at each prefix using the grounding expert’s image-induced prediction shift.

    Source: §1 · pp. 1–2

  • Results

    LEGO-OPD consistently outperforms evaluated OPD baselines on multimodal and text-only reasoning tasks, and adaptive calibration strengthens visual grounding while preserving text-only reasoning.

    Source: §1 · p. 2

  • Takeaways & Limitations

    In the evaluated Qwen3 setting, factorized composition and adaptive calibration support stronger visual perception without sacrificing text-only reasoning performance.

    Source: §1 · p. 2

    The visual likelihood estimate is model-dependent because its image-free prediction is obtained by removing the image rather than marginalizing over images.

    Source: §3.2 · p. 5

Abstract

from arXiv · show

Multimodal on-policy distillation (OPD) aims to improve visual grounding while preserving the strong reasoning capabilities of language models. Recent multi-teacher approaches combine LLM and VLM teachers to provide complementary supervision. However, directly using a VLM's full predictive distribution entangles its visual grounding signal with its own language prior, preventing the grounding information from being transferred independently. Conversely, increasing the strength of visual supervision can improve perception but may overemphasize visual evidence and degrade language reasoning. To address this trade-off, we introduce LEGO-OPD, which selectively composes factors from a Language Expert and a Grounding expert into One teacher distribution for multimodal OPD. Under a generalized Bayesian formulation, the language expert provides a prior over candidate tokens, while the grounding expert contributes a visual likelihood that updates this prior, rather than transferring its complete predictive distribution. This factorized composition allows language reasoning and visual grounding to be controlled independently. We further introduce adaptive calibration to determine how strongly the visual likelihood should update the language prior at each decoding prefix. Specifically, LEGO-OPD uses the grounding expert's image-induced prediction shift as a prefix-dependent reference, preventing both insufficient and excessive visual supervision. Experiments with Qwen3 models show that LEGO-OPD consistently outperforms the evaluated single- and multi-teacher OPD baselines on both multimodal and text-only reasoning tasks. Moreover, it improves the initial student's visual perception while preserving text-only reasoning.

1 INTRODUCTION

Multimodal on-policy distillation needs to combine reliable visual grounding with reasoning over grounded information, but existing teacher combinations can entangle these signals or overemphasize visual evidence. LEGO-OPD composes a language prior with a calibrated visual factor, and Qwen3 experiments report improvements across multimodal and text-only reasoning tasks.

Introduction

  • Multimodal OPD distills teacher predictions at student-generated prefixes and requires both reliable visual grounding and reasoning over grounded information.Source: §1 · p. 1
  • LEGO-OPD consistently outperforms evaluated OPD baselines on multimodal and text-only reasoning tasks, while adaptive calibration strengthens visual grounding without sacrificing text-only reasoning.Ablations indicate that both factorized teacher composition and adaptive calibration are essential to the reported improvements.Source: §1 · p. 2
  • The method composes a language prior from an external LLM with a visual likelihood estimate from a VLM, using generalized Bayesian updating to form one teacher distribution.The composed teacher supervises the student through standard reverse-KL on-policy distillation.Source: §1 · p. 2
  • Source-referenced calibration adapts the visual update at each prefix using the grounding expert’s image-induced prediction shift, while a global scale controls overall visual emphasis.This addresses the difficulty of independently transferring complementary strengths and the risk that excessive visual supervision may degrade reasoning.Source: §1 · pp. 1–2

2 RELATED WORKS AND BACKGROUND

Related work combines specialized teachers or strengthens visual supervision, while prediction-space composition methods typically use fixed coefficients. The background describes generalized Bayesian updating and on-policy distillation at student-generated prefixes.

2.1 RELATED WORK

  • Existing multimodal distillation approaches route examples or mix complete teacher predictions, but mixing predictions can entangle a VLM’s language prior and visual grounding signal.Other work strengthens visual supervision, with reported degradation under excessive visual emphasis motivating balance between visual and language-prior supervision.Source: §2.1 · p. 3
  • Prediction-space methods transfer signals across models but use fixed composition coefficients, leaving suitable strengths to predefined settings or empirical tuning.Source: §2.1 · p. 3
  • LEGO-OPD composes complementary expert factors and adapts the visual update at each prefix rather than mixing complete predictions with a fixed strength.Source: §2.1 · p. 3

2.2 GENERALIZED BAYESIAN UPDATING

  • Generalized Bayesian updating incorporates new information into a prior through an observation-dependent loss, equivalently represented here by an evidence score whose higher values indicate greater support.The evidence scale controls influence relative to the prior; when the score is a model log-likelihood and the scale is 1, the update recovers that model’s Bayesian posterior.Source: §2.2 · p. 3

2.3 ON-POLICY DISTILLATION

  • On-policy distillation minimizes reverse KL between student and teacher distributions at prefixes generated by the student, with prefixes and teacher targets fixed during each update.Source: §2.3 · p. 3

3 METHOD

LEGO-OPD uses a language expert’s textual-context distribution as a prior and a grounding expert’s image-induced prediction change as a visual factor. Generalized Bayesian updating composes these factors, while source-referenced calibration adapts the visual update at each prefix.

3.1 LEGO-OPD: MULTIMODAL OPD WITH FACTORIZED TEACHER COMPOSITION

  • The language expert supplies a textual-context prior, while the grounding expert’s image-conditioned versus image-free predictions provide a visual likelihood estimate for a unified teacher used in reverse-KL on-policy distillation.The image-free distribution is obtained by removing the image input, not by marginalizing over images, so the ratio is a model-dependent proxy for visual evidence rather than necessarily a true likelihood.Source: §3.2 · pp. 4–5, §3.1 · p. 4

3.2 LEGO-OPD: THEORETICAL BACKGROUND

  • Because independently sourced factors need not form an exact Bayesian posterior, generalized Bayesian updating treats the language distribution as prior and the log visual-likelihood estimate as evidence.The evidence scale controls the visual update relative to the language prior.Source: §3.2 · p. 5
  • Source-referenced calibration chooses the update scale so the composed teacher’s KL divergence from the language prior matches a scaled reference based on the grounding expert’s image-induced shift.For a nonconstant residual, a finite positive solution exists when the requested KL budget is within its attainable range.Source: §3.2 · p. 5
  • A second-order local approximation around zero update yields a practical coefficient instead of requiring a root search at every generated position.For a fixed KL budget, greater residual variance under the language prior gives a smaller coefficient; finite updates can miss the requested budget because the approximation is local.Source: §3.2 · pp. 5–6
  • On shared prefixes from 4B experts, approximate coefficients broadly track numerical solutions across source-KL intervals, while teacher KL matches the source KL for small updates but exceeds it for larger updates.The analysis used 24,576 prefixes from 768 rollouts on 256 MMMU questions at student steps 0, 100, and 200.Source: §3.2 · p. 6, Figure 2 · p. 6
  • The reverse-KL objective decomposes student–teacher alignment into language-prior and visual-residual contributions, with the grounding residual scaled by the adaptive coefficient.Source: §3.2 · p. 6

4 EXPERIMENTS

LEGO-OPD outperforms the evaluated OPD baselines across multimodal and text-only reasoning averages, and its adaptive calibration outperforms tested fixed coefficients. Expert analyses find complementary contributions: the language prior supports reasoning across modalities, while the visual residual adds multimodal gains with little change in text-only performance.

4.1 EXPERIMENTAL SETUP

  • The experiments use Qwen3-VL-2B-Instruct as student, Qwen3-4B-Instruct-2507 as language expert, and Qwen3-VL-4B-Instruct as grounding expert.Training uses a 54,840-example MMFineReason-123K subset; evaluation covers multimodal reasoning, text-only reasoning, and visual perception.Source: §4.1 · pp. 6–7
  • Baselines include standard OPD and VA-OPD with a VLM teacher, plus M-OPD and H-OPD with both VLM and LLM teachers.Comparisons match model scale, optimization settings, and response budget whenever compatible while preserving each method’s defining supervision structure.Source: §4.1 · p. 7

4.2 MAIN RESULTS

  • At ρ = 1 and ρ = 2, LEGO-OPD outperforms evaluated OPD baselines in multimodal and text-only reasoning averages.Both settings also outperform H-OPD on all six multimodal reasoning benchmarks using the same expert pair.Source: §4.2 · p. 7, Table 1 · p. 7
  • Adaptive calibration at ρ = 1 outperforms all tested fixed coefficients in both reasoning averages without empirical tuning of a fixed coefficient.Fixed-coefficient performance varies across benchmarks, and adaptive-calibration gains are larger in multimodal than text-only reasoning.Source: §4.2 · p. 8, Table 2 · p. 8
  • With fixed coefficients, increasing α improves perception but reduces text-only reasoning performance.Increasing α from 2 to 4 also markedly lowers the multimodal reasoning average; across ρ ∈ {1, 2, 4}, larger budgets improve perception relative to ρ = 1 while maintaining or improving both reasoning averages.Source: §4.2 · p. 8

4.3 ANALYSIS OF LANGUAGE AND GROUNDING EXPERTS

  • The full composition combines the language prior’s cross-modal reasoning support with the visual residual’s additional multimodal gains and little change in text-only performance.The passage describes this pattern as supporting joint transfer of language reasoning and visual grounding to the student.Source: §4.3 · pp. 8–9, Table 3 · p. 9
  • With the 4B grounding expert fixed, replacing the base 4B LLM with the 1.7B LLM lowers both text-only and multimodal reasoning scores.With the 4B language expert fixed, scaling the grounding model from 2B to 4B to 8B improves the multimodal reasoning average while text-only averages remain nearly unchanged.Source: §4.3 · p. 9, Table 4 · p. 9

5 CONCLUSION

LEGO-OPD composes a language prior and visual grounding signal into a unified teacher for multimodal on-policy distillation, with source-referenced calibration controlling visual contribution at each student-generated prefix. Across Qwen3 models, it achieves the best average performance among evaluated single- and multi-teacher OPD baselines on multimodal and text-only reasoning.

5 CONCLUSION

  • LEGO-OPD composes a language prior and visual grounding signal into a unified teacher for multimodal on-policy distillation.A generalized Bayesian formulation enables factorized teacher composition, while source-referenced calibration controls visual contribution at each student-generated prefix.Source: §5 · p. 9
  • Across Qwen3 models, LEGO-OPD achieves the best average performance among evaluated single- and multi-teacher OPD baselines on multimodal and text-only reasoning.Ablations show complementary benefits from the language prior and visual residual, and adaptive calibration outperforms fixed weighting in both reasoning averages.Source: §5 · p. 9
  • LEGO-OPD provides an effective framework for balancing language reasoning and visual grounding in multimodal distillation.Adaptive calibration allows stronger visual emphasis without degrading reasoning, as reported in the conclusion.Source: §5 · p. 9

REPRODUCIBILITY STATEMENT

The reproducibility statement maps the paper’s experimental, theoretical, implementation, and calibration details to sections and appendices, and reports that code is available.

REPRODUCIBILITY STATEMENT

  • Section 4.1 describes the experimental models, training data, optimization settings, and evaluation benchmarks.Appendix C details training settings, evaluation protocols, prompt templates, and baseline implementations.Source: REPRODUCIBILITY STATEMENT · p. 10
  • Appendix A provides theoretical analysis and proofs, while Appendix B specifies practical teacher construction and the student update.Appendix D gives configurations for calibration ablations and diagnostics.Source: REPRODUCIBILITY STATEMENT · p. 10
  • The paper states that its code is available at https://anonymous.4open.science/r/LEGO_OPD-51C1.Source: REPRODUCIBILITY STATEMENT · p. 10

AI USE STATEMENT

The appendix documents the authors’ use of generative AI and analyzes the teacher composition and calibration rule. It characterizes the composed teacher variationally and specifies when exact KL-budget calibration exists.

AI USE STATEMENT

  • AI tools supported literature review, formulation, experiments, implementation, and writing; authors retained research decisions, checked arguments and code, and used AI for semantic answer interpretation.AI-assisted suggestions were examined rather than treated as authoritative outputs.Source: AI USE STATEMENT · p. 10

A THEORETICAL ANALYSIS

  • The theoretical analysis assumes a finite vocabulary and strictly positive language, image-conditioned, and image-free distributions at a fixed student-generated prefix.It uses natural logarithms and the convention 0 log 0 = 0 for KL divergences.Source: A THEORETICAL ANALYSIS · p. 14

A.1 TEACHER COMPOSITION

  • The composed teacher has a variational characterization, and its probability-space and logit-space forms are algebraically equivalent up to token-independent constants.The identity does not require the image-free prediction to equal a marginal over images.Source: §A.1 · p. 14

A.2 EXACT KL CALIBRATION: EXISTENCE AND UNIQUENESS

  • For a nonconstant residual, the KL displacement increases continuously and strictly with its coefficient, allowing a unique finite positive calibration solution exactly when 0 < b < K∞.The limiting displacement is K∞ = −log qL(M); the zero-budget solution is the language prior.Source: §A.2 · pp. 14–15
  • With a finite coefficient cap, attainable budgets form [0, K(αmax)]; requests above that range return the cap and remain unmet.The source-referenced budget specifies the target; it does not follow from constrained optimality.Source: §A.2 · p. 15

A.3 LOCAL APPROXIMATION AND ERROR

The appendix bounds the local KL-budget approximation and describes special-case invariances, the loss decomposition, and practical residual stabilization. The approximation is local and need not match the requested budget at observed prefixes.

A.3 LOCAL APPROXIMATION AND ERROR

  • The local coefficient approximation follows a second-order expansion around zero, but its error is not uniformly small across prefixes and it may overshoot or undershoot the budget.The approximation statements hold for a fixed prior and residual; no finite exact coefficient exists when the budget is at least K∞.Source: §A.3 · p. 16

A.4 SPECIAL CASES AND INVARIANCES

  • At zero source KL, exact calibration recovers the language prior; a positive rescaling of the residual rescales the unique calibrated coefficient while preserving the teacher.A token-independent residual shift leaves the teacher and its KL displacement unchanged at every coefficient.Source: §A.4 · pp. 16–17

A.5 LOSS DECOMPOSITION

  • The loss decomposes into language-prior and visual-residual contributions, but correspondence with a detached visual target does not by itself establish gradient equivalence.The target holds the student prior fixed, whereas the compared objective differentiates through it.Source: §A.5 · p. 17

B PRACTICAL TEACHER CONSTRUCTION AND STUDENT UPDATE

B.1 VISUAL RESIDUAL STABILIZATION

  • Residual flooring reduces extreme low-probability ratios, while protection preserves contrast for selected image-favored tokens and centering preserves the normalized teacher.Protection applies to tokens in the image-conditioned 0.99-mass nucleus that increase with the image and whose image-free logits fall below the floor.Source: §B.1 · p. 18

B.2 CALIBRATION COEFFICIENT ESTIMATION

The practical calibration estimate uses a shared high-probability support and a regularized, capped coefficient. The teacher remains normalized over the full vocabulary, and the student update uses a separate student-selected support.

B.2 CALIBRATION COEFFICIENT ESTIMATION

  • Calibration uses the union of three teacher nuclei, retaining at least 0.95 probability mass per branch, to estimate the source contrast and residual variance.The retained mass bounds excluded probability mass but does not guarantee closeness to the full-vocabulary source KL.Source: §B.2 · p. 18
  • The coefficient is a regularized, capped second-order estimate, not an exact KL-budget solution; support restriction and residual stabilization can also change its displacement.The cap limits amplification when the estimated variance is small, but cannot make an unattainable KL target attainable.Source: §B.2 · pp. 18–19

B.3 TEACHER NORMALIZATION AND STUDENT UPDATE

  • Teacher normalization spans the full vocabulary, while the update selects 16 tokens using detached student scores and renormalizes only their weights.Teacher-derived support is used for coefficient estimation; student-derived support selects update terms, with gradients flowing only through the final student log-probability factor.Source: §B.3 · p. 19

C.1 MODELS AND TRAINING

The experiments use a shared on-policy training setup and evaluate multimodal and text-only reasoning separately from visual perception. Baselines retain the common configuration where compatible, with method-specific adaptations.

C.1 MODELS AND TRAINING

  • The common baseline configuration follows LEGO-OPD where compatible, while each baseline retains its method-specific objective and hyperparameters.Training and sampled-reasoning evaluation settings are held fixed across compared methods; deviations are described separately.Source: Table 5 · p. 20, §C.5 · p. 23

C.2 BENCHMARKS AND METRICS

  • Evaluation covers six multimodal reasoning benchmarks, five text-only reasoning benchmarks, and separate BLINK and QBench1,2 perception evaluations.The six and five reasoning scores are averaged within their groups; perception benchmarks are excluded from both reasoning averages.Source: §C.2 · p. 21

C.3 EVALUATION PROTOCOL

  • Reasoning uses sampled decoding, whereas perception uses greedy direct-answer generation under separate evaluation protocols.Reasoning scores average five sampled responses per question; perception uses one response per question.Source: §C.3 · p. 21, §C.3.1 · p. 21, §C.3.2 · p. 21

C.3.1 REASONING BENCHMARKS

  • Reasoning prompts preserve benchmark-specific formats, with MMLU-Pro using category-conditioned five-shot prompts without the shared system prompt.The main reasoning results report mean accuracy across five evaluation seeds, with standard deviations provided in the appendix.Source: §C.3.1 · p. 21

C.3.2 PERCEPTION BENCHMARKS

  • Perception evaluation uses each benchmark’s prompt format and exact-match option scoring, counting responses without an extractable answer letter as incorrect.BLINK accuracy covers 1,901 validation questions and QBench1,2 covers 2,495 pooled Group A questions.Source: §C.3.2 · p. 21, Table 6 · p. 22

C.4 PROMPT TEMPLATES

  • Figures 3 and 4 provide the shared QA template and gold-blind semantic option-mapping prompt, while benchmark-specific and perception settings remain distinct.The option mapper sees only the extracted answer and candidate options.Source: §C.4 · p. 22, Figure 3 · p. 22, Figure 4 · p. 22

C.5 BASELINE IMPLEMENTATIONS

  • Baseline implementations adapt model configuration, data, and training budget while retaining reported objectives and method-specific hyperparameters when possible.The compared baselines include direct VLM distillation, visual-importance weighting, routed teacher supervision, and confidence-weighted teacher combination.Source: §C.5 · p. 23

D ANALYSIS OF SOURCE-REFERENCED CALIBRATION

  • The calibration analysis examines fixed versus adaptive coefficients, zero visual-update budgets, support-mass sensitivity, coefficient approximation, and recomposed-teacher distillation.The appendices evaluate calibration behavior and its practical implementation across these distinct settings.Source: D ANALYSIS OF SOURCE-REFERENCED… · p. 23

D.1 A THREE-TOKEN CALIBRATION EXAMPLE

The appendix tests how source-referenced calibration behaves across prefixes and implementation choices. Its examples and analyses show why a single coefficient can mis-scale visual updates, while practical recomposition achieves similar aggregate performance to direct VLM supervision.

D.1 A THREE-TOKEN CALIBRATION EXAMPLE

  • The example’s reasoning and grounding prefixes require coefficients that differ by more than fourfold, so a fixed coefficient cannot suit both.Increasing a fixed coefficient to 2 improves the grounding prefix but raises the reasoning-prefix displacement to 60 times the source KL; adaptive calibration ties displacement to the grounding expert’s image-induced change.Source: §D.1 · pp. 23–24

D.2 FIXED AND ADAPTIVE CALIBRATION

  • Fixed-composition ablations use constant coefficients, while adaptive variants vary the KL-budget scale ρ under otherwise retained settings.The comparison uses the same 2B student and 4B language and grounding experts.Source: §D.2 · p. 25, Table 7 · p. 25

D.3 LANGUAGE-ONLY SUPERVISION AT ZERO KL BUDGET

  • Language-only controls show that text-only reasoning gains do not by themselves ensure multimodal transfer, while a frozen student’s visual residual can contribute without a stronger grounding model.The controls compare multimodal and text-only data at ρ = 0 with ρ = 1 expert-composition configurations.Source: §D.3 · pp. 25–26, Table 8 · p. 26

D.4 SENSITIVITY TO CALIBRATION SUPPORT MASS

  • 45.84 versus 45.93 for multimodal reasoning and 56.01 versus 56.02 for text-only reasoning, support masses 0.95 and 0.99 produced similar final averages.Training diagnostics were broadly similar, with some differences in the top-k KL estimate; the results indicate limited sensitivity in this configuration.Source: §D.4 · pp. 26–27, Table 9 · p. 27, Figure 6 · p. 27

D.5 APPROXIMATION ACCURACY OF THE PRACTICAL COEFFICIENT

  • The practical coefficient broadly tracks a numerical KL-matching solution across source-KL intervals, with larger deviations at larger source KL.The analysis uses 24,576 prefixes from student rollouts; binned means can agree even when individual prefixes deviate.Source: §D.5 · p. 28, Figure 7 · p. 28

D.6 DISTILLATION WITH A RECOMPOSED VLM TEACHER

  • 40.18 vs. 40.37 for multimodal reasoning and 51.18 vs. 51.72 for text reasoning, the recomposed teacher achieved similar aggregate performance to direct VLM supervision.The comparison combines coefficient approximation and residual stabilization, so it cannot isolate their separate training effects.Source: Table 10 · p. 29, §D.6 · p. 29

E.1.1 SAMPLED DECODING

The appendix reports sampled- and greedy-decoding evaluations, expert-combination settings, and a separate reproduction attempt. Greedy decoding places LEGO-OPD at ρ = 2 highest among evaluated 2B students on both reasoning averages; the adapted Decomposed OPD implementation scores below Standard OPD on both.

E.1.1 SAMPLED DECODING

  • Sampled-decoding results report accuracies as means and standard deviations across five evaluation sampling seeds for the distilled 2B students.The table includes the initial 2B student and original 4B VLM reference.Source: Table 11 · p. 30, §E.1.1 · p. 30

E.1.2 GREEDY DECODING

  • Under greedy decoding, LEGO-OPD at ρ = 2 achieves the highest multimodal and text reasoning averages among evaluated 2B students.Table 12 marks the best score among 2B students, excluding the 4B reference.Source: §E.1.2 · p. 30, Table 12 · p. 31

E.2 EXPERT CONFIGURATIONS AND FACTOR CONTRIBUTIONS

  • Expert-combination evaluations train the same 2B student for 200 steps at ρ = 1 while varying the language or grounding expert.Configurations include substituting the frozen initial student for either factor; the supplied results description does not state specific winners or values.Source: §E.2 · p. 31, Table 13 · p. 32

E.3 DECOMPOSED OPD REPRODUCTION ATTEMPT

  • The paper-based Decomposed OPD implementation scored below Standard OPD on both multimodal and text-only benchmarks in the adapted setup.This result does not establish the original method’s performance under its original experimental conditions.Source: §E.3 · p. 32, Table 14 · p. 33

F LIMITATIONS

The limitations concern visual-evidence reliability and the interpretation and compatibility of the visual residual.

F LIMITATIONS

  • Calibration measures the magnitude of a grounding expert’s visual update, not its correctness, so erroneous evidence can be amplified.The method does not determine whether the visual evidence is reliable.Source: F LIMITATIONS · p. 33
  • The visual residual is a conditional model contrast, not an isolated visual factor, because prefixes may already contain image-inferred information.The implementation also requires a shared tokenization and output vocabulary, restricting composition to compatible models within the same family.Source: F LIMITATIONS · p. 33

Related papers

Loading 2610.00333v1…