PaperlayerView exact revision on arXiv

Source-linked AI summary

Where-OPD: Spatially Guided On-Policy Self-Distillation of MLLMs with Synthetic Scenes

Sophia Sirko-Galouchenko, Monika Wysoczanska, Andrei Bursuc, Nicolas Thome, Spyros Gidaris

arXiv:2610.02117v1cs.CVcs.AIcs.CLcs.LG

TL;DR

MLLM on-policy self-distillation remains relatively unexplored, and prior visual-privilege approaches report gains concentrated in tasks suited to zooming. Where-OPD trains with procedural scene-derived spatial hints for the teacher and the image and question for the student. It improves task-specific benchmarks and transfers to real-world perception benchmarks despite synthetic-only post-training.

  • Problem

    How to use privileged information for MLLM visual perception remains underexplored, while prior approaches based on improved visual access do not necessarily transfer uniformly across visual tasks.

    Source: §1 · p. 2

  • Method

    Where-OPD uses procedurally generated scenes to give the teacher textual hints locating question-relevant visual elements, while the student learns from the image and question alone.

    Source: §1 · pp. 2–3

  • Results

    Where-OPD improves counting, reading, and chart benchmarks and transfers to real-world perception, with average gains of 3.23, 1.07, and 1.29 points across seven benchmarks for the three MLLMs.

    Source: §1 · pp. 2–3

  • Takeaways & Limitations

    Ablations indicate that spatial grounding matters: removing the hint or providing the answer without locations yields lower performance.

    Source: §5 · p. 9

Abstract

from arXiv · show

On-policy self-distillation has recently emerged as an effective approach for improving language-model reasoning by supervising students with a frozen or EMA version of themselves that receives privileged information. Its application to multimodal large language models (MLLMs), however, remains largely unexplored. Recent approaches use privileged visual information, such as image crops corresponding to a question, to improve fine-grained perception, but their gains are confined to tasks that benefit from such visual zooming and require either human-annotated grounding data or external teacher models. We introduce a different form of on-policy self-distillation for MLLMs that provides the teacher with textual, spatially grounded guidance identifying the visual elements relevant to a query. We use procedurally generated scenes with automatically available object identities and spatial coordinates, enabling scalable and annotation-free post-training. The teacher uses this spatial guidance to locate and integrate evidence from multiple relevant image regions, while the student learns to reproduce the resulting behavior from the image and question alone. Our approach consistently improves performance on counting, document and chart understanding benchmarks across multiple models. Importantly, although post-training uses only synthetic scenes, the resulting improvements transfer to real-world perception benchmarks, yielding a 3.23-point gain in average performance across CVBench, V*, ZoomBench, BLINK, HR-Bench, and MME-RealWorld. These results show that spatially grounded privileged information can induce broader perceptual capabilities through on-policy self-distillation, enabling substantial synthetic-to-real transfer beyond the task and data distribution used for post-training. Project page: https://github.com/sirkosophia/Where-OPD

1 INTRODUCTION

The introduction asks what privileged information can help MLLMs use visual evidence beyond tasks suited to zoomed views. Where-OPD gives a teacher spatial text hints from procedural scenes, and reports improvements across task-specific and real-world visual benchmarks.

  • On-policy self-distillation uses a frozen or EMA copy of the student as a teacher with privileged information unavailable to the student.The introduction frames the open question as which privileged information can support MLLM visual perception and what capabilities that advantage can transfer.Source: §1 · p. 2
  • Prior MLLM self-distillation methods often give teachers cropped, zoomed, or otherwise altered views, with strong gains on localized perception but uneven transfer across tasks.Vision-OPD improves Qwen3.5-4B on V* by 8.90 points and ZoomBench by 14.56 points, but decreases CountQA by 10.73 points.Source: §1 · p. 2
  • The proposed guidance identifies relevant visual elements and their image locations, addressing tasks that require combining evidence distributed across an image.The introduction names finding all query-matching instances, collecting text from different regions, and associating chart labels, values, and graphical elements.Source: §1 · pp. 2–3
  • Procedurally generated scene metadata supplies question-relevant location hints without pre-existing datasets, human annotations, or external teacher models.During post-training the teacher receives the hint alongside the image and question; the student receives only the image and question and operates without hints at inference.Source: §1 · pp. 2–3
  • Where-OPD improves counting, reading, and chart benchmarks, while real-world benchmark averages rise 3.23, 1.07, and 1.29 points for the three MLLMs.The real-world averages cover CVBench, V*, HR-Bench 4K, HR-Bench 8K, ZoomBench, MME-RealWorld, and BLINK.Source: §1 · pp. 2–3

2 RELATED WORKS

Related work addresses MLLM visual reasoning through changes to pretraining, decoding, instruction-tuning data, and post-training. Recent on-policy self-distillation methods differ in how they create teacher–student asymmetry; Where-OPD uses procedurally generated, spatially grounded text hints.

Improving visual reasoning in MLLMs

  • Visual reasoning work spans pretraining architectures, decoding objectives, instruction-tuning data, and post-training; GRPO-style multimodal methods can provide sparse sequence-level rewards.The related-work discussion motivates token-level on-policy self-distillation as denser supervision on how responses use visual input.Source: §2 · p. 3

Visual On-policy Self-Distillation

3 METHOD

Where-OPD gives an on-policy teacher textual spatial hints while the student sees only the image and question. Procedurally generated scenes supply the hints, and the training objective distills the teacher’s predictions on student-visited states.

3 METHOD

  • The framework pairs a student receiving only the image and question with a teacher receiving automatically generated spatial hints, then distills the teacher’s predictions.The hints identify question-relevant elements and their locations; they are used during post-training only.Source: §3 · p. 4, §3.2 · p. 5, §3.2.2 · p. 6

3.1 PRELIMINARIES: ON-POLICY SELF-DISTILLATION

  • On-policy self-distillation supervises student-generated trajectories with teacher predictions on the same prefixes, using states visited by the current student policy.The teacher is a frozen or EMA copy of the student augmented with privileged information.Source: §3.1 · p. 4

3.2 SPATIALLY GROUNDED PRIVILEGED GUIDANCE

  • The teacher receives textual guidance locating question-relevant visual evidence, while the student must identify that evidence from the unchanged image and question.The hint is absent at inference time.Source: §3.1 · p. 4, §3.2 · p. 5

3.2.1 PROCEDURAL GENERATION OF SPATIAL GUIDANCE

  • Procedurally generated scenes make object identities, attributes, and coordinates known by design, enabling spatial guidance without human annotations or an external grounding model.Counting questions require locating all instances matching a queried category, providing supervision over multiple relevant image locations.Source: §3.2.1 · p. 5
  • The generator selects a present color–shape category, identifies matching object locations, and builds a hint listing those locations and their count.The complete training tuple is generated automatically; the main set has 3,000 image–question pairs from 1024×1024 scenes with 12–40 objects.Source: §3.2.1 · p. 5

3.2.2 SPATIALLY PRIVILEGED ON-POLICY SELF-DISTILLATION

  • Training uses the same pretrained initialization for student and teacher, with the teacher frozen in the main setting and the student–teacher inputs differing only by the hint.The objective minimizes generalized Jensen–Shannon divergence; at inference, only the image and question are provided.Source: §3.2.2 · pp. 5–6

4 EXPERIMENTS

Where-OPD is evaluated on three MLLMs across visual benchmarks, with results showing broad gains and transfer from synthetic counting scenes to real-world tasks. Ablations examine the training signal, spatial hints, scene-generation choices, and teacher updates.

4.1 EXPERIMENTAL SETUP

  • The evaluation covers three base models, uses 3,000 generated counting scenes for one epoch, and measures accuracy across 15 visual benchmarks.Where-OPD results for Qwen3.5-4B and Qwen3.5-9B in Table 1 average three seeded runs; other post-training results use one run per setting.Source: §4.1 · p. 6

4.2 MAIN RESULTS

  • Where-OPD achieves the highest average across all three MLLMs; on Qwen3.5-4B, it improves the 15-benchmark average by 3.28 points, versus 0.45 for Vision-OPD.Improvements span multiple capabilities, while methods using enhanced or localized visual observations show less consistent gains across tasks.Source: Table 1 · p. 7, §4.2 · p. 7, §4.3 · p. 9
  • Despite post-training only on procedurally generated geometric scenes, Where-OPD improves real-world visual benchmarks and tasks beyond counting.MME-RealWorld, HallusionBench, V*, HR-Bench 8K, GQA, and AMBER improve across all three MLLMs.Source: §4.2 · p. 7

4.3 EXPERIMENTAL ANALYSIS

  • With the base model, synthetic scenes, and training steps held fixed, OPSD achieves the highest average among the compared training signals, while SFT and GRPO yield marginal or no gains.The comparison covers SFT, GRPO, and OPSD variants on the same 3,000 synthetic scenes.Source: §4.3 · pp. 7–8, Table 2 · p. 8
  • On Qwen3.5-4B, removing privileged information leaves OPSD at the base-model average, while spatial coordinates alone outperform answer-only guidance and coordinates plus the count perform best.The reported averages are 72.35 for coordinates alone, 71.27 for the answer, and 72.94 for coordinates plus the count.Source: §4.3 · p. 8
  • Among tested data sizes, 3,000 scenes gives the best performance with one epoch; the default 1024 × 1024 resolution and 12–40 objects per scene give the highest average across tested scene settings.CountQA benefits from denser scenes.Source: §4.3 · p. 8
  • Keeping the teacher frozen (η = 0) gives the best average performance, and increasing the EMA update rate generally degrades it.The authors hypothesize that updating on synthetic images may introduce a slight domain shift.Source: §4.3 · p. 8, Figure 3 · p. 8
  • In qualitative examples, Where-OPD attends to marked question-relevant regions on EvoChart and HallusionBench and follows stacked-box boundaries when correctly counting five instances on CountQA.The base model and Vision-OPD miss marked regions in the first examples and fail the CountQA example.Source: Figure 4 · p. 9, §4.3 · p. 9

5 CONCLUSION

Where-OPD uses spatially grounded privileged guidance for on-policy self-distillation and improves across a broader range of tasks than methods using enhanced visual inputs. Its gains transfer to real-world images despite training exclusively on synthetic scenes.

5 CONCLUSION

  • Where-OPD gives the teacher textual hints identifying question-relevant elements and their locations, while the student learns from the image and question alone.The hints are generated from procedurally generated scenes.Source: §5 · p. 9
  • Compared with methods that provide enhanced visual inputs to the teacher, Where-OPD improves across a broader range of evaluated tasks and transfers to real-world images.Post-training uses synthetic scenes exclusively.Source: §5 · p. 9
  • Ablations indicate that spatial grounding matters: removing the hint or providing the answer without locations yields lower performance.The authors conclude that guidance toward relevant visual evidence provides a transferable distillation signal.Source: §5 · p. 9

A IMPLEMENTATION DETAILS

The implementation uses procedurally generated counting scenes whose full state is known before rendering, allowing exact spatial hints to be generated automatically. Training examples and question formats vary by model size.

A.1 SYNTHETIC COUNTING DATA

  • All training sets are generated procedurally, using 1024×1024 scenes with non-overlapping colored geometric objects on solid-color backgrounds.Objects combine one of twelve colors with one of seven shapes.Source: §A.1 · p. 13
  • Scenes contain 12–40 objects, and questions target a present color–shape pair selected with probability proportional to its count.The 3,000-scene set has a median of 26 objects per scene; target counts range from 1 to 14, with a median of 3.Source: §A.1 · p. 13
  • Because the generator knows every object's position, it creates exact hints without manual annotation, narrating a search over target coordinates and ending with the count.The hints provide the target object type, instance locations, and answer.Source: §A.1 · p. 13
  • The 4B models use multiple-choice questions, while the 9B model uses open-ended questions because multiple-choice prompts often elicited direct answers without intermediate reasoning tokens.Source: §A.1 · p. 13

A.2 TRAINING DETAILS

The training setup applies on-policy rollouts to three MLLMs, with students receiving image–question inputs and teachers receiving an appended grounded hint. Evaluation uses a shared scoring pipeline, and the GRPO and supervised fine-tuning baselines use the same 3,000 examples.

A.2 TRAINING DETAILS

  • All models train for one epoch with eight on-policy rollouts per prompt; Qwen3.5-9B uses batch size 48, while the other setup uses 96.The learning rate is 2 × 10−6 with 10 warm-up steps; rollouts use stochastic sampling with thinking disabled.Source: §A.2 · p. 13
  • Students receive the image and question, while teachers receive the same input with a grounded hint appended.The 4B models use multiple-choice questions and Qwen3.5-9B uses open-ended questions.Source: Figure 5 · p. 14

A.3 EVALUATION PROTOCOL

  • Accuracy is the fraction judged correct after rule-based answer checking, option-letter matching, and an LLM judge fallback; failed or unparseable responses count as incorrect.The judge and scoring rules are identical across methods.Source: §A.3 · p. 14

A.4 BASELINES: GRPO AND SUPERVISED FINE-TUNING

  • Both baselines use the same 3,000 examples as Where-OPD and train for one epoch; GRPO uses eight rollouts and a binary answer-match reward.Supervised fine-tuning minimizes cross-entropy on the grounded coordinate hint and the final count.Source: §A.4 · p. 14

B.1 ARE RESULTS DRIVEN BY IN DOMAIN TASKS?

The category-level analysis examines whether gains are mainly driven by counting, the training task, and reports improvements across diverse visual categories. Table 4 organizes Qwen3.5-4B accuracy by category across six benchmarks.

B.1 ARE RESULTS DRIVEN BY IN DOMAIN TASKS?

  • Improvements span diverse visual categories, so benchmark-level gains are not driven mainly by counting questions, the task seen during training.The per-category results are reported for six benchmarks.Source: §B.1 · p. 14
  • The analysis specifically tests whether benchmark-level gains are driven mainly by counting questions.Counting is identified as the task seen during training.Source: §B.1 · p. 14
  • Table 4 compares Qwen3.5-4B and Where-OPD accuracy by category across six benchmarks, with Δ defined as Where-OPD minus base accuracy.The table also reports the number of examples in each category.Source: Table 4 · p. 15

B.2 SCENE-GENERATION AND TRAINING ABLATIONS

Ablations compare scene resolution and object-count ranges, training-set sizes, teacher update rates, and evaluations with thinking enabled. The reported best averages among tested settings favor the default scene setup and 3,000 training scenes.

B.2 SCENE-GENERATION AND TRAINING ABLATIONS

  • The default 1024^2 resolution and 12–40 objects per scene achieve the highest seven-benchmark average among the tested scene-generation settings.The ablations vary image resolution and the number of objects per scene.Source: §B.2 · p. 15
  • 3,000 scenes produce the highest benchmark average among tested training-set sizes for both MLLMs; larger sets improve some individual benchmarks but not the average.The main experiments therefore use 3,000 scenes.Source: §B.2 · pp. 15–16
  • The teacher update-rate study compares per-benchmark accuracy across EMA rates, with η = 0 denoting a frozen teacher.The table reports unweighted averages across seven benchmarks.Source: Table 7 · p. 16, §B.2 · p. 16
  • A separate evaluation reports Qwen3.5-4B accuracy across 15 benchmarks with thinking enabled for both models.The table gives percentage-point changes from the base model evaluated in the same mode.Source: §B.2 · p. 16

B.3 ADDITIONAL QUALITATIVE RESULTS

Qualitative examples contrast Where-OPD’s stated spatial locations and evidence coverage with base-model errors in distance comparisons, object localization, and chart or image reasoning. Additional tables describe evaluations with thinking enabled and attention-layer selection.

B.3 ADDITIONAL QUALITATIVE RESULTS

  • In another distance comparison, the base selects the trailer, while Where-OPD locates the truck adjacent to the bus’s lane and answers truck.The question asks whether the truck or trailer is closer to the bus.Source: §B.3 · p. 17
  • In CVBench distance examples, the base model relies on how colored boxes align, while Where-OPD describes objects’ positions in the room before answering.For the lamp question, the base chooses the door from apparent box alignment, whereas Where-OPD places the table in the foreground and answers table.Source: §B.3 · pp. 16–17, Figure 6 · p. 17
  • Across GQA and V*, Where-OPD’s stated locations align with queried objects, while the base model gives missing or misplaced boxes and identifies incorrect objects.Examples include the base placing a box on the wrong vehicle and none of ten boxes on a candle.Source: Figure 7 · p. 18, §B.3 · p. 19
  • Table 8 reports Qwen3.5-4B accuracy across 15 benchmarks with thinking enabled, including percentage-point changes from the base model in the same mode.All models in that table are evaluated with thinking enabled.Source: Table 8 · p. 19
  • Table 9 selects the attention layer with the highest target-object enrichment, measured on 158 held-out images containing at least three targets.Enrichment of 1 means targets receive attention proportional to their area.Source: Table 9 · p. 19
  • For EvoChart and HallusionBench examples, Where-OPD attends to the chart or image regions needed to answer, while the base model and Vision-OPD omit some relevant regions in the chart examples.The examples require combining chart evidence across regions or comparing a USA label with land-area values.Source: §B.3 · p. 19

B.4 LAYER SELECTION FOR ATTENTION MAP VISUALIZATION

The attention analysis selects visualization layers using target-object attention enrichment, then compares where the models attend on held-out examples. Where-OPD attends to all marked targets in the shown counting examples, unlike the base model and Vision-OPD.

B.4 LAYER SELECTION FOR ATTENTION MAP VISUALIZATION

  • Enrichment compares attention assigned to a token set with its share of visual tokens; 1.0 is area-proportional, while higher values indicate attention concentrated on evidence.Source: §B.4 · p. 19
  • The enrichment measurement uses target objects in 158 held-out synthetic scenes and the full-attention layers of Qwen3.5-4B.Source: §B.4 · p. 19
  • The enlarged EvoChart and HallusionBench visualizations show each input, question, ground-truth answer, and attention maps for Qwen3.5-4B, Vision-OPD, and Where-OPD.Source: Figure 8 · p. 20, Figure 9 · p. 21
  • In two held-out counting examples, Where-OPD attends to all marked targets and counts correctly, while Qwen3.5-4B and Vision-OPD attend to fewer.Source: Figure 10 · p. 21, §B.4 · p. 22
  • Target-object enrichment peaks at L27 for the base model and L15 for Vision-OPD and Where-OPD; Where-OPD has higher enrichment at L15, the layer used for its visualization.Source: §B.4 · p. 22
Loading 2610.02117v1…