VISTA: Internalizing Collective Visual Experience via On-Policy Distillation for Active Multimodal Agents

• CVED organizes the rollout group’s visual artifacts and interaction context as training-only privilege, aligning that context with each student’s original history.Source: §1 · p. 2
• HAPI gives successful trajectories reward-aligned RLVR updates and unsuccessful trajectories experience-conditioned OPSD guidance, without replacing the student’s sampled response or history.The teacher scores the exact response prefix sampled by the student; the rollout group and collective context are removed at deployment.Source: §1 · p. 2
• Outcome-based supervision underuses complementary visual discoveries, and final success does not fully capture the informational value of a trajectory’s experience.A successful trajectory may benefit from reinforcement, while an unsuccessful one may still contain useful observations.Source: §1 · p. 2
• VISTA leads evaluated active multimodal agents of comparable size in average visual-reasoning accuracy and improves over same-backbone baselines across fine-grained perception and general reasoning tasks.Source: §1 · p. 2

2 Related Work

• Reinforcement-learning agents use visual tools and task feedback to explore, improve reasoning–action alignment, and scale interaction depth.Source: RLVR for multimodal agents · p. 2

RLVR for multimodal agents.

• Prior multimodal RLVR work primarily improves exploration and learning from task feedback, while VISTA reuses observations from same-input rollouts as shared supervision.Source: RLVR for multimodal agents · p. 2

OPD for multimodal agents.

• Visual OPD methods use regional evidence, generated visual thoughts, or visual dependence and contrast; VISTA instead combines decision-aligned collective experience with outcome-dependent reinforcement and distillation.Figure 1 depicts the complete rollout group supplying CVED experience while HAPI allocates reinforcement and distillation by trajectory outcomes.Source: OPD for multimodal agents · p. 2, Figure 1 · p. 3

3 VISTA

• CVED builds decision-aligned guidance from the rollout group’s visual experience, while HAPI reinforces successful trajectories and distills collective experience into unsuccessful ones.The teacher evaluates student-sampled response prefixes rather than generating replacement trajectories.Source: §3 · p. 3, §3.2 · p. 4

3.1 Problem Formulation

• For each input, the active policy generates responses and observes tool results; trajectory rewards and terminal outcomes guide learning while the group’s observations remain potential experience.The histories used to sample responses remain unchanged during policy improvement.Source: §3.1 · p. 3

3.2 Collective Visual Experience Distillation

• CVED organizes visual artifacts by their producing decisions and originating interactions, preserving crop relationships, region coverage, and separate occurrences of repeated views.Both successful and unsuccessful attempts can contribute evidence.Source: §3.2 · pp. 3–4
• The teacher augments a student’s history with budgeted collective experience, distinguishing evidence already seen from peer or later observations while evaluating the student’s sampled prefixes.Its distribution is held fixed during optimization, and the student is trained under its ordinary local history.Source: §3.2 · p. 4

3.3 Heterogeneity-Aware Policy Improvement

• HAPI applies outcome-based optimization to successful trajectories and experience-guided distillation to unsuccessful ones, using collective experience from both groups.Unsuccessful attempts include invalid and incomplete trajectories; distillation can guide intermediate responses as well as final answers.Source: §3.3 · p. 4

4.1 Experimental Setup

• Training comparisons use matched backbones, data, optimizers, schedules, tools, and interaction limits; evaluation results are averaged over three seeds.The evaluation covers visual benchmarks and, for training-method comparisons, MMStar, MMBench, and MathVista.Source: §4.1 · p. 5

4.2 Main Results

• VISTA improves over its base checkpoints across all six benchmarks at both scales and outperforms compared thinking-with-images baselines throughout the suite.The trained 4B model also exceeds the base 8B checkpoint across this comparison.Source: §4.2 · p. 5, Table 1 · p. 6

4.3 Cross-Task Generalization

• VISTA leads both fine-grained visual and general-reasoning task groups at 4B and 8B, with the pattern extending to visual mathematical reasoning.Across methods and scales, its advantage holds on every evaluated benchmark rather than arising from one task family.Source: §4.3 · p. 6, Table 2 · p. 7

4.4 Ablation Study

• With HAPI fixed, CVED performs best across three tasks, ahead of Oracle ROI, Unstructured, and Answer-only supervision.Its advantage over Unstructured holds with the same selected observations and context budget, supporting the value of organizing evidence and aligning it with the student's decision.Source: §4.4 · pp. 6–7, Table 3 · p. 7
• With experience construction fixed, HAPI outperforms Uniform hybrid and either learning objective alone across all three tasks.Unsuccessful attempts still contribute observations to CVED, although HAPI routes their behavioral learning differently from successful trajectories.Source: §4.4 · p. 7, Table 4 · p. 7

4.5 Training Design Analysis

• EMA gives the stronger average, while a frozen teacher does better on HR-Bench-8K and MME-RW-CN; JSD has the best divergence average but is not best on every benchmark.JSD and reverse KL both outperform forward KL across the suite, and all three improve over the base model.Source: §4.5 · p. 8, Table 5 · p. 8, Table 6 · p. 8

4.6 Backbone Generalization

• On Qwen3.5-4B and Qwen3.5-9B, VISTA leads on all three tasks for both backbones, though its advantage varies by model and task.On Qwen3.5-9B, VISTA improves over both OPSD and GRPO throughout the suite.Source: §4.6 · p. 8, Table 7 · p. 8

4.7 Scalability

• Larger rollout groups improve or preserve accuracy on all three benchmarks, with diminishing gains toward the largest tested groups.Because the artifact budget stays fixed, larger groups expand the candidate experience pool without requiring the teacher to process every new observation.Source: §4.7 · p. 9, Figure 2 · p. 9

5 Conclusion

• VISTA uses CVED and HAPI to combine successful-behavior reinforcement with experience-guided distillation on unsuccessful trajectories.The conclusion frames grouped active-agent training as experience internalization rather than only scalar-advantage estimation.Source: §5 · p. 10

AI use statement

• The authors used language models only for language polishing, reviewed and revised the edits, and take responsibility for the manuscript’s technical claims.They state that experimental results and implementation details were checked and confirmed by the authors.Source: AI use statement · p. 10

Reproducibility statement

• The paper identifies its sections and appendices for learning, experience construction, teacher supervision, and evaluation, and provides a code repository.Appendix B.4 contains the prompt templates; Appendix B.2 specifies teacher information.Source: Reproducibility statement · p. 10

A VISTA Training Algorithm

• Each input yields K interactive trajectories; the full group supplies rewards for relative advantages and experience for CVED.The algorithm retains original histories, response tokens, tool observations, and rollout log probabilities before verifying outcomes.Source: A VISTA Training Algorithm · p. 13
• HAPI applies the GRPO policy loss to successful trajectories and CVED distillation to unsuccessful trajectories.The algorithm combines the losses with weight λ and normalizes distillation over eligible unsuccessful response tokens.Source: A VISTA Training Algorithm · p. 13
• For unsuccessful responses, the teacher evaluates the student’s sampled token prefixes with selected experience appended only to the teacher’s context.The student’s original history and response remain unchanged; the teacher does not generate replacement targets.Source: A VISTA Training Algorithm · p. 13
• After the student update, the teacher parameters are updated by exponential moving average.The teacher is held fixed during the student optimization step.Source: A VISTA Training Algorithm · p. 13

B.1 Experience Representation and Projection

• The experience compiler retains artifact provenance and relations, merges duplicates, and maps nested crops to coordinates in the original image.Projection creates a separate view for each decision, prioritizing student-visible and peer evidence while accounting for artifact budgets and future-only evidence.Source: §B.1 · p. 14

B.2 Teacher Information and Supervision

• The teacher receives selected visual observations, interaction context, and categorical outcome feedback, but no reference answers or generated final answers.Unsuccessful trajectories include incorrect, protocol-invalid, and incomplete attempts; all their response turns receive distillation supervision without replacing sampled responses.Source: §B.2 · p. 14

B.3 Distribution Matching and Teacher Updates

• CVED matches student and teacher token distributions on unsuccessful responses, while the GRPO term updates successful trajectories using full-group relative advantages.The teacher is initialized from the student and updated by EMA after student optimization; the default divergence is equal-mixture JSD.Source: §B.3 · p. 15

B.4 Core Prompt Templates

• The teacher prompt and wrapper serialize visual experience, interaction context, and outcome feedback into the teacher input while leaving the student history unchanged.Final answers are not included in the main teacher context, and the augmentation does not create an inference-time memory.Source: §B.4 · p. 16, Figure 4 · p. 17, Figure 5 · p. 17

C Additional Experimental Details

• Training uses 6,241 Vision-OPD-6K examples with full images and no target-box hints; default runs use K = 8, λ = 0.1, and B = 4.Rewards depend on answer correctness, tool use is optional, and the shared training schedule is 65 optimizer steps.Source: C Additional Experimental Details · p. 18
• Evaluation pairs V*Bench and HR-Bench tasks with MMStar, MMBench, and MathVista to assess fine-grained perception and general reasoning.Accuracy is reported as a percentage, with table averages defined as unweighted means of the listed benchmarks.Source: C Additional Experimental Details · p. 18
• The controlled comparison matches starting checkpoints, data, student inputs, optimization, tools, and interaction limits within each backbone.Supervision differs by method; schedule matching does not imply equal computational cost because privileged-context construction and teacher evaluation add overhead.Source: C Additional Experimental Details · p. 18
• The privileged-context ablations vary teacher-side information or organization while keeping the student’s image-question task unchanged.Answer-only supplies reference answers to the teacher, Oracle ROI supplies annotated crops, and Unstructured presents the same selected observations in trajectory order rather than with CVED relations and alignment.Source: C Additional Experimental Details · pp. 18–19
• Parameter analyses vary rollout count, artifact budget, distillation weight, and teacher updates; the λ sweep keeps HAPI routing fixed.Reference comparisons include open-source, proprietary, and thinking-with-images models, with model sizes and baselines specified in the experimental protocol.Source: C Additional Experimental Details · pp. 18–19

D Effect of Cross-Trajectory Visual Experience

• The comparison tests peer observations and interaction records against hindsight restricted to the target trajectory, including later-acquired observations.Both settings use Qwen3-VL-4B with K = 8, λ = 0.1, and B = 4.Source: D Effect of Cross-Trajectory Visual… · p. 20
• 83.83% average accuracy versus 81.70% for self-only hindsight, a gain of 2.13 percentage points.The collective-experience comparison uses the same outcome-feedback records, HAPI routing, teacher updates, and context-serialization format in both settings.Source: D Effect of Cross-Trajectory Visual… · p. 20, Table 10 · p. 20
• Collective visual experience gains 3.14 points on V*Bench, 1.25 on HR-Bench-4K, and 2.00 on HR-Bench-8K.Source: D Effect of Cross-Trajectory Visual… · p. 20

E Disentangling HAPI Routing

• 83.83% average accuracy with HAPI, 0.96 percentage points above the uniform hybrid, and highest on every benchmark.Neither routing mask alone improves average accuracy over the uniform hybrid; their benefits depend on one another under the stated normalization conventions.Source: E Disentangling HAPI Routing · pp. 20–21, Table 11 · p. 20

F.1 RLVR for Multimodal Agents

• Unlike approaches focused on exploration or reward-based credit assignment, VISTA transfers observations from same-input rollouts into decision-aligned supervision.Source: §F.1 · p. 21

F.2 OPD for Multimodal Agents

• VISTA's CVED aligns collective interaction experience with student decisions, while HAPI routes successful behavior to reinforcement and failed attempts to experience-guided distillation.Source: §F.2 · p. 21

G Limitations and Future Work

• Collective experience depends on observation diversity and relevance, motivating adaptive exploration and evidence selection when rollouts are redundant or miss useful regions.Source: G Limitations and Future Work · p. 22
• HAPI routes supervision using terminal outcomes, motivating finer-grained feedback that distinguishes productive intermediate decisions from unsuccessful ones.Source: G Limitations and Future Work · p. 22
• Experience construction and teacher inference add training overhead; the evaluation is limited to image-question answering with crop-and-zoom interaction and Qwen-family backbones.The paper proposes compact experience representations, selective teacher queries, matched-compute comparisons, and evaluation on other model families, video, and broader tool-use environments.Source: G Limitations and Future Work · p. 22