PaperlayerView exact revision on arXiv

Source-linked AI summary

Beyond Visual CoT: Internalized Visual Thinking for Proactive Video Reasoning

Xiaoyu Zhu, Xinke Deng, Suresh Taddewadikar, Arnab Kumar Mondal, Zhongyu Jiang, Ian Fasel, Joerg Liebelt

arXiv:2608.15869v1cs.CVcs.AIcs.CLcs.LGcs.MM

TL;DR

Proactive video reasoning needs visual foresight, but explicit Visual CoT adds inference overhead. IVT internalizes predictive visual thinking during training and, under matched settings, outperforms Visual CoT on four of six splits while reducing latency by more than 5×.

  • Problem

    Can multimodal language models learn visual thinking during training while reasoning efficiently at inference without generating intermediate images?

  • Method

    IVT jointly trains textual prediction and latent future-frame representation prediction over unlabeled videos, enabling direct answer generation at inference.

  • Results

    Under matched settings, IVT outperforms Visual CoT on four of six splits while reducing average end-to-end inference latency by more than 5×.

  • Takeaways & Limitations

    Effective visual reasoning need not require explicitly rendering visual thoughts; predictive world modeling can be internalized during training for efficient multimodal reasoning.

Abstract

from arXiv · show

Multimodal large language models increasingly use visual chain-of-thought (Visual CoT) to reason about spatial, temporal, and embodied environments. By generating intermediate reasoning images, Visual CoT provides an intuitive mechanism for visual foresight but introduces substantial inference overhead, which is particularly problematic for proactive video reasoning. We ask whether models can learn to think visually during training while reasoning directly at inference. We introduce Internalized Visual Thinking (IVT), a post-training framework that jointly optimizes textual prediction and next-embedding prediction over unlabeled videos. Given a partially observed video, IVT predicts latent representations of future frames together with the target textual answer, encouraging the model to capture motion, object transitions, interactions, and latent intent. At inference, IVT generates the answer directly without synthesizing or re-encoding future frames. We conduct controlled studies across target representations, decoder designs, prediction horizons, data mixtures, training curricula, and predictive objectives. IVT improves over direct-answer fine-tuning on all six evaluation settings while retaining the same inference pathway. Compared with explicit Visual CoT, IVT achieves comparable or better performance and reduces average end-to-end latency by more than 5x. Together, our findings suggest that explicit pixel-space generation at inference time, as used in visual chain-of-thought, may not be necessary for effective proactive video reasoning. Predictive world modeling can be internalized during training to produce multimodal reasoners that are both more accurate and substantially more efficient.

1 Introduction

Proactive video reasoning requires anticipating future events from incomplete observations, but explicit Visual CoT imposes costly image generation and re-encoding at inference. IVT internalizes predictive visual thinking during training, enabling direct answer generation while improving reasoning efficiency and performance.

  • Motivation: Explicit Visual CoT preserves visual dynamics but substantially increases inference cost by generating an intermediate image and re-encoding it into visual tokens.This burden is especially problematic for latency-critical proactive reasoning applications.
  • Motivation: Proactive video reasoning requires anticipating incomplete or not-yet-started actions by inferring latent intent, causal dynamics, and plausible future states.The paper distinguishes early-event prediction from next-event prediction, including actions that may occur several seconds ahead.
  • Method: IVT jointly predicts future-frame latent embeddings and textual answers during post-training on unlabeled videos, then generates answers directly at inference.Future-embedding supervision targets object transitions, motion, interactions, and scene dynamics without synthesizing future images at inference.
  • Findings: IVT surpasses strong Visual CoT counterparts on 4 of 6 benchmarks under matched base models, training data, and evaluation protocols.The result suggests explicit Visual CoT may not be necessary for proactive video reasoning.
  • Findings: IVT’s gains depend on sharing the language-model decoder, balancing visual-prediction and text-generation batches, choosing the predictive horizon, and using a suitable curriculum.The paper reports that auxiliary prediction loss alone does not automatically produce improvements.
  • Conclusion: Training-time predictive world modeling preserves visual foresight benefits while retaining the inference efficiency required by proactive reasoning tasks.The approach encodes predictive visual knowledge in the representations used to reason rather than rendering imagined visual states.

2 Internalized Visual Thinking

Internalized Visual Thinking augments textual supervision with prediction of latent future-frame representations from observed video context. It trains predictive world modeling while preserving the Answer-Only SFT inference pathway, without additional image generation.

  • Internalized Visual Thinking: IVT predicts latent representations of future frames from the observed video prefix and textual prompt, supplementing standard language supervision that lacks explicit future-state constraints.A target encoder produces future-frame latent tokens, while projected predictive hidden states are optimized with an auxiliary predictive objective.
  • Target representations: The study compares Flux-VAE, DINOv2, and SigLIP2 target spaces, including adaptive jointly optimized and frozen target encoders.The target spaces differ in their emphasis on reconstruction-oriented appearance and semantic features.
  • Controlled study dimensions: The study varies decoder coupling, future prediction horizons, supervision mixtures, training curricula, and predictive objectives.Architectures include shared Dense decoding and separate-expert MoE decoding; mixtures are 1:1, 3:1, and 5:1 understanding-to-prediction examples; curricula compare joint and two-stage training; objectives compare direct feature regression with rectified-flow matching.
  • Inference: At inference, IVT receives only the observed video prefix and prompt, then autoregressively generates the textual response through the same pathway as Answer-Only SFT.IVT introduces no additional image generation at inference.

3 Experiments … 3.5 How Should Future Representations Be Predicted?

The experiments evaluate proactive video prediction across three egocentric-video benchmarks and show that internalized future-state prediction can improve reasoning without explicit visual generation. Results identify effective target representations, balanced predictive supervision, and representation-dependent predictive objectives, while highlighting the latency cost and limited consistency of explicit Visual CoT.

  • 3.1 Experimental Protocol: The evaluation covers early-event and next-event prediction on Ego-Exo4D, Ego4D, and EPIC-KITCHENS-100 using open-ended language metrics and Top-5 recall for next-event prediction.EEP uses observation ratios of 0.1, 0.3, 0.5, and 0.7; metrics include ROUGE-L, METEOR, CIDEr, and BERTScore.
  • 3.2 Motivation: When Does Explicit Visual CoT Help, and at What Cost?: Accurate future frames substantially improve both tasks, whereas self-generated frames recover only part of the early-event benefit and provide no consistent next-event gain.This separates the value of future visual context from the cost and fidelity of externalizing it as pixels.
  • 3.2 Motivation: When Does Explicit Visual CoT Help, and at What Cost?: Visual CoT improves early-event prediction but yields negligible or negative effects on most next-event metrics while increasing mean latency by 6.2× and 5.0×, respectively.Early-event gains include 5.9% in ROUGE-L and 17.9% in CIDEr; next-event METEOR improves by 4.3%.
  • 3.3 Which Future Representation Should the Model Predict?: Flux-VAE is the strongest and most consistent future-state target, outperforming semantic alternatives across tasks and metrics, while adaptive SigLIP2 generally outperforms frozen SigLIP2.DINOv2 improves three of four early-event metrics and all four next-event metrics but remains below Flux-VAE.
  • 3.4 How Much Predictive Supervision Is Needed?: The balanced 1:1 understanding-to-prediction mixture achieves the strongest result on every metric at 20k steps, despite understanding-heavy mixtures adapting faster initially.At 20k steps, the balanced mixture reaches ROUGE-L 34.5, METEOR 27.4, CIDEr 2.02, and BERTScore 36.5.
  • 3.5 How Should Future Representations Be Predicted?: Direct regression substantially outperforms flow matching for DINOv2, whereas the objectives are closer for VAE targets and flow matching slightly surpasses regression on final-checkpoint BERTScore.For VAE, the final BERTScore values are 36.5 for flow matching and 34.7 for regression.
  • 3.5 How Should Future Representations Be Predicted?: The target representation and predictive objective should be designed jointly because direct regression is strong for DINOv2 and comparable to flow matching for VAE.This conclusion also holds despite BAGEL performing visual generation in VAE latent space with rectified flow.

3.6 Which Training Curriculum Best Integrates Predictive Supervision? · 3.7 How Should Future Prediction Be Structured? · 3.8 Main Results: Quality and Efficiency

Joint predictive-and-text training is the strongest curriculum, while decoder sharing works best for short horizons. Across six settings, IVT consistently improves answer-only fine-tuning and usually surpasses Visual CoT while preserving the same inference pathway.

  • 3.6 Which Training Curriculum Best Integrates Predictive Supervision?: Joint Training improves over Answer-Only SFT by 12.6% in ROUGE-L and 26.3% in CIDEr, whereas Two-Stage Training underperforms it on every metric.Two-Stage Training decreases ROUGE-L by 6.5%, BERTScore by 5.6%, METEOR by 5.5%, and CIDEr by 3.5%.
  • 3.6 Which Training Curriculum Best Integrates Predictive Supervision?: Next visual state prediction is most effective when jointly optimized with text generation throughout post-training, keeping predictive representations aligned with the downstream language objective.Sequential predictive training followed by task-only fine-tuning substantially weakens transfer compared with joint optimization.
  • 3.7 How Should Future Prediction Be Structured?: Both Dense and Mixture-of-Experts architectures outperform Answer-Only SFT across all four metrics and every evaluated prediction horizon.The study assigns both objectives to one shared decoder in Dense and to separate experts in MoE.
  • 3.7 How Should Future Prediction Be Structured?: At H = 2, Dense achieves 37.4 ROUGE-L, compared with 33.8 for MoE; Dense is strongest at H = 1–2, while MoE peaks around H = 3.Dense declines as the horizon increases, whereas MoE changes only modestly between H = 3 and H = 4.
  • 3.7 How Should Future Prediction Be Structured?: The dense architecture transfers predictive supervision most effectively when the future target is short and well constrained, while longer horizons make tight coupling more ambiguous.Shared decoder gradients directly shape representations used for answer generation, explaining Dense’s advantage at H = 1 and H = 2.
  • 3.8 Main Results: Quality and Efficiency: IVT improves all four metrics across all six dataset–task settings, whereas Visual CoT improves four settings and Text CoT generally degrades performance relative to Answer-Only SFT.The controlled comparison uses matched BAGEL-7B-based understanding-branch initialization for Text CoT, Visual CoT, and IVT.
  • 3.8 Main Results: Quality and Efficiency: Compared with Visual CoT, IVT performs better in four of six settings, wins all three next-event benchmarks, and improves average ROUGE-L from 45.9 to 49.7.Visual CoT remains stronger on two early-event benchmarks, exceeding IVT by 1.3 ROUGE-L points on Ego4D and 1.2 points on EPIC-KITCHENS-100.
  • 3.8 Main Results: Quality and Efficiency: IVT retains the Answer-Only SFT inference pathway while moving future-state prediction from inference-time computation to training-time supervision.Across Tables 3–5, IVT is competitive with state-of-the-art 7B-scale models but trails the larger Qwen3-VL-8B model.

4 Related Work

Prior work frames proactive video reasoning around early- and next-event prediction, while related approaches explore explicit visual intermediates, latent representation prediction, and unified multimodal understanding and generation. IVT differs by jointly predicting future latent visual representations and future textual states internally.

  • Proactive Video Reasoning: Proactive video reasoning covers early-event prediction before an ongoing action ends and next-event prediction for actions that have not yet begun.The latter may involve anticipating actions several seconds into the future.
  • Proactive Video Reasoning: IVT learns future dynamics internally by jointly predicting the next latent visual representation and the future textual state.
  • Visual Chain-of-Thought: Visual reasoning methods externalize intermediate visual states through question-relevant bounding boxes, auxiliary visual marks, specialist vision tools, or textual spatial states.
  • Latent Visual Reasoning and Next-Embedding Prediction: Joint-embedding predictive architectures use latent semantic representation prediction as a more abstract learning signal than raw-pixel reconstruction.I-JEPA is described as predicting masked image representations from visible content.
  • Unified Understanding and Generation Models: Unified multimodal models seek shared architectures for visual understanding and generation, including autoregressive modeling of interleaved image and text sequences.Emu and Chameleon are examples of interleaved image-text autoregressive models.

5 Conclusion

The conclusion presents Internalized Visual Thinking (IVT) as a post-training framework that learns future visual states during training while answering directly at inference. Across proactive video benchmarks, IVT improves over text-only post-training and offers strong efficiency and performance relative to Visual CoT.

  • IVT is introduced as a post-training framework for proactive video reasoning.
  • IVT jointly supervises future latent representations and textual predictions, transferring visual foresight into internal representations without explicit inference-time generation.
  • More than 5× lower average end-to-end inference latency accompanies IVT’s improvement over text-only post-training and its stronger performance than Visual CoT on four of six splits.

A Training Algorithms

IVT jointly computes textual and predictive objectives in one forward pass, using disjoint supervision masks and detached predictive targets. The two-stage baseline instead separates predictive training from task fine-tuning with independent optimizer states.

  • Joint IVT training: IVT computes textual and predictive objectives within the same forward pass using disjoint supervision masks for answer and future-prediction positions.The data-mixture ratio controls the relative frequency of understanding and prediction examples in packed training data.
  • Joint IVT training: Predictive targets are detached through sg(·), while the total loss combines textual and predictive losses as L = Ltext + λpredLpred.The predictive loss uses feature regression or rectified-flow matching when future-prediction positions are present.
  • Joint IVT training: λpred = 1 and the understanding-to-prediction mixture is 1:1 in the specified joint-training procedure.The procedure packs batches, encodes observed frames and prompts into an interleaved multimodal sequence, and updates parameters with AdamW and EMA weights.
  • Two-stage baseline: The two-stage baseline optimizes Lpred for Spred = 10k steps, then Ltext for Stext = 20k steps after initializing from Stage-1 EMA weights.Stage 2 uses a fresh AdamW optimizer and a newly initialized learning-rate schedule.

B Dataset Construction and Preprocessing

The dataset construction jointly uses disjoint native training splits from Ego-Exo4D, Ego4D, and EPIC-KITCHENS-100. Examples pair sampled observed-video prefixes with native textual targets and future-frame targets for EarlyEvent and NextEvent reasoning.

  • Dataset splits: Training jointly uses the native training splits of Ego-Exo4D, Ego4D, and EPIC-KITCHENS-100, with official train/validation partitions and mutually disjoint datasets.Algorithm 3 defines the construction of proactive video reasoning examples.
  • Sampling setup: Observation ratios R = {0.1, 0.3, 0.5, 0.7} and anticipation gap τ_a = 1 parameterize example construction.Observed frames are sampled at 1 fps, with at most 16 frames in the EarlyEvent procedure.
  • Example structure: Examples contain an observed video prefix X≤t, a textual target y, and future-frame targets {I_t+h}_h∈H.The construction samples observations and extracts future-frame targets for predictive training.
  • EarlyEvent examples: For EarlyEvent, the observation boundary is set within a selected action segment, while y is its native action description.The boundary follows t ← t_s + ρ(t_e − t_s), and future-frame targets are extracted for each observation ratio.
  • NextEvent examples: For NextEvent, the context ends before the next action onset by τ_a, and y is the native description of that next action.The procedure identifies the next action a_next with onset t_next and enforces t ≤ t_next − τ_a.

C Implementation Details

Table 6 defines the default training configuration for the target-representation ablations and main IVT experiments, while architecture and curriculum variants change only their specified design choices.

  • Training configuration: Table 6 summarizes the training configuration used in the target-representation ablations and main IVT experiments.
  • Training configuration: The table presents a default configuration shared by the target-representation ablations and main IVT experiments.
  • Training configuration: Architecture and curriculum variants modify only the corresponding design choices described in the main paper.

D Additional Experimental Results · D.1 Out-of-Domain Transfer to Charades

The Charades study tests whether future-state representations transfer from Ego-Exo4D to an unseen video distribution and output format. IVT shows the strongest transfer among evaluated BAGEL-7B methods, with Dense IVT retaining its advantage across prediction horizons.

  • D.1 Out-of-Domain Transfer to Charades: Charades is held out entirely from training, while models post-trained on Ego-Exo4D early-event data are evaluated directly under distribution shift.The evaluation probes transfer beyond the in-domain setting used in the main experiments.
  • D.1 Out-of-Domain Transfer to Charades: For the reported IVT configurations, Dense uses H=1 and MoE uses H=3, matching the settings used in the main results.These configurations correspond to Tables 3–5.
  • D.1 Out-of-Domain Transfer to Charades: Dense IVT exceeds Answer-Only SFT across all evaluated prediction horizons and consistently outperforms the corresponding MoE variants.Figure 6 shows that this transfer advantage persists as the prediction horizon H changes.
  • D.1 Out-of-Domain Transfer to Charades: Latency is measured end to end on 1×B200 with bf16 precision, batch size 1, fixed decoding, one warm-up example, and maximum output length.Frame loading is excluded from timing; CUDA synchronization brackets the complete method-specific inference pipeline, and mean and P95 elapsed times are returned.
  • D.1 Out-of-Domain Transfer to Charades: The transfer task differs in output format: source experiments use partially observed actions, whereas Charades requires selecting an answer option.Performance therefore reflects both video-distribution transfer and preservation of instruction-following and output-format capabilities.
  • D.1 Out-of-Domain Transfer to Charades: IVT achieves the strongest out-of-domain performance among the evaluated BAGEL-7B-based methods on Charades.Table 7 reports transfer from Ego-Exo4D to Charades using the main-result IVT configurations.

D.2 Additional Inference-Efficiency Analysis

The analysis measures complete end-to-end inference latency under matched hardware, precision, input sampling, and decoding conditions. Across six dataset–task settings, Visual CoT is substantially slower than Answer-Only SFT, while IVT retains a comparable inference profile to Answer-Only SFT.

  • Latency protocol and aggregate results: 6.56 s per sample is the average latency for Visual CoT across six dataset–task settings, versus 1.20 s for Answer-Only SFT.Measurements use batch size 1 under identical hardware and precision, with matched input-frame sampling and decoding limits.
  • Latency protocol and aggregate results: The end-to-end timing includes future-frame generation, VAE decoding, visual re-encoding, and final answer generation for Visual CoT.For Answer-Only SFT and IVT, timing includes input-frame encoding and answer generation, capturing each method’s complete inference pipeline.
  • Per-benchmark latency: IVT retains an inference profile comparable to Answer-Only SFT, whereas Visual CoT incurs substantially higher latency from intermediate future-frame generation and visual re-encoding.Table 8 reports mean and P95 per-sample latency over 200 validation examples per benchmark on a single B200 GPU using batch size 1 and bf16 precision.
Loading 2608.15869v1…