PaperlayerView exact revision on arXiv

Source-linked AI summary

Imagine Before You Predict: Interleaved Latent Visual Reasoning for Video Event Prediction

Tianxiang Jiang, Linquan Wu, Sheng Xia, Songze Li, Ziang Yan, Haoyu Yang, Yu Qiao, Yi Wang

arXiv:2606.05769v1cs.CV

TL;DR

Video event prediction remains difficult because text-only intermediate reasoning can lose fine-grained dynamic visual structure. Future-L1 interleaves language with latent visual spans and achieves state-of-the-art results across FutureBench and TwiFF-Bench, including a 24.4-point gain for Qwen3-VL-8B on FutureBench.

  • Problem

    Existing video MLLMs verbalize future reasoning in text, while VEP requires temporally organized latent processes that preserve fine-grained dynamic visual structure.

  • Method

    Future-L1 alternates language tokens with continuous latent visual spans, training them through FUTURE-L1-50K, future-frame alignment, and LA-DAPO trajectory optimization.

  • Results

    85.4 versus 61.0 for Qwen3-VL-8B on FutureBench, while TwiFF-Bench’s average score rises from 2.44 to 3.04.

  • Takeaways & Limitations

    Across FutureBench and TwiFF-Bench, preserving dynamic future visual structure in latent space supports stronger multiple-choice and open-ended future reasoning.

  • Takeaways & Limitations

    The reported evidence is scoped to the FUTURE-L1-50K construction, supervised fine-tuning, and reinforcement-learning training setup.

Abstract

from arXiv · show

Video event prediction (VEP) requires models to infer unobserved future states from partial video evidence. Existing video MLLMs usually verbalize intermediate future reasoning in text space: once visual evidence is verbalized, fine-grained motion, geometry, and interaction cues can be lost, leading to plausible but visually ungrounded hallucinations. We introduce Future-L1, an interleaved latent visual reasoning framework that lets an MLLM alternate between language tokens and continuous latent visual spans during autoregressive decoding. To train this capability, we construct Future-L1-50K by selecting examples where future visual hints help prediction and align latent states to future-frame embeddings, then further optimize sampled latent trajectories with LA-DAPO, a latent-aware RL objective with outcome-contrastive and temporal-diversity rewards. Future-L1 achieves new state-of-the-art results on both benchmarks: on FutureBench, it improves Qwen3-VL-8B from 61.0 to 85.4 and exceeds the previous best Video-CoE by 10.4 points; on TwiFF-Bench, it improves the average score from 2.44 to 3.04. These results suggest that future-oriented video reasoning benefits from preserving intermediate visual semantics in latent space rather than translating every reasoning step into text.

1 Introduction

Video event prediction requires inferring unobserved dynamic visual states, but text-based intermediate reasoning can lose fine-grained visual structure. FUTURE-L1 addresses this with interleaved latent visual spans, curated supervision, and latent-aware reinforcement learning, achieving strong gains on FutureBench and TwiFF-Bench.

  • Video event prediction requires constructing internal hypotheses about unobserved object motion, entity interactions, and scene evolution from partially observed videos.
  • Existing video MLLMs usually verbalize future reasoning in text, while static or one-shot visual latents cannot update imagined dynamic states over multiple reasoning steps.
  • FUTURE-L1 alternates textual tokens with continuous latent visual spans during autoregressive decoding, using language to organize reasoning while preserving intermediate dynamic visual structure.
  • FUTURE-L1-50K uses visual-gain data curation, while LA-DAPO optimizes sampled latent trajectories with outcome-contrastive and temporal-diversity rewards.
  • 85.4 on FutureBench improves Qwen3-VL-8B from 61.0 and exceeds Video-CoE by 10.4 points, while TwiFF-Bench average score rises from 2.44 to 3.04.

2 Related Work

Related work spans mainstream multimodal language models for visual and video understanding, latent-space reasoning with visual supervision, and video event prediction as inference over unobserved futures. These lines of research motivate distinguishing future-oriented prediction from standard visible-content video benchmarks.

  • Multimodal Large Language Models: MLLMs connect visual encoders with large language model backbones and have become a mainstream framework for visual understanding.Video-focused MLLMs add temporal frame sampling, video instruction tuning, longer-context modeling, and large-scale video-text corpora.
  • Reasoning in Latent Space: Latent reasoning replaces discrete textual reasoning tokens with continuous hidden states fed back into the language model.Prior work compresses chain-of-thought into a compact thinking space and distills or supervises latent steps.
  • Reasoning in Latent Space: Visual latent-reasoning methods align latent slots with helper-image embeddings, constrain visual thoughts with teacher-guided attention, or alternate text tokens and continuous visual states.Examples include Mirage, LVR, LaViT, Monet, SkiLa, and SwimBird.
  • Video Event Prediction: Video event prediction differs from standard video understanding by requiring inference of unobserved future events from a video prefix.The setting includes low-level action anticipation, future-frame prediction, and high-level semantic next-event prediction.

3 Method

FUTURE-L1 enables video event prediction through autoregressive reasoning that interleaves text with bounded continuous latent visual spans. It trains this capability with future-frame-aligned supervised traces and latent-aware reinforcement learning that rewards predictive and temporally diverse trajectories.

  • Autoregressive Reasoning with Latent Visual Spans: FUTURE-L1 alternates textual reasoning, bounded latent visual spans, and a final answer, with latent states fed back as subsequent input embeddings.Three special tokens delimit latent spans, whose hidden states form a latent visual reasoning channel.
  • Training Objective: SFT trains interleaved traces by aligning latent states with corresponding future-frame embeddings while preserving standard next-token modeling for textual positions.This provides a grounded cold start before reinforcement learning.
  • LA-DAPO: LA-DAPO optimizes sampled latent trajectories with outcome-contrastive and temporal-diversity rewards, linking trajectories to answer correctness while discouraging repeated visual thoughts across spans.It can optimize latent reasoning during RL without intermediate-frame annotations.
  • Dynamic Latent Budget at Inference: Latent spans end when the model emits <|latent_end|>, are capped by Lmax, and may recur multiple times to allocate computation across reasoning stages.This gives inference a dynamic latent budget rather than a fixed span length.
  • Visual-Gain Data Curation: FUTURE-L1-50K filters TwiFF candidates by visual gain, retaining samples with pt ≤6 and pv −pt ≥2 after eight-rollout evaluations per condition.The hinted condition adds intermediate future frames from the same authentic video, while retained examples require measurable lift over text-only reasoning.

4 Experiments

Experiments on FutureBench and TwiFF-Bench show that FUTURE-L1 substantially improves video event prediction over generic, text-centric, and prior latent-reasoning baselines. Ablations further attribute the gains to visual-gain filtering, latent-aware rewards, adaptive latent usage, and compact inference.

  • FutureBench: 3-Hop and Interp. remain difficult for prior models, with Video-CoE reaching only 71.6 and 71.4, respectively.These splits require longer event-chain extrapolation and reasoning over non-consecutive future states.
  • FutureBench: FUTURE-L1-RL reaches 85.4 on FutureBench, exceeding Qwen3-VL-30B-A3B by 18.5 points and Video-CoE by 10.4 points.FUTURE-L1-SFT reaches 73.2 from the Qwen3-VL backbone’s 61.0, while the text-only SFT control reaches 65.0.
  • TwiFF-Bench: 3.04 average on TwiFF-Bench marks FUTURE-L1-RL’s best result, combining 3.11 CoT and 2.97 Ans and surpassing TwiFF-2.7M’s 2.79.FUTURE-L1-SFT raises the average from 2.44 to 2.52, while LA-DAPO improves both dimensions.
  • Inference efficiency: 85.4 accuracy at 0.91 seconds with 195.3 tokens makes FUTURE-L1-RL more efficient than explicit multi-turn baselines.Video-R1 emits 398.5 tokens at 3.28 seconds, while Video-o3 emits 348.6 tokens at 25.90 seconds.

5 Conclusion · A Baselines

FUTURE-L1 preserves dynamic future visual structure in continuous latent spans and trains them with curated future-frame grounding plus latent-aware reinforcement learning. The baselines span general MLLMs, text-centric video-reasoning models, latent visual-reasoning models, and unified understanding-generation systems.

  • 5 Conclusion: FUTURE-L1 keeps dynamic future visual structure in a continuous latent channel instead of verbalizing every intermediate hypothesis as text.This is the framework’s central idea for video event prediction.
  • 5 Conclusion: FUTURE-L1-50K grounds latent spans with future-frame embeddings selected by visual-gain curation.The curated data makes latent visual grounding practical.
  • 5 Conclusion: LA-DAPO optimizes sampled latent trajectories using outcome-contrastive and temporal-diversity rewards.The passage identifies LA-DAPO as the second component enabling the framework.
  • A Baselines: General MLLM baselines include GLM-4.1V, LLaVA-NeXT-Video, MiMo-VL, InternVL3, Qwen2.5/3-VL, GPT-4o, and GPT-5.These broadly trained open-source and proprietary models test whether generic video-language instruction following suffices for future-event prediction.
  • A Baselines: Video-reasoning baselines include Video-RFT, Video-R1, VideoAuto-R1, Video-o3, NEP, and Video-CoE.Most use SFT, RL, or both to strengthen textual reasoning over video, making them the closest text-centric competitors.
  • A Baselines: Latent visual-reasoning baselines include LVR, Monet, and SwimBird, developed primarily outside dense future-event prediction.Their transfer performance separates general latent reasoning benefits from FUTURE-L1’s data curation and latent-aware RL.
  • A Baselines: For TwiFF-Bench, baselines include Qwen2.5-VL, InternVL3.5, DeepEyes, Janus-Pro, and Bagel.Janus-Pro and Bagel support both visual understanding and generation, making them relevant for future-frame reasoning beyond pure text QA.

B Implementation Details

The implementation summarizes SFT and LA-DAPO training hyperparameters in dedicated tables and uses Easy-R1 for the reinforcement-learning stage.

  • SFT and LA-DAPO training hyperparameters are summarized in Tables 8 and 9, respectively.
  • The reinforcement-learning stage is implemented with the Easy-R1 framework.

C Additional Evaluation Details · C.1 Benchmark Details · C.2 lmms-eval Evaluation Configuration

The evaluation uses FutureBench for multiple-choice future-event prediction and TwiFF-Bench for open-ended future-frame reasoning. Both benchmarks are evaluated with deterministic decoding, with benchmark-specific frame and token limits.

  • C.1 Benchmark Details: FutureBench presents an observed video prefix, a question, four candidate future-event continuations, and one correct option.Its examples are organized by temporal reasoning depth.
  • C.1 Benchmark Details: FutureBench distinguishes 1-Hop, 2-Hop, and 3-Hop reasoning depths, plus Interp. for non-consecutive future events under partial intermediate evidence.The hop settings require progressively longer event chains.
  • C.1 Benchmark Details: TwiFF-Bench provides input frames, a forecasting question, reference future reasoning with intermediate reasoning images, and a ground-truth answer.It covers instructional, predictive, and camera-centric scenarios.
  • C.1 Benchmark Details: Unlike FutureBench, TwiFF-Bench is open-ended rather than multiple-choice and scores both reasoning trajectories and final answers on a 0–5 scale.The benchmark evaluates future-frame reasoning.
  • C.2 lmms-eval Evaluation Configuration: FutureBench evaluation uses up to 32 input frames and permits at most 2,048 new tokens.These limits define its lmms-eval configuration.
  • C.2 lmms-eval Evaluation Configuration: TwiFF-Bench permits at most 4,096 new tokens, while both benchmarks use temperature 0, top-p 1, beam size 1, and disabled sampling.The shared decoding setup is deterministic.

D Details of FUTURE-L1-50K

FUTURE-L1-50K is a 50K subset selected to coldstart latent visual reasoning before LA-DAPO. It retains interleaved trajectories where future visual hints substantially improve prediction reliability, while latent-span usage increases with inference difficulty rather than SFT trace length.

  • Dataset construction: FUTURE-L1-50K is the 50K subset used to coldstart latent visual reasoning before LA-DAPO.It is selected from interleaved trajectories by the visual-gain probe.
  • Dataset construction: Examples are selected when future visual hints substantially improve prediction reliability, targeting cases where visual imagination is empirically useful.Each example contains a video prefix frame, one or more future reasoning frames, and an interleaved textual reasoning trace.
  • Dataset composition: FUTURE-L1-50K covers all three TwiFF task categories and is dominated by high visual-gain samples.Only 4.2% of examples contain three or more future reasoning frames.
  • Inference behavior: Latent usage scales with inference difficulty: FUTURE-L1 allocates three-or-more latent spans increasingly often as FutureBench depth grows, rather than mirroring SFT trace length.This contrasts with the dataset’s low frequency of traces containing three or more future reasoning frames.

E Additional Analyses · F Prompts

Additional analyses show that Future-L1 develops distinct, stage-wise latent visual states and that LA-DAPO produces more stable, better-aligned training rewards than standard DAPO. The paper also documents the prompt templates used for interleaved reasoning and benchmark evaluation.

  • E Additional Analyses: FUTURE-L1-50K is characterized through category, visual-gain, reasoning-frame-count, word-count, and word-frequency distributions.Figures 6 and 7 provide dataset statistics and vocabulary-frequency analyses.
  • E Additional Analyses: Sequential latent spans form distinct, compact clusters separated from text and vision tokens, supporting stage-wise updates rather than repeated visual thoughts.These representations update the model’s internal future hypothesis before final prediction.
  • E Additional Analyses: FUTURE-L1 produces higher and more stable overall, accuracy, format, and contrastive visual reward trajectories than standard DAPO.The advantage extends beyond final-answer optimization to the latent visual reasoning signal.
  • E Additional Analyses: The contrastive visual reward improves under LA-DAPO, indicating alignment between latent visual states and successful prediction trajectories.This provides training-time evidence that latent-aware rewards improve reinforcement learning for future-event reasoning.
  • F Prompts: The system prompt enables interleaved textual and latent visual reasoning during model operation.This prompt is specified in Figure 10.
  • F Prompts: Evaluation prompts define TwiFF-Bench user inputs, judge payloads for reasoning quality and answer accuracy, and binary answer judging for the LA-DAPO accuracy reward.These templates are provided in Figures 11–14.

G Case Study

The case studies show that FUTURE-L1 uses interleaved verbal anchors and latent spans to track visually distinct future transitions while keeping forecasts readable. A failure case shows that latent invocation alone can lose fine-grained event identity, motivating LA-DAPO’s trajectory optimization.

  • Successful cases: FUTURE-L1 alternates short verbal anchors with latent spans at future-state transitions, preserving readable trajectories while marking intermediate visual hypotheses.The transitions include entering rooms, manipulating objects, changing usage contexts, and progressing across action stages.
  • Failure case: FUTURE-L1 recognizes the baseball-dog context but predicts a generic continuation instead of the ground-truth carpet, refrigerator, and dugout events.This failure reflects event-specific detail loss despite correct high-level context recognition.
  • Failure case: The failure suggests that latent spans must preserve fine-grained event identity, motivating LA-DAPO with outcome-contrastive and temporal-diversity rewards.LA-DAPO optimizes the sampled latent trajectories rather than relying on latent invocation alone.
  • Successful cases: Successful examples cover grooming, product demonstration, and staged martial-arts sequences across distinct scenes and action stages.The cases include beard trimming and returning to bed, SHOVEL HELPER use from setup to outdoor application, and a montage ending in meditation.
Loading 2606.05769v1…