Source-linked AI summary
DF$^3$: World Modeling via Decoder-Free Feature Forecasting in Autonomous Navigation
Jiaming Chen, Guoan Xu, Aoshen Huang, Haozhuo Zhang, Yang Li, Wei Pan
TL;DR
Pixel-space forecasting is computationally burdensome, and latent approaches still rely on heavy decoders for task outputs. DF3 forecasts features and derives task predictions within a frozen vision encoder using queries and motion-aware fusion. It reports competitive forecasting accuracy with reduced inference latency and peak GPU memory.
Problem
Pixel-space forecasting is computationally burdensome and requires downstream perception, while latent methods still rely on heavy decoders to model context and map features to task outputs.
Source: §I · p. 1Method
DF3 injects prediction and task queries into a frozen ViT and uses motion-aware context fusion to forecast future features and derive task outputs without heavy decoders.
Source: §I · p. 1, C. Motion-Aware Context Fusion Mechanism · p. 4, B. Structured Query Attention · p. 4Results
DF3 maintains competitive forecasting accuracy against state-of-the-art latent models while reducing inference latency by 70% and peak GPU memory by 67%.
Source: §I · p. 2, B. Comparison Results and Efficiency · p. 5Takeaways & Limitations
DF3 is demonstrated on Cityscapes and in zero-shot autonomous-navigation simulation, with reported efficiency gains supporting real-time robotic applications.
Source: §I · p. 2Forecasts can over-smooth spatial details and lose sharpness in noise and thin structures because prediction occurs in the frozen ViT's downsampled feature space.
Source: D. Qualitative Results · p. 7
Abstract
from arXiv · showhide
Forecasting future states from video sequences is a critical challenge for autonomous robotic systems and a fundamental objective of world modeling. Prior generative methods operating at the pixel level inevitably overemphasize task-irrelevant details, leading to prohibitive computational overhead. While latent-based approaches attempt to mitigate this by predicting features directly, the persistent reliance on heavy decoders for state-to-task mapping remains a computational bottleneck. In this work, we propose Decoder-Free Feature Forecasting (DF$^3$), a novel framework that models world evolution entirely within the latent space and directly derives task outputs, completely eliminating the need for a decoder. Specifically, DF$^3$ injects learnable spatial queries into the terminal blocks of a frozen vision foundation model to extract future state representations directly. By employing a lightweight, unified Motion-Aware Context Fusion (MACF) mechanism that seamlessly integrates coarse flow warping with fine-grained latent cross-correlation, these queries interact with historical token representations to explicitly align and forecast the feature of the next frame. Subsequently, a specialized set of task queries probes these forecasted features for the downstream task. Extensive experiments on public benchmarks and zero-shot deployment in a robotic simulator demonstrate that DF$^3$ achieves performance comparable to state-of-the-art methods while offering superior efficiency and flexibility for integrated perception and control.
I. INTRODUCTION
DF3 forecasts future features and derives task outputs within a frozen vision encoder, addressing the computational costs of pixel-space generation and decoder-heavy latent methods. It uses learnable queries and motion-aware context fusion, and is reported to achieve competitive forecasting accuracy with improved efficiency.
I. INTRODUCTION
- Pixel-space forecasting is computationally burdensome and requires a further perception stage to translate generated video into task outputs.For navigation, semantic scene dynamics matter for planning and control rather than photorealistic future frames.Source: §I · p. 1
- Prior latent methods predict semantic features but often require heavy contextual and task-specific decoders, adding computational overhead.Their high-dimensional feature representations are harder to supervise, motivating heavy contextual modeling.Source: §I · p. 1
- DF3 forecasts future states and derives task outputs within a frozen vision encoder, eliminating heavy feature and task decoders.Learnable prediction queries produce future features, while task queries extract downstream representations.Source: §I · pp. 1–2
- MACF combines coarse flow warping and fine-grained latent cross-correlation to align temporal context without fine-tuning encoder weights.The aligned context lets injected queries forecast future token representations.Source: §I · pp. 1–2
- DF3 achieves competitive forecasting accuracy while reducing inference latency and peak GPU memory, and is demonstrated in zero-shot autonomous-navigation simulation.The framework is validated on Cityscapes and deployed in simulation.Source: §I · p. 2
II. RELATED WORK
This section reviews vision foundation models, forecasting approaches, and query-based architectures as context for DF3. It contrasts pixel-space generation with latent forecasting and describes the encoder-only query design that informs DF3.
A. Vision Foundation Models
- DINOv2 and DINOv3 illustrate the scale and performance of self-supervised vision foundation models, which this work explores for temporal forecasting.The passage reports DINOv3 training on approximately 1.7 billion images with a 7-billion-parameter teacher.Source: A. Vision Foundation Models · p. 2
B. World Models and Future Forecasting
- Pixel-space forecasting is costly and models task-irrelevant detail, while latent forecasting reduces pixel-level noise but still relies on heavy decoders.Latent models use spatio-temporal decoders to align context and reconstruct features, plus task-specific decoders for downstream decisions.Source: B. World Models and Future Forecasting · p. 2
C. Efficient and Query-based Vision Architectures
DF3 adapts query-based, encoder-only prediction to future-feature forecasting, avoiding the pixel-level generation and decoder stages used by other approaches. Its pipeline forecasts features with prediction queries and uses task queries to derive downstream outputs.
C. Efficient and Query-based Vision Architectures
- DF3 differs from pixel generators by predicting semantic features and from prior latent methods by forecasting and deriving task outputs within the frozen encoder.The conceptual comparison presents DF3 as using query injection and lightweight motion-aware fusion without decoders.Source: C. Efficient and Query-based Vision… · p. 3, Figure 1 · p. 3
A. DF3 Pipeline
- The pipeline uses a frozen DINO-based ViT and earlier-layer features as context because final-layer features lack the dense spatial structure needed for pixel-level prediction.Context features include patch and prefix tokens.Source: A. DF3 Pipeline · p. 3
- Learnable prediction queries enter the frozen encoder's terminal blocks with historical context tokens to produce the next-frame feature representation.The query outputs are refined by motion-aware context fusion before forming the forecasted feature.Source: A. DF3 Pipeline · p. 3
- Learnable task queries attend to forecasted tokens in the same terminal blocks, and lightweight MLP heads produce task-specific outputs.For segmentation, the heads produce mask and class logits that are combined into a dense map.Source: A. DF3 Pipeline · p. 3
B. Structured Query Attention
Structured query attention injects prediction queries alongside historical context tokens in the frozen encoder’s terminal blocks. Its asymmetric mask lets prediction queries access the sequence while preventing context tokens from attending to those queries.
B. Structured Query Attention
- The section uses an asymmetric attention mask to separate context modeling from future prediction.Context tokens attend freely to one another; prediction queries can attend to all tokens.Source: B. Structured Query Attention · pp. 3–4
- The terminal-block attention sequence combines prediction queries with the historical context features.The input is X = [Qpred, FL1(I1), . . . , FL1(IT)].Source: B. Structured Query Attention · p. 4
- Prediction queries can attend to all tokens, while context tokens are prevented from attending back to the prediction queries.The mask is designed to prevent information leakage and keep context representations uncorrupted during temporal interaction.Source: B. Structured Query Attention · p. 4
C. Motion-Aware Context Fusion Mechanism
MACF equips the frozen image encoder to model temporal dynamics by combining coarse flow warping with fine-grained latent cross-correlation. Its parallel branches refine prediction queries using a warped feature update and motion inferred from historical features.
C. Motion-Aware Context Fusion Mechanism
- MACF combines coarse flow warping with fine-grained latent cross-correlation to model temporal dynamics without changing the pretrained encoder weights.The method targets the challenge of capturing local motion trajectories in an abstract feature space.Source: C. Motion-Aware Context Fusion Mechanism · p. 4
- The flow branch predicts a dense latent flow field from spatial prediction queries to model macroscopic displacement into the next frame.It warps the most recent historical feature map to produce a coarsely aligned future feature.Source: C. Motion-Aware Context Fusion Mechanism · p. 4
- The warp branch forms an update direction from the difference between the warped feature and the current query state, rather than replacing the queries.The update is based on the coarsely aligned future feature.Source: C. Motion-Aware Context Fusion Mechanism · p. 4
- The cross-correlation branch soft-matches previous features within a local neighborhood, and historical semantic motion is the difference between the current feature and that matched feature.The resulting historical motion is projected to guide prediction.Source: C. Motion-Aware Context Fusion Mechanism · p. 4
L1 (IT )
The fusion update combines a warped-feature update with historical motion, then uses a learned sigmoid gate to modulate the residual added to the spatial query. A lightweight temporal extrapolator also predicts changes in global prefix semantics from consecutive prefix queries.
L1 (IT )
- The local similarity uses a learnable temperature and cosine similarity between a current feature location and a neighbor in the previous-frame feature.The resulting historical velocity is projected to form a semantic motion injection.Source: L1 (IT ) · p. 4
- The learned sigmoid gate modulates a fused residual formed by concatenating the warp update and historical motion injection before adding it to the spatial query.The fused residual is produced by an MLP.Source: L1 (IT ) · p. 5
- For global prefix tokens, a lightweight temporal extrapolator uses differences between consecutive intermediate prefix queries to predict global semantic evolution.The extrapolated motion is injected to produce the final prefix feature.Source: L1 (IT ) · p. 5
- The fusion strategy is intended to align temporal context and provide inductive biases for modeling complex dynamics.Source: L1 (IT ) · p. 5
D. Training Objectives
DF3 trains prediction queries with feature-level supervision, separately from downstream decoders, to learn task-agnostic dynamics. Its objective aligns predicted and ground-truth frozen-encoder features using cosine similarity and Huber loss.
D. Training Objectives
- DF3 trains the world model with feature-level supervision rather than joint training with downstream decoders, separating prediction-query training from task-specific prediction.The prediction queries are intended to learn generalizable, task-agnostic dynamics and can later be coupled with task-specific queries.Source: D. Training Objectives · p. 5
- The primary objective aligns predicted future features with ground-truth frozen-encoder features using cosine similarity and Huber loss.Cosine similarity encourages directional alignment, while Huber loss stabilizes feature magnitude.Source: D. Training Objectives · p. 5
- The similarity loss is computed for both patch tokens and prefix tokens to represent the scene holistically.Source: D. Training Objectives · p. 5
IV. EXPERIMENTS
Experiments evaluate DF3 on Cityscapes and demonstrate deployment in a quadruped-robot navigation simulator. Cityscapes tests use short-term and autoregressive mid-term forecasting settings, with segmentation and efficiency metrics.
IV. EXPERIMENTS
- Experiments evaluate DF3 on Cityscapes and demonstrate deployment in MATRiX, an autonomous navigation simulator for quadruped robots.The evaluation includes quantitative baseline comparisons, efficiency comparisons, ablations, and qualitative visualizations.Source: §IV · p. 5
A. Experimental Setup
- Cityscapes provides 2,975 training, 500 validation, and 1,525 test sequences; evaluation covers next-frame forecasting and autoregressive predictions through T + 3.Sequences contain 30 frames, with fine-grained semantic labels on the 20th frame.Source: A. Experimental Setup · p. 5
- Using a pretrained DINOv3-ViT-B/16 backbone whose weights remain frozen, the model is trained for 400 epochs; evaluation reports mIoU and moving-object mIoU alongside per-frame GFLOPs, latency, peak GPU memory, and parameters.Source: A. Experimental Setup · p. 5
B. Comparison Results and Efficiency
On Cityscapes, DF3 reports competitive forecasting accuracy with lower computational and memory costs. Ablations favor combining motion-fusion branches and using multiple cross-correlation search radii.
B. Comparison Results and Efficiency
- DF3 achieves 69.9 mIoU and 68.7 MO-mIoU for short-term predictions, with 36% fewer GFLOPs, 70% lower latency (292 ms per frame), 67% lower peak memory, and 41% fewer forecast parameters.DINO-Foresight yields slightly higher accuracy, while DF3 maintains robust mid-term performance.Source: B. Comparison Results and Efficiency · p. 5, Table I · p. 6
C. Ablation Studies
- The combined Warp-Xcorr fusion achieves 69.9 mIoU and 68.7 MO-mIoU, outperforming Flat, attention-only, warp-only, and xcorr-only variants.Warping handles coarse displacement and occlusion, while cross-correlation refines local alignment and fast motion.Source: C. Ablation Studies · p. 6, Table II · p. 6
- The search-radius set r = {1, 2, 4, 8} achieves the best results at 69.9 mIoU and 68.7 MO-mIoU; adding radius 16 slightly reduces performance.The authors attribute the radius-16 drop to a matching neighborhood that may be excessive for single-step prediction.Source: C. Ablation Studies · p. 6, Table III · p. 6
D. Qualitative Results
DF3’s forecasted features capture dynamic-object motion and preserve static scene layout in Cityscapes visualizations. In MATRiX, Cityscapes-trained forecasts support quadruped trajectory planning despite coarser visual alignment, while fine-grained feature fidelity remains limited.
D. Qualitative Results
- DF3 captures moving-object positions and preserves static background structure in Cityscapes segmentation visualizations against the Oracle.Examples include a rapidly moving car and crossing cyclists.Source: D. Qualitative Results · p. 6
- Forecasting within a frozen ViT’s downsampled feature space produces coarse object boundaries, edge noise, and loss of sharpness in thin structures.The authors describe this spatial over-smoothing as an inherent compromise of the approach.Source: D. Qualitative Results · pp. 6–7
- Cityscapes-trained DF3 transfers without fine-tuning to quadruped trajectory planning in MATRiX, with predicted trajectories spatially aligned to Oracle trajectories.The visualizations are coarser with slight spatial misalignments, yet the authors report that the predictions retain motion dynamics needed for safe planning.Source: D. Qualitative Results · p. 7, Figure 4 · p. 7
V. CONCLUSIONS
DF3 forecasts future states and derives task outputs in latent space using learnable queries and motion-aware context fusion. The authors report state-of-the-art forecasting performance with reduced computational overhead, but the framework currently supports only observation-based forecasting.
V. CONCLUSIONS
- DF3 uses learnable spatial queries in a frozen vision foundation model with lightweight motion-aware fusion to align historical tokens and forecast future features.The framework directly derives task outputs without heavy decoders.Source: §V · p. 7
- DF3 achieves state-of-the-art forecasting performance while significantly reducing computational overhead, according to the authors.The paper presents the framework as a scalable foundation for real-time robotic decision-making.Source: §V · p. 7
- DF3 is limited to observation-only future forecasting and does not support action-conditional prediction for interactive closed-loop control and planning.The authors identify incorporating action conditions as future work.Source: §V · pp. 7–8