PaperlayerView exact revision on arXiv

Source-linked AI summary

Representations Before Pixels: Semantics-Guided Hierarchical Video Prediction

Efstathios Karypidis, Spyros Gidaris, Nikos Komodakis

arXiv:2604.11707v1cs.CV

TL;DR

Future video prediction must preserve both scene semantics and visual fidelity, but direct generation entangles these demands. Re2Pix forecasts semantic VFM representations before using them to guide latent diffusion, and experiments report improved semantic consistency, perceptual quality, and training efficiency over strong baselines.

  • Problem

    Future video prediction must simultaneously preserve high-level scene semantics and photorealistic visual details, a challenge that remains largely unsolved in dynamic driving environments.

  • Method

    Re2Pix forecasts future features from a frozen vision foundation model and conditions a latent diffusion generator on those predicted representations, using nested dropout and mixed supervision for robust conditioning.

  • Results

    Re2Pix improves temporal semantic fidelity and perceptual quality over strong baselines and accelerates convergence by up to 7× for generation and 14× for segmentation metrics.

  • Takeaways & Limitations

    Explicitly modeling hierarchical semantic structure supports semantically consistent, temporally coherent, photorealistic video prediction with improved training efficiency across multiple datasets.

  • Takeaways & Limitations

    The authors identify broader VFM feature diversity, richer 3D cues, and explicit controllability as future directions for extending Re2Pix.

Abstract

from arXiv · show

Accurate future video prediction requires both high visual fidelity and consistent scene semantics, particularly in complex dynamic environments such as autonomous driving. We present Re2Pix, a hierarchical video prediction framework that decomposes forecasting into two stages: semantic representation prediction and representation-guided visual synthesis. Instead of directly predicting future RGB frames, our approach first forecasts future scene structure in the feature space of a frozen vision foundation model, and then conditions a latent diffusion model on these predicted representations to render photorealistic frames. This decomposition enables the model to focus first on scene dynamics and then on appearance generation. A key challenge arises from the train-test mismatch between ground-truth representations available during training and predicted ones used at inference. To address this, we introduce two conditioning strategies, nested dropout and mixed supervision, that improve robustness to imperfect autoregressive predictions. Experiments on challenging driving benchmarks demonstrate that the proposed semantics-first design significantly improves temporal semantic consistency, perceptual quality, and training efficiency compared to strong diffusion baselines. We provide the implementation code at https://github.com/Sta8is/Re2Pix

1 Introduction

Re2Pix separates future video prediction into semantic forecasting and representation-guided visual synthesis, addressing the difficulty of jointly modeling scene dynamics and photorealistic appearance. It uses robust conditioning strategies to handle imperfect predicted features and reports gains in semantic fidelity, perceptual quality, and training efficiency.

  • Raw-video prediction must jointly model scene semantics and photorealistic details across temporal scales, a challenge that remains largely unsolved.
  • Direct VAE-latent prediction entangles semantic structure with visual detail, contributing to identity drift, structural degradation, flickering, slower convergence, and higher data requirements.
  • Re2Pix first forecasts future scene structure in frozen VFM feature space, then conditions latent diffusion on those representations to render photorealistic frames.
  • Nested dropout and mixed supervision improve robustness to imperfect autoregressive features caused by the train-test mismatch between clean and predicted semantics.Mixed supervision uses a 90/10 mixture of ground-truth and predicted features.
  • Re2Pix improves temporal semantic fidelity and perceptual quality over strong baselines while accelerating convergence by up to 7× for generation and 14× for segmentation metrics.

2 Related Work

Related work spans latent video prediction, semantic future forecasting, and VFM-guided generation. Re2Pix differs by evolving VFM features through time as an intermediate representation for hierarchical RGB video synthesis.

  • Video Generation and Prediction: Modern video prediction commonly uses autoregressive or masked modeling in VAE latent space to capture temporal dependencies.
  • Semantic Future Prediction: Semantic future prediction forecasts representations from pretrained networks, including dense VFM features for multi-task driving and world-modeling applications.
  • Semantic Future Prediction: V-JEPA methods predict masked video regions to learn representations, whereas Re2Pix uses VFM feature prediction as an intermediate for future RGB generation.
  • Leveraging VFM features for visual generation: VFM-guided generation has aligned VAE latents or diffusion features with semantic representations, and has extended these objectives to video.
  • Leveraging VFM features for visual generation: Unlike approaches treating VFM features as static conditioning or alternative latents, Re2Pix evolves them through time to guide temporally coherent, content-aware frame synthesis.

3 Methodology

Re2Pix separates video prediction into semantic forecasting and semantics-guided visual synthesis. It predicts future VFM representations first, then uses a latent diffusion model to generate future frames while addressing imperfect semantic conditioning.

  • High-Level Semantic Prediction: The framework predicts future scene structure in a pretrained VFM feature space before synthesizing fine-grained visual content.The semantic stage abstracts low-level details so prediction can focus on structural reasoning.
  • High-Level Semantic Prediction: A masked transformer autoregressively predicts future semantic features from the observed context, using Smooth L1 regression during training.At inference, each predicted feature frame is fed back to generate subsequent predictions.
  • Semantics-Guided Video Generation: A latent video diffusion model denoises future VAE latents conditioned on context frames, semantic features, predicted future features, and the diffusion noise step.The denoising loss applies only to future frames, after which a 3D VAE decoder reconstructs pixel-space outputs.
  • Semantics-Guided Video Generation: The VAE compresses videos across spatial and temporal dimensions, while semantic processing uses every 1/r frame to align temporal resolutions between the two stages.Here, r denotes the VAE encoder’s temporal subsampling ratio.
  • Diffusion Architecture: Semantic features and VAE latents are embedded independently and fused by channelwise summation at the diffusion model’s input.This early alignment provides joint semantic conditioning without the cross-attention layers used in the original architecture.
  • Training Strategies for Robust Semantic Conditioning: Teacher-forced ground-truth semantic features create a train–test mismatch because inference uses noisier autoregressive predictions, motivating robustness strategies.Nested dropout and mixed supervision are introduced to improve robustness to imperfect semantic inputs.

4 Experiments

Experiments evaluate Re2Pix on driving datasets using semantic consistency and generation-quality metrics, alongside baseline comparisons, convergence analysis, scaling, and ablations. Re2Pix consistently improves semantic fidelity, perceptual quality, generalization, and training convergence, while nested dropout and mixed supervision strengthen robustness.

  • Experimental Setup: Experiments use Cityscapes, nuScenes, CoVLA, and KITTI, with larger-scale training evaluated in-domain and zero-shot.The primary setup trains and evaluates on Cityscapes; extended training combines Cityscapes, nuScenes, and CoVLA, with zero-shot evaluation on KITTI.
  • Experimental Setup: Evaluation measures temporal semantic consistency through segmentation and depth, and generation quality through FID and FVD.Segmentation uses mIoU, while depth uses AbsRel and δ1; FID and FVD assess spatial realism and temporal coherence.
  • Video Prediction Results: Re2Pix improves semantic consistency and generation quality over standard diffusion, REPA, VideoREPA, and larger baselines.Table 1 reports gains across segmentation, depth, FID, and FVD; a stronger parameter-matched baseline remains inferior on both semantic and generation metrics.
  • Training Convergence: 7× acceleration applies to FID and FVD convergence, while segmentation mIoU converges 14× faster than the baseline.Re2Pix reaches FID 15 in 20k iterations versus 140k for the baseline, and FVD shows a similar 7× acceleration.
  • Scaling and Generalization: On extended data, Re2Pix outperforms baselines across Cityscapes, nuScenes, and zero-shot KITTI, surpassing Vista and competing with Cosmos-Predict 2.These results are reported despite the compared large-scale systems using several orders of magnitude more pretraining data and compute.
  • Ablation Studies: Re2Pix remains stronger than the baseline when DINOv2 is replaced by SigLIP-2, indicating robustness to the selected vision foundation model.The DINOv2 variant yields slightly stronger results in the reported comparison.

5 Conclusion

Re2Pix combines VFM representation forecasting with diffusion-based pixel synthesis to produce semantically consistent, temporally coherent, photorealistic videos. Experiments report improvements over strong baselines in quality and training efficiency.

  • Re2Pix integrates VFM representation forecasting with diffusion-based video generation in a hierarchical semantic-to-pixel framework.
  • The method first predicts future semantic representations and then uses them to guide pixel-space synthesis.
  • Re2Pix improves temporal semantic consistency, perceptual quality, and training efficiency over strong baselines, including REPA.
  • The experiments report semantically consistent, temporally coherent, and photorealistic generated videos.
  • The work received support from project MIS 5154714, GRNET, and GENCI-IDRIS HPC resources.

6 Additional Results

Additional experiments examine stochastic sampling and a CFG-inspired nested representation-guidance mechanism. The guidance can selectively improve semantic fidelity or overall generation quality, but it is not enabled in the main experiments.

  • Multiple Sampling Results: Three independent sampling runs are averaged with standard deviations to provide more robust performance estimates.Averaging slightly improves semantic metrics for both methods but modestly increases FVD.
  • CFG-style Representation Guidance with Nested Feature Dropout: Nested representation guidance contrasts predictions conditioned on all 1152 VFM feature channels with predictions using only c components.The final prediction uses guidance weight w to combine the two predictions.
  • CFG-style Representation Guidance with Nested Feature Dropout: Tables 8 and 9 vary the component count c and guidance weight w to evaluate nested representation guidance.Table 8 fixes w = 0.4, while Table 9 fixes c = 128.
  • CFG-style Representation Guidance with Nested Feature Dropout: The CFG-inspired guidance can selectively improve semantic fidelity or overall generation quality depending on hyperparameter configuration.
  • CFG-style Representation Guidance with Nested Feature Dropout: The guidance mechanism is excluded from the main experiments and left for future work.

7 Qualitative Results

Qualitative comparisons show that Re2Pix more faithfully preserves scene geometry and object boundaries than the baseline across diverse driving scenes. It also produces sharper semantic structures and smoother future feature rollouts.

  • Figures 4–8 compare context frames, future ground truth, baseline predictions, Re2Pix predictions, and predicted semantic features.Context frames are marked with green borders, while future frames use blue borders.
  • Across five diverse scenes, Re2Pix more faithfully preserves scene geometry and object boundaries than the baseline.
  • Re2Pix shows greater stability in open-road and urban traffic scenarios, with sharper semantic structures and smoother future feature rollouts.

8 Dataset Details

The experiments use four driving datasets spanning diverse urban environments, real-world driving footage, and sequences recorded in Karlsruhe. Their scales, frame rates, resolutions, and splits vary substantially.

  • Cityscapes: Cityscapes contains 2,975 training and 500 validation sequences from 50 cities, recorded at 16 fps and 1024 × 2048 resolution.Each sequence contains 30 frames captured from a driving vehicle.
  • nuScenes: nuScenes contains 1,000 driving scenes from Boston and Singapore, split into 750 training and 150 validation scenes.The scenes last 20 seconds and are recorded at 12 fps at 900 × 1600 resolution.
  • CoVLA: CoVLA comprises 10,000 approximately 30-second clips totaling over 80 hours of real-world driving footage.It is split into 8,000 training and 2,000 validation clips and recorded at 20 fps at 1208 × 1928 resolution.
  • KITTI: The dataset suite also includes KITTI driving sequences recorded in Karlsruhe at 10 fps.

9 Implementation Details

The implementation combines log-normal noise sampling, high-noise augmentation, preconditioning, and DINO-Foresight-based semantic feature prediction. DPT decoder heads provide semantic guidance from VFM features.

  • Diffusion training: Noise levels are sampled with a sigmoid-transformed standard normal variable, following Cosmos-Predict and EDM frameworks.
  • Diffusion training: 5% of training samples use very high noise levels from log σn ∼ U(log 200, log 100000) to improve initial-step denoising.
  • Diffusion training: Input and output preconditioning is applied to stabilize diffusion training through noise-level-dependent coefficients and loss weighting.
  • Semantic feature prediction: The feature predictor uses DINO-Foresight with DINOv2-Reg ViT-B/14 features sampled every fourth frame to match the WAN2.1 VAE temporal ratio r = 4.
  • Semantic feature prediction: DPT decoder heads extract semantic segmentation and depth guidance from VFM features using 256-dimensional features and specified multi-scale output channels.

10 Limitations and Future Work

Future work centers on broadening the visual foundation model guidance and adding explicit controllability for more general conditional video generation.

  • Future directions: Future research could incorporate richer 3D perception, scene geometry, and collaborative encoders to provide more structured semantic priors.
  • Future directions: Text prompts, trajectory constraints, and high-level scene graphs could extend Re2Pix toward user-driven editing and synthesis.

11 Broader Impact

The paper presents semantic video prediction as potentially useful across urban autonomy, robotics, and infrastructure monitoring. It also emphasizes that deployment quality, reliability, and fairness depend on foundation-model pretraining data and inherited biases.

  • Potential applications: Semantic forecasting through vision foundation model features is intended to support deployment across urban autonomy, robotics, and infrastructure monitoring.
  • Responsible use: The authors state that deployment quality, reliability, and fairness depend on pretraining data and biases inherited from large vision foundation models.
  • Responsible use: High-stakes applications require careful evaluation and responsible model use.
Loading 2604.11707v1…