PaperlayerView exact revision on arXiv

Source-linked AI summary

HERMES: A Unified Self-Driving World Model for Simultaneous 3D Scene Understanding and Generation

Xin Zhou, Dingkang Liang, Sifan Tu, Xiwu Chen, Yikang Ding, Dingyuan Zhang, Feiyang Tan, Hengshuang Zhao, Xiang Bai

arXiv:2501.14729v3cs.CV

TL;DR

Existing driving world models generally generate future scenes without fully understanding them, while driving vision-language models understand current scenes without predicting their evolution. HERMES unifies both capabilities through BEV representations and world queries, achieving reported gains in generation accuracy and scene understanding.

  • Problem

    Existing driving world models emphasize scene generation but lack explicit scene understanding, while driving VLMs lack predictive capabilities for future scene evolution.

  • Method

    HERMES uses a BEV-based unified framework with world queries that bring LLM-derived world knowledge into future-scene generation.

  • Results

    32.4% generation error reduction and 8.0% CIDEr improvement over the cited state-of-the-art methods demonstrate gains across generation and understanding.

  • Takeaways & Limitations

    HERMES demonstrates the feasibility of integrating 3D driving-scene understanding and future-scene generation within one world model.

  • Takeaways & Limitations

    The framework does not yet explore autonomous-driving perception tasks or future-image generation.

Abstract

from arXiv · show

Driving World Models (DWMs) have become essential for autonomous driving by enabling future scene prediction. However, existing DWMs are limited to scene generation and fail to incorporate scene understanding, which involves interpreting and reasoning about the driving environment. In this paper, we present a unified Driving World Model named HERMES. We seamlessly integrate 3D scene understanding and future scene evolution (generation) through a unified framework in driving scenarios. Specifically, HERMES leverages a Bird's-Eye View (BEV) representation to consolidate multi-view spatial information while preserving geometric relationships and interactions. We also introduce world queries, which incorporate world knowledge into BEV features via causal attention in the Large Language Model, enabling contextual enrichment for understanding and generation tasks. We conduct comprehensive studies on nuScenes and OmniDrive-nuScenes datasets to validate the effectiveness of our method. HERMES achieves state-of-the-art performance, reducing generation error by 32.4% and improving understanding metrics such as CIDEr by 8.0%. The model and code will be publicly released at https://github.com/LMD0311/HERMES.

1. Introduction

HERMES addresses the separation between driving-scene understanding and future-scene generation by unifying both capabilities in one world model. It uses BEV features and world queries to connect spatial scene information with LLM-based contextual reasoning, achieving strong generation and understanding results.

  • Motivation: Existing driving world models predict scene evolution but provide limited interpretation, description, question answering, and contextual reasoning about the environment.Driving VLMs improve current-scene understanding but lack predictive capabilities for future evolution.
  • Contribution: HERMES unifies 3D scene understanding and future scene generation within a single driving world model.The framework extends LLMs to understand large-scale spatial environments while predicting future scenes.
  • Method: BEV tokenization compresses multi-view observations into a unified latent space while preserving geometric relationships and interactions across views.This addresses LLM token-length constraints when processing multiple surrounding camera views.
  • Method: World queries acquire world knowledge through causal attention and connect LLM-processed BEV features to future-scene generation.The design aims to make generated scene evolution contextually aware and bridge understanding with generation.
  • Results: 32.4% generation error reduction and 8.0% CIDEr improvement over the cited state-of-the-art methods demonstrate gains on both tasks.The generation comparison and CIDEr comparison are reported on the paper’s evaluated driving datasets.

2. Related Work

Prior driving world models primarily generate future scenes, while driving LLMs primarily understand current environments. HERMES targets the resulting gap by combining scenario comprehension with scene-evolution generation.

  • Driving World Models: Driving world-model research mainly focuses on generating future driving scenes in 2D or 3D.3D approaches include occupancy generation, rendering, and future point-cloud forecasting.
  • Driving World Models: Future point-cloud forecasting provides geometric representations of scene evolution for autonomous-driving systems.ViDAR is cited as an image-based, self-supervised approach for predicting future point clouds.
  • Research Gap: Existing driving world models overlook explicit understanding of the driving environment.The paper frames unified comprehension and scene-evolution generation as its target.
  • Large Language Models for Driving: Driving LLM research primarily applies language models to scenario understanding and perceptual or decision-making outputs.The cited direction emphasizes interpreting driving scenes rather than predicting their future evolution.

3. Preliminaries

Driving world models encode observations into latent states, predict future latent states, and decode them into future observations. BEV provides a spatially structured representation for combining multi-view geometry with future point-cloud generation.

  • Driving World Models: Driving world models learn world representations by encoding an observation, predicting its next latent state, and decoding the resulting future observation.The workflow is Ot → Lt → Lt+1 → Ot+1.
  • Driving World Models: The encoder and decoder represent the scene, while the world predictor maps latent state Lt to the next state Lt+1.These components define the preliminary DWM formulation.
  • Bird’s-Eye View: BEV offers a unified representation that preserves geometric relationships across views and captures interactions among objects and agents.BEV is presented as a natural representation for multi-view driving scenes.
  • Bird’s-Eye View: HERMES uses current multi-view observations in a BEV-based representation to generate future point clouds with geometric relationships between objects and environments.The formulation links BEV scene understanding with future point-cloud generation.

4. HERMES

HERMES encodes multi-view images into compressed BEV features, uses an LLM for scene understanding and world-knowledge acquisition, and renders linked future BEV features as point clouds. World queries and a current-to-future link connect language-based understanding with controllable future generation.

  • World Tokenizer and Render: The BEV-based world tokenizer converts multi-view images into a compressed representation that preserves semantic and geometric information for the LLM.Downsampling reduces the large BEV feature before flattening and projection into the LLM feature space.
  • World Tokenizer and Render: The BEV-to-point renderer upsamples processed BEV features, adds a height dimension, and applies 3D convolutions to reconstruct volumetric scene features for point-cloud generation.The renderer then constructs rays and uses differentiable volumetric rendering to obtain geometric information.
  • Unification: The LLM processes flattened BEV features and user instructions to describe driving scenes and answer visual questions through next-token prediction.HERMES uses the LLM to interpret the current autonomous-driving scenario.
  • Unification: World queries are initialized from peak BEV responses, repeated across future steps, and combined with ego-motion and frame embeddings for controllable generation.The query construction is summarized by the paper’s Equation 2.
  • Unification: Causal attention lets world queries access knowledge from the understanding process, while a current-to-future link injects that information into future BEV features.Three cross-attention blocks generate future BEV features before rendering.
  • Training Objectives: Training combines next-token prediction with depth supervision for rendered point clouds, using total loss L = LN + 10LD.The depth objective applies weighted L1 supervision across future frames and rays.

5. Experiments

Experiments evaluate HERMES on nuScenes and OmniDrive-nuScenes using understanding metrics and Chamfer distance for future point-cloud generation. Results show strong performance across both tasks, while ablations examine task interaction, world queries, query initialization, and BEV size.

  • Main Results: HERMES achieves competitive performance against specialist models on both scene understanding and future generation tasks.The comparison uses MTETOR, CIDEr, ROUGE, and Chamfer distance on the validation sets.
  • Main Results: ∼32% Chamfer Distance reduction in 3s point clouds is achieved compared to ViDAR using only the current frame instead of a 3s history horizon.HERMES uses multi-view inputs and a simple volumetric representation, while ViDAR uses latent rendering and a pre-trained FCOS3D backbone.
  • Main Results: 8% higher CIDEr than OmniDrive demonstrates strong caption quality relative to understanding specialists.HERMES also performs strongly on MTETOR and ROUGE, while OmniDrive uses extensive 2D pre-training, 3D-object supervision, and lane detection.
  • Ablation Study: The unified approach produces significantly better generation than separated unification while maintaining strong understanding performance.Separated unification shares flattened BEV features but uses separate processing for understanding and generation.
  • Ablation Study: 10% lower 3s point-cloud Chamfer Distance results from adding world queries, with a 1% CIDEr decrease in understanding performance.LLM processing of the queries further improves generation, supporting their role in transferring world knowledge.
  • Ablation Study: 50×50 flattened BEV spatial size improves CIDEr by 7.3% and 0s generation by 10% compared with 25×25.The improvement is attributed to reduced information loss from less aggressive downsampling.

6. Conclusion

HERMES integrates 3D scene understanding and future scene generation in one Driving World Model using BEV representations and world queries enhanced through large language models. Experiments report improvements in future scene prediction accuracy and understanding metrics.

  • HERMES unifies 3D scene understanding and future scene generation within a single Driving World Model.
  • The framework uses Bird’s-Eye View representations and world queries enhanced through large language models to connect understanding with generation.
  • Extensive experiments demonstrate significant improvements in future scene prediction accuracy and understanding metrics.
  • The authors identify un explored perception tasks and future-image generation as directions for future work.

S1. Additional Experiments

Additional experiments examine HERMES’s training setup, scaling, controllability, generation, and understanding. Results show gains from model scaling and world queries, while performance depends on query count and challenging future changes remain relevant to evaluation.

  • Training and setup: HERMES is trained through tokenizer learning, BEV-text alignment and refinement, and a final stage unifying understanding with future point-cloud generation.
  • Ablation studies: Scaling up the LLM yields consistent gains in 3D scene understanding and point-cloud generation, motivating a 1.8B-parameter model as a trade-off.
  • Ablation studies: Four world queries per group are selected because increasing their number causes performance to decline, likely from redundant information and optimization challenges.
  • Generation evaluation: Compared with Copy&Paste, HERMES learns future point-cloud changes caused by movement and occlusion rather than duplicating the current scene.
  • Understanding evaluation: HERMES outperforms LLaVA by 14.5% and CenterPoint+MCAN by 2.4% on NuScenes-QA without 3D object-detection supervision.

S2. Discussion

HERMES uses BEV tokenization to consolidate multi-view driving inputs into a spatially coherent representation for LLM processing. This preserves cross-view geometry and object interactions while reducing the burden of processing separate image streams.

  • BEV tokenization compresses multi-view inputs into a unified spatial coordinate system for LLMs.
  • The BEV representation preserves geometric relationships across views and maintains object interaction patterns.
  • This consolidation addresses the difficulty of interpreting disconnected 2D projections as a holistic 3D driving environment.

S3. More Qualitative Results

Qualitative results show controllable future generation and joint scene understanding, including conditioning on ego-motion such as stopping or turning right. The model remains challenged by complex turns, occlusions, and nighttime conditions.

  • Controllability: HERMES generates future point-cloud evolution conditioned on ego-motion such as “stop” or “turn right”.
  • Understanding and generation: The model captures future scene changes while producing detailed descriptions and answers about road layout, weather, vehicles, and traffic signs.
  • Limitations: HERMES faces challenges in complex scenes with significant turns or occlusions and in low-quality nighttime conditions.
  • Generation results: Chamfer Distance increases from 0.20 at 0s to 0.74 at 3s in one qualitative sequence.
  • Understanding and generation: The qualitative examples pair generated scene evolution with scene-understanding outputs from the same multi-view inputs.
Loading 2501.14729v3…