PaperlayerView exact revision on arXiv

Source-linked AI summary

HERMES++: Toward a Unified Driving World Model for 3D Scene Understanding and Generation

Xin Zhou, Dingkang Liang, Xiwu Chen, Feiyang Tan, Dingyuan Zhang, Hengshuang Zhao, Xiang Bai

arXiv:2604.28196v1cs.CV

TL;DR

Existing driving world models often separate 3D scene understanding from future geometry prediction. HERMES++ unifies both through LLM-compatible BEV representations and semantic-to-geometric knowledge transfer, outperforming specialist approaches on both tasks.

  • Problem

    Existing driving world models often separate future scene generation from comprehensive 3D scene understanding, while LLMs cannot predict future geometric evolution.

  • Method

    HERMES++ unifies both tasks using BEV representations, LLM-enhanced world queries with a Current-to-Future Link, and Joint Geometric Optimization.

  • Results

    HERMES++ reduces 3s point-cloud-generation error by 8.2% versus DriveX and improves OmniDrive-nuScenes CIDEr by 9.2% versus Omni-Q.

  • Takeaways & Limitations

    HERMES++ demonstrates the feasibility of a shared driving world model that supports both 3D scene understanding and future geometry prediction.

Abstract

from arXiv · show

Driving world models serve as a pivotal technology for autonomous driving by simulating environmental dynamics. However, existing approaches predominantly focus on future scene generation, often overlooking comprehensive 3D scene understanding. Conversely, while Large Language Models (LLMs) demonstrate impressive reasoning capabilities, they lack the capacity to predict future geometric evolution, creating a significant disparity between semantic interpretation and physical simulation. To bridge this gap, we propose HERMES++, a unified driving world model that integrates 3D scene understanding and future geometry prediction within a single framework. Our approach addresses the distinct requirements of these tasks through synergistic designs. First, a BEV representation consolidates multi-view spatial information into a structure compatible with LLMs. Second, we introduce LLM-enhanced world queries to facilitate knowledge transfer from the understanding branch. Third, a Current-to-Future Link is designed to bridge the temporal gap, conditioning geometric evolution on semantic context. Finally, to enforce structural integrity, we employ a Joint Geometric Optimization strategy that integrates explicit geometric constraints with implicit latent regularization to align internal representations with geometry-aware priors. Extensive evaluations on multiple benchmarks validate the effectiveness of our method. HERMES++ achieves strong performance, outperforming specialist approaches in both future point cloud prediction and 3D scene understanding tasks. The model and code will be publicly released at https://github.com/H-EmbodVis/HERMESV2.

I. INTRODUCTION

HERMES++ addresses the gap between driving world models that predict future scene evolution and vision-language models that understand 3D scenes but lack geometric forecasting. It unifies 3D scene understanding and future geometry prediction through BEV representations, LLM-enhanced world queries, temporal linking, and joint geometric optimization.

  • Motivation: Existing driving world models forecast visual or 3D geometric scene evolution but generally lack semantic interpretation, causal context, and mechanisms for direct scene queries.This creates a capability gap between prediction and the contextual awareness required for real-world driving.
  • Motivation: Driving-adapted VLMs interpret complex environments, answer queries, describe scenes, and reason about spatial relationships, but do not replace accurate future geometry prediction.The paper motivates combining the complementary strengths of generation-centric world models and vision-language models.
  • Framework: HERMES++ unifies 3D scene understanding and future geometry prediction using a BEV representation that consolidates multiview spatial information while remaining compatible with LLM processing.The BEV tokenizer transforms multiview images into BEV features, then downsamples and flattens them into LLM-compatible tokens.
  • Framework: LLM-enhanced world queries transfer semantic and world knowledge from scene understanding to future generation, while the Current-to-Future Link conditions predicted evolution on processed BEV features and temporal context.Textual Injection further allows semantic reasoning from scene understanding to guide future scene evolution.
  • Results: Extensive experiments show strong performance across generation and understanding, with HERMES++ outperforming prior unified baselines and several specialist approaches.The results support the feasibility and effectiveness of a shared framework for both tasks.
  • Contributions: Joint Geometric Optimization combines explicit ground-truth point-cloud constraints with implicit latent-space regularization to align predicted features with intrinsic 3D geometry.The extended version adds implicit regularization beyond the conference version’s explicit point-cloud constraints to improve future point-cloud decoding.

II. RELATED WORK … III. PRELIMINARIES

The paper reviews driving world models, LLM/VLM-based driving understanding, and BEV representations before positioning HERMES++ as a unified framework for scene comprehension and future geometric evolution. It motivates BEV as shared geometric substrate and semantic reasoning as guidance for prediction.

  • A. World Models for Driving: Driving world models learn environmental representations and forecast future scene evolution from observations and actions, supporting risk assessment and motion planning.Recent work improves consistency, resolution, controllability, temporal coherence, and 3D geometric representation through large-scale data, pretraining, DiT, autoregressive, and multimodal approaches.
  • A. World Models for Driving: Existing driving world models often predict future states without intrinsically interpreting or reasoning about the generated driving scenes.Unified generation-and-understanding models have emerged, but exploration of these capabilities remains nascent.
  • A. World Models for Driving: HERMES++ unifies driving-scenario understanding with future geometric scene evolution in a holistic, interpretable, and predictive framework.Its design addresses the gap between scene interpretation and geometric prediction rather than treating them as isolated tasks.
  • B. Large Language Models for Driving: LLMs and VLMs use large-scale pretraining to provide world knowledge and causal reasoning for semantic interpretation, agent-behavior reasoning, and natural-language explanations in autonomous driving.Driving adaptations include commands and justifications, scene-graph reasoning, 3D spatial representations, multi-task comprehension, and embodied spatial-temporal modeling.
  • B. Large Language Models for Driving: HERMES++ enables language models to comprehend the present scenario and predict its future evolution by using semantic reasoning to guide geometric prediction.This integration is intended to leverage world knowledge for structurally coherent future scenes and accurate prediction.
  • III. PRELIMINARIES: Driving world models forecast future dynamics by encoding observations, transitioning latent states conditioned on actions, and decoding predicted observations.Formally, Z_t = E(O_t), Z_t+1 = M(Z_t, A_t), and Ō_t+1 = D(Z_t+1), where the latent space filters irrelevant details.
  • III. PRELIMINARIES: BEV provides a top-down coordinate system that fuses multi-view information, preserves geometric relationships, and supports both scene understanding and generation.The representation F_BEV ∈ R^(H×W×C) is formed using learnable grid queries and deformable cross-attention over projected camera features, making it a shared geometric substrate.

IV. METHOD · A. Visual Tokenizer and BEV-to-Point Render

HERMES++ uses a BEV-based visual tokenizer to preserve multi-view geometry while compressing visual information into LLM-compatible tokens. A differentiable BEV-to-Point Render then decodes BEV features into 3D point clouds through volumetric refinement and SDF-based neural rendering.

  • IV. METHOD: HERMES++ transforms multi-view images into BEV features, compresses them into visual tokens, and combines them with instructions and learnable world queries for LLM processing.The LLM generates text while aggregating semantic context into the world queries.
  • A. Visual Tokenizer and BEV-to-Point Render: The BEV representation preserves geometric structure across views while enabling efficient compression for language interaction.The design addresses spatial discontinuity in multi-view inputs and LLM token-length constraints.
  • A. Visual Tokenizer and BEV-to-Point Render: The tokenizer encodes multi-scale perspective features with a vision encoder and spatial cross-attention to produce a BEV feature map F_bev ∈ R^(w×h×c).The BEV space captures both semantic and geometric information, with w and h denoting grid dimensions and c feature channels.
  • A. Visual Tokenizer and BEV-to-Point Render: A downsampling module using strided convolutions and pooling reduces BEV resolution by a factor of 4 before flattening and linear projection for LLM processing.This reduces token count while balancing spatial-detail preservation with computational efficiency.
  • A. Visual Tokenizer and BEV-to-Point Render: The differentiable BEV-to-Point Render operates on compressed or LLM-processed BEV features, restoring spatial resolution and expanding them into volumetric representations.Nearest-neighbor interpolation, convolutional layers, height reshaping, and 3D convolutions produce a refined volume V̂_t ∈ R^(w×h×z×c′).
  • A. Visual Tokenizer and BEV-to-Point Render: The renderer models scene geometry as an implicit signed distance function field sampled along LiDAR rays.Each ray is discretized into ordered sample points, whose local trilinearly interpolated features are mapped to SDF values by a shallow MLP.
  • A. Visual Tokenizer and BEV-to-Point Render: SDF-derived opacities and transmittance weights determine rendered ray depths, which are converted back into 3D coordinates to form the point cloud P_t.The opacity uses a sigmoid with learnable parameter τ, and the final point cloud contains K reconstructed points.

B. Unification of Understanding and Generation

HERMES++ unifies scene understanding with future evolution prediction by transferring language-based reasoning into geometric prediction. It does so through world queries that aggregate semantic information and a Current-to-Future Link for future generation.

  • The unification mechanism integrates scene understanding and future evolution prediction within HERMES++.
  • Its central challenge is transferring knowledge from language-based reasoning to geometric prediction.This transfer enables world knowledge acquired during understanding to guide future generation.
  • Two designs address this challenge: world queries aggregate semantic information, while a Current-to-Future Link supports future generation.

1) Language-based Scene Understanding: · 2) World Queries for Knowledge Transfer: · 3) Current-to-Future Link:

HERMES++ combines LLM-based multiview scene understanding with world queries and a Current-to-Future Link that transfers semantic context into controllable future geometric prediction. The design conditions spatial evolution on language, world knowledge, and ego-motion while preserving geometric and semantic alignment.

  • 1) Language-based Scene Understanding:: The LLM processes BEV features and tokenized user instructions jointly, interpreting driving scenes through next-token prediction.This makes language-based reasoning the central scene-understanding mechanism.
  • 2) World Queries for Knowledge Transfer:: World queries are injected into the LLM input sequence as latent placeholders that aggregate visual and textual semantics for knowledge transfer.They replace separate task-specific processing branches with shared representations.
  • 2) World Queries for Knowledge Transfer:: World queries are initialized across future time steps from pooled BEV features, then conditioned on ego-motion and frame embeddings before projection into the LLM dimension.The construction produces temporally conditioned queries Qw ∈R(∆t×n)×C.
  • 2) World Queries for Knowledge Transfer:: Causal attention enriches world queries with preceding BEV and textual context, while LLM pretraining contributes generalized world knowledge and causal priors.The resulting queries encode both context-specific details and generalized knowledge for geometric forecasting.
  • 3) Current-to-Future Link:: The Current-to-Future Link propagates current BEV spatial information into dense future BEV features using world queries and text embeddings.This addresses the sparsity of world queries, which provide only n queries per future time point.
  • 3) Current-to-Future Link:: Cross-attention jointly aggregates geometric information from world queries and semantic guidance from text to explicitly direct spatial evolution.Stacked blocks combine cross-attention, self-attention, and feed-forward layers.
  • 3) Current-to-Future Link:: Ego Modulation adapts self-attention and feed-forward representations using future ego-motion, while zero initialization stabilizes training and preserves semantic cross-attention.The mechanism conditions spatial representations on driving maneuvers without modulating the semantic aggregation branch.
  • 3) Current-to-Future Link:: Processed features are upsampled into future BEV representations and rendered as future point-cloud evolutions, maintaining semantic consistency and geometric alignment with vehicle behavior.The Current-to-Future Link bridges language reasoning and controllable geometric prediction.

C. Joint Geometric Optimization Strategy

The Joint Geometric Optimization strategy combines explicit depth supervision with implicit latent regularization to resolve structural ambiguity and align representations with geometry-aware 3D priors. A training-only geometric extractor provides local and global feature constraints without increasing inference cost.

  • Joint Geometric Optimization Strategy: Joint optimization constrains both observations and latent representations by combining ground-truth point-cloud geometry with implicit regularization.The strategy addresses structural ambiguity caused by rendering-only supervision, particularly for intricate geometric structures and foreground objects.
  • Explicit Geometric Constraints: An L1 rendering loss on predicted depths minimizes discrepancies from ground-truth depths, ensuring decoded point clouds align with physical measurements.The loss is weighted by frame and normalized over the number of rays.
  • Implicit Geometric Regularization: A self-supervised point-cloud reconstruction network generates geometry-aware latent priors from voxelized ground-truth point clouds using sparse 3D convolution and differentiable rendering.Pre-training reconstructs the input point cloud, enabling spatially meaningful representations.
  • Implicit Geometric Regularization: Predicted volumetric features are aligned with frozen geometry-aware features through cosine similarity for local consistency and Gram loss for global structural correlations.The Gram loss compares feature correlations across orthogonal spatial projections using the Frobenius norm.
  • Implicit Geometric Regularization: The geometry extractor is discarded after training, introducing no additional inference-time cost.It serves solely as a training-time regularizer.

D. Training Objectives

HERMES++ is trained with a composite objective that jointly optimizes language understanding, geometric rendering, and structural alignment. The total loss combines understanding and generation objectives, with geometric supervision provided by Joint Geometric Optimization.

  • Training Objectives: The composite objective jointly optimizes language understanding, geometric rendering, and structural alignment.This objective governs HERMES++ training.
  • Training Objectives: Scene understanding uses next-token prediction under a standard auto-regressive language-modeling objective.The conditional probability is modeled by LLM parameters Θ over text tokens Ti, conditioned on flattened BEV features Ft and preceding tokens.
  • Training Objectives: Future point-cloud generation is supervised through Joint Geometric Optimization.The objective integrates explicit geometric constraints from Eq. 9 with implicit geometric regularization from Eq. 10 and Eq. 12.
  • Training Objectives: The overall objective is the summation of the understanding and generation losses.This combines the language-understanding and future-geometry-generation training signals.

V. EXPERIMENTAL SETUP … VI. RESULTS AND ANALYSIS

HERMES++ is evaluated across four driving datasets plus NuInteract for vision-language alignment, using geometric, language-generation, visual-question-answering, and reasoning metrics. The experiments also describe a three-stage training setup and assess the model’s overall effectiveness.

  • A. Datasets and Evaluation Metric: Experiments use four datasets and additionally include NuInteract for vision-language alignment.NuInteract establishes initial alignment between BEV visual features and the LLM’s semantic space.
  • A. Datasets and Evaluation Metric: NuScenes provides multi-view images for unified training, with synchronized point clouds as geometric ground truth.Future-scene consistency is evaluated using bidirectional Chamfer Distance within the ROI x, y ∈[−51.2m, 51.2m] and z ∈[−3m, 5m].
  • A. Datasets and Evaluation Metric: OmniDrive-nuScenes supports refinement and instruction tuning with scene descriptions and visual QA pairs, evaluated using CIDEr, METEOR, and ROUG.The dataset targets reasoning about object interactions and traffic contexts on its validation set.
  • A. Datasets and Evaluation Metric: DriveLM evaluates graph visual question answering through official-server hybrid metrics linking perception, prediction, and planning reasoning.Its logical-chain annotations assess alignment between scene understanding and action planning.
  • B. Implementation Details: The visual tokenizer uses an OpenCLIP ConvNeXt-L backbone and encodes scenes into a 180 × 180 BEV grid with 256 channels, while InternVL2 supplies language initialization.The visual tokenizer and Render are initialized from scratch.
  • B. Implementation Details: Training proceeds through three progressive stages, beginning with geometry-aware pre-training and then reconstructing current point clouds from multi-view images.The sparse 3D encoder is frozen as a static prior after self-supervised point-cloud reconstruction, while tokenizer and Render training uses Lrender and Lcos.
  • VI. RESULTS AND ANALYSIS: The experiments are designed to validate the effectiveness of HERMES++ comprehensively.The supplied results passage states the evaluation goal but provides no numerical findings.

A. Unification of Understanding and Generation … 1) Analysis of BEV Input Representation:

HERMES++ unifies future geometric prediction and 3D scene understanding, achieving strong benchmark performance through BEV-based spatial representation and task interaction. Qualitative and ablation studies further show that BEV inputs preserve fine-grained semantics, improve future geometry prediction over multi-view inputs, and require balancing spatial resolution against computational efficiency.

  • A. Unification of Understanding and Generation: 41.6% lower 3s Chamfer Distance than ViDAR demonstrates HERMES++’s stronger future-scene generation from only current-frame observations.The comparison includes leading generation methods such as 4D-Occ and ViDAR.
  • A. Unification of Understanding and Generation: 2.3% and 11.6% CIDEr gains over Omni-L and OmniDrive-2D, respectively, are achieved without detection or map-based auxiliary supervision.The model uses BEV representation and standard instruction tuning, with consistent METEOR and ROUGE gains.
  • A. Unification of Understanding and Generation: 13.7% lower 3s generation error than the conference version accompanies improved understanding metrics and a new state-of-the-art across both tasks.The gains are attributed to deeper task interaction enabled by the newly introduced technical improvements.
  • A. Unification of Understanding and Generation: Scaling the LLM to 3.8B parameters reduces generation error to 0.97 and raises CIDEr to 0.772.The improvements occur across both generation and understanding domains.
  • B. Qualitative Evaluation: Qualitative results show that HERMES++ identifies fine-grained signage, such as “Shaw Foundation Alumni House,” and infers a likely campus setting.These results indicate that the BEV representation preserves granularity for fine-grained semantic reasoning.
  • 1) Analysis of BEV Input Representation:: Downsample (×4) provides the best generation–understanding trade-off, achieving the lowest CD at 0–3s and a CIDEr score of 0.720.The ablation compares ×4 and ×8 downsampling with Direct Query, highlighting a trade-off between geometric consistency and LLM token-processing capacity.
  • 1) Analysis of BEV Input Representation:: BEV inputs match multi-view inputs within 0.001 METEOR while reducing 3s Chamfer Distance by ∼32%.The study matches both inputs at 2,532 tokens; flattened image tokens retain semantics but undergo spatial structural collapse that harms 3D geometry recovery.

2) Analysis of Joint Geometric Optimization: · 3) Analysis of Current-to-Future Link:

Joint Geometric Optimization improves future structure generation by combining explicit constraints with implicit regularization, while the Current-to-Future Link progressively improves geometry prediction through attention, semantic context, and ego-motion conditioning.

  • 2) Analysis of Joint Geometric Optimization:: At the 3s horizon, explicit geometric constraints alone produce a CD of 1.637, whereas adding Lcos reduces the CD to 1.441.Voxel-wise feature consistency with geometry-aware priors is essential for accurate structure generation.
  • 2) Analysis of Joint Geometric Optimization:: Combining both implicit geometric regularizers achieves a CD of 1.436, indicating complementary benefits from local feature consistency and global structural coherence.The joint constraints encourage plausible 3D representations while maintaining understanding performance.
  • 2) Analysis of Joint Geometric Optimization:: Explicit geometric constraints alone cause depth ambiguity, ray-shaped artifacts along camera projection lines, and excessive response concentration at the ego-center.These effects overshadow essential scene geometry in the learned BEV features.
  • 3) Analysis of Current-to-Future Link:: Without the Current-to-Future Link, directly copying Bt and adding future ego-motion states yields a 3s CD of 2.377 and a CIDEr drop to 0.433.This naive propagation fails to model dynamic environmental evolution.
  • 3) Analysis of Current-to-Future Link:: Adding the Simple Link with 3 vanilla attention layers reduces the 3s CD to 1.542, resolving the bottleneck of naive future propagation.The Simple Link provides an attention-based mechanism for modeling future scene evolution.
  • 3) Analysis of Current-to-Future Link:: Textual Injection lowers the CD to 1.506, showing that semantic abstractions and world knowledge condition geometric generation through linguistic priors.The injected context is derived from the understanding branch.
  • 3) Analysis of Current-to-Future Link:: Ego Modulation further lowers the CD to 1.442 by injecting ego-vehicle kinematic states into feature modulation.This module provides motion-specific conditioning for future prediction.

4) Analysis of task interaction: · 5) Analysis of Hyperparameters and Configurations: · D. Generalization to Additional Tasks

The analyses show that deep task interaction improves joint future generation and scene understanding, while perceptually anchored queries and temporally continuous horizons are important for geometric prediction. The default 0–3s configuration achieves a balance between long-term prediction and semantic understanding.

  • 4) Analysis of task interaction:: Joint training improves the separated-unification baseline from CD 1.634 and CIDEr 0.703 to CD 1.436 and CIDEr 0.720.Semantic context guides geometric evolution, while geometric constraints ground language reasoning.
  • 4) Analysis of task interaction:: Textual Injection alone yields CD 1.634, whereas adding world queries for BEV aggregation improves CD to 1.526 without LLM reasoning.The comparison isolates the effect of integrating world queries with the LLM.
  • 5) Analysis of Hyperparameters and Configurations:: The study evaluates query initialization, world-query count, and temporal prediction horizon as key hyperparameters and configurations.These experiments systematically assess how architectural and temporal choices affect performance.
  • 5) Analysis of Hyperparameters and Configurations:: Randomly initialized world queries produce CD 1.448 at 3s, while BEV-based initialization consistently improves performance.BEV initialization perceptually anchors the queries to current scene features.
  • 5) Analysis of Hyperparameters and Configurations:: Using n = 0 world queries yields CD 1.478 at 3s, while n = 4 reduces the 3s CD by 0.04.World queries bridge temporal BEV features and LLM reasoning for future generation.
  • 5) Analysis of Hyperparameters and Configurations:: Short 0–1s horizons achieve CD 0.550, but discontinuous 0s-and-3s prediction degrades CD to 1.677 because intermediate states are necessary bridges.Temporal continuity supports modeling long-term geometric evolution.
  • 5) Analysis of Hyperparameters and Configurations:: The default 0–3s horizon maintains long-term prediction with CD 1.436 while preserving high semantic understanding capabilities.This setting is presented as the optimal balance between prediction length and semantic performance.

1) Understanding Capability on Other Datasets: … VII. CONCLUSION

Across additional benchmarks, motion planning, and LLM variants, HERMES++ demonstrates broad generalization while retaining strong 3D understanding and future-geometry capabilities. The conclusion highlights its unified architecture, BEV representation, and LLM-enhanced world queries for connecting semantic reasoning with geometric evolution.

  • 1) Understanding Capability on Other Datasets:: HERMES++ generalizes to NuScenes-QA and DriveLM, covering 3D spatial perception statistics and integrated perception, prediction, and planning reasoning.These evaluations extend testing beyond the original benchmarks to diverse driving scenarios.
  • 1) Understanding Capability on Other Datasets:: 61.3% accuracy makes HERMES++ state of the art on NuScenes-QA, outperforming camera-based Omni-Q by 2.1% and surpassing CenterPoint+MCAN.The passage attributes weaker LLaVA performance to limited 3D spatial modeling.
  • 1) Understanding Capability on Other Datasets:: 0.59 Final Score (FS) matches Team NVIDIA and exceeds Omni-Q and FSDrive on DriveLM.DriveLM requires integrated reasoning across perception, prediction, and planning.
  • 2) Motion Planning:: 0.37m average L2 error and 0.29% collision rate demonstrate competitive open-loop motion planning on the nuScenes validation set.A lightweight MLP head hierarchically regresses future trajectories from world queries, conditioning future point-cloud generation; training uses text instructions and future geometric supervision.
  • 2) Motion Planning:: HERMES++ achieves a 0.08% lower collision rate than ORION while maintaining comparable L2 error, and improves over OmniDrive and surpasses OmniDrive++ in average performance.The passage reports these comparisons for the nuScenes motion-planning evaluation.
  • 3) Generalization to Different LLMs:: Using 25% of the training data, HERMES++ generalizes across three representative LLM architectures with comparable parameter counts.InternVL2 provides superior generation and understanding results, including the lowest prediction error at 3s.
  • 3) Generalization to Different LLMs:: 12.5% lower 3s prediction error for the 3.8B model versus the 0.8B variant shows that larger InternVL2 models improve future prediction and scene understanding.The study reports a positive correlation between parameter count and performance.
  • VII. CONCLUSION: HERMES++ unifies 3D scene understanding and future geometry prediction through BEV features, LLM-enhanced world queries, and semantic-geometric interaction.The BEV representation consolidates multi-view visual information into an LLM-compatible format, while world queries enable knowledge transfer.
Loading 2604.28196v1…