PaperlayerView exact revision on arXiv

Source-linked AI summary

LighTROcc: Lightweight 4D Occupancy Forecasting via Instance-Centric 3D Gaussians

Hwanhee Jung, SeungHyeon Kim, Inkyu Koo, Qixing Huang, Sang Ho Yoon, Sangpil Kim

arXiv:2610.09444v1cs.CV

TL;DR

Camera-based occupancy forecasting must predict future occupied space, but existing representations can be costly and do not directly maintain object identities. LighTROcc represents movable objects with compact queries, localizes them in 3D, and propagates their occupancy forward. It outperforms evaluated baselines in occupancy metrics and reports the highest inference speed among 4D forecasting methods.

  • Problem

    Dense spatial representations can be costly as resolution and forecasting horizon grow, and existing approaches do not directly maintain object identities over time.

    Source: §1 · pp. 1–2

  • Method

    LighTROcc uses compact learned instance queries, sequentially propagates query states, and models each instance’s volumetric extent with a Gaussian mixture.

    Source: §1 · p. 2, §3.2 · p. 4, §3.4 · p. 5

  • Results

    LighTROcc outperforms evaluated baselines in IoU and AP while achieving the highest inference speed among 4D forecasting methods.

    Source: §5 · p. 9

  • Takeaways & Limitations

    The compact instance-centric formulation forecasts present and future occupancy in a single forward pass while maintaining instance-specific representations.

    Source: §1 · p. 2, §5 · p. 9

    The task is scoped to forecasting general movable object occupancy from historical multi-view images with corresponding camera parameters and ego poses.

    Source: §3.1 · p. 3

Abstract

from arXiv · show

Forecasting future 3D occupancy from surround-view cameras is essential for autonomous driving, yet existing approaches rely on dense voxel or bird's-eye-view representations whose cost grows rapidly with spatial resolution and prediction horizon. Because these representations do not explicitly maintain object identities, they also struggle to preserve instance consistency over time. We present LighTROcc, a lightweight instance-centric framework that represents movable objects with a compact set of learned queries and predicts present and future occupancy in a single forward pass. LighTROcc localizes each query through attention-guided forward lifting, combining image-space cross-attention, query-specific depth, and camera geometry to estimate its 3D center. Each instance is modeled as a mixture of anisotropic 3D Gaussians and propagated across future steps using predicted displacements, producing continuous, temporally consistent occupancy forecasts. Experiments on nuScenes and supplemented nuScenes-Occupancy show that LighTROcc outperforms the evaluated dense and instance-wise baselines in instance-level forecasting accuracy while maintaining strong voxel-level occupancy quality. Across different model configurations, LighTROcc achieves a favorable balance between forecasting accuracy and computational efficiency, demonstrating the potential of compact instance-centric modeling for camera-based 4D occupancy forecasting.

1 INTRODUCTION

Future occupancy forecasting matters for planning because vehicles must anticipate whether space will remain traversable. LighTROcc addresses the cost and identity-consistency limits of existing approaches with compact instance queries and reports stronger results across evaluated baselines.

1 INTRODUCTION

  • Current-state occupancy alone is insufficient for planning, which requires anticipating whether free space will remain traversable or become occupied.4D occupancy forecasting predicts future 3D occupancy states from past observations and commonly focuses on general movable objects.Source: §1 · p. 1
  • Existing dense voxel and BEV forecasts incur rapidly growing computation with resolution and horizon, while indirect instance recovery can merge distinct objects or fragment one object across frames.Column-based reconstruction can also inadequately capture complex geometry and fine-grained volumetric extents.Source: §1 · p. 2
  • Spatially initialized instance queries often require large candidate sets and represent the current scene rather than persistent object-specific states across future frames.This can create redundant predictions and unnecessary computational overhead.Source: §1 · p. 2
  • LighTROcc localizes compact queries through attention-guided forward lifting, represents instances with Gaussian mixtures, and propagates them using predicted displacements.It predicts multiple future occupancy states in one forward pass; experiments on nuScenes and supplemented nuScenes-Occupancy report better performance than evaluated baselines.Source: §1 · p. 2

2 RELATED WORK

The related work describes occupancy representations spanning dense grids, factorized layouts, continuous primitives, and query-based approaches. Camera-only 4D forecasting methods model future occupancy, while LighTROcc instead maintains object-specific queries across frames.

2.1 3D OCCUPANCY REPRESENTATIONS AND QUERY-BASED PERCEPTION

  • Occupancy work spans dense voxel volumes, lower-dimensional BEV or cylindrical representations, and continuous Gaussian alternatives; geometry-indexed and object-centric queries offer distinct ways to represent 3D scenes.Spatially indexed queries remain coupled to predefined discretization, while some object-centric methods still use reference points or anchors and large candidate sets.Source: §2.1 · pp. 2–3

2.2 CAMERA-ONLY 4D OCCUPANCY FORECASTING

  • Camera-only 4D forecasting methods process voxel, BEV, height, or flow representations, whereas LighTROcc represents each object with an instance-specific query propagated across future frames.The related-work passage characterizes this as preserving object-level consistency throughout the forecasting horizon.Source: §2.2 · p. 3

3 METHOD

LighTROcc uses compact learned queries to represent movable objects and forecast present and future occupancy. Its method sequentially processes multi-view observations, lifts queries into 3D, models geometry with Gaussian mixtures, and trains matched queries with multiple objectives.

3.1 PROBLEM FORMULATION

  • Each query represents an instance with confidence, depth, Gaussian-mixture parameters, and trajectory, whose continuous Gaussian volumes are rasterized into present and future voxel occupancy.The task uses two past frames and forecasts four frames.Source: §3.1 · p. 3

3.2 TRANSFORMER-BASED QUERY DECODER

  • Learned queries attend to multi-view image features and are refined by self-attention and a feed-forward network; sequentially passing each frame’s final query state to the next provides compact temporal memory.Decoder layers attend to progressively higher-resolution image features, with three layers and four attention heads.Source: §3.2 · p. 4

3.3 ATTENTION-GUIDED FORWARD LIFTING

  • Attention-guided forward lifting combines a query’s image-plane attention location with predicted depth and camera geometry to recover its 3D center.Soft-argmax provides a differentiable image-plane center, while the depth head predicts a distribution over predefined bins.Source: §3.3 · pp. 4–5

3.4 OCCUPANCY FORECASTING

  • Each instance’s volumetric extent is represented by anisotropic 3D Gaussians, and predicted displacements translate the mixture across future steps while retaining its geometry and occupancy weights.The continuous occupancy field can be discretized at different voxel resolutions; multiple future states are generated in one forward pass.Source: §3.4 · p. 5

3.5 OPTIMIZATION

  • Hungarian matching pairs permutation-invariant queries with ground-truth instances; matched pairs receive geometric, occupancy, and trajectory supervision, while unmatched queries receive background confidence targets.The objective combines confidence, center, depth, attention-localization, occupancy, and trajectory losses.Source: §3.5 · pp. 5–6

4 EXPERIMENTS

LighTROcc is evaluated on nuScenes and nuScenes-Occupancy for present and future movable-object occupancy, with voxel-level and instance-level metrics. It outperforms the evaluated baselines while balancing forecast accuracy and inference efficiency.

4.1 EXPERIMENTAL SETUP

  • Training uses 700 nuScenes scenes and 150 evaluation scenes, with two past observations, one present observation, and four future targets at 0.5–2.0-second horizons.The evaluation grid is 512×512×40 at 0.2 m voxel size; instance AP is averaged across voxel-IoU thresholds 0.1, 0.2, and 0.3.Source: §4.1 · pp. 6–7

4.2 EXPERIMENTAL RESULTS

  • IoUavg = 16.19 and APavg = 35.20 at 12.80 FPS are LighTROcc’s best reported GMO forecasting results across the listed IoU and AP metrics.Dense voxel-wise methods achieve competitive voxel-wise IoU, but have relatively low AP and weaker object-level consistency over time.Source: §4.2 · p. 7
  • LighTROcc more accurately captures future object locations while preserving consistent object representations, whereas GUIDE lacks future-state modeling and OccProphet often loses consistent geometry.Figure 4 visualizes instance-wise occupancy over time; darker colors indicate later forecasting steps.Source: §4.2 · p. 7, Figure 4 · p. 7

4.3 COMPUTATIONAL EFFICIENCY AND MODEL COMPACTNESS

  • Among 4D forecasting methods, LighTROcc has the highest inference speed and best accuracy; resolution-matched LighTROcc† runs about 1.8× faster than SparseOcc with higher accuracy.The full-resolution model maintains real-time inference while processing high-resolution inputs and forecasting multiple frames in one pass.Source: Table 2 · p. 8, §4.3 · p. 8

4.4 ABLATION STUDIES

  • Soft-argmax forward lifting achieves the best results across IoUavg, APavg, APctr, and Recallloc, while direct 3D regression performs worst and hard argmax remains sensitive to outlier peaks.Soft argmax aggregates the attention distribution to provide more reliable center grounding.Source: Table 3 · p. 8, §4.4 · p. 8, Figure 5 · p. 9
  • With full-resolution inputs, 200 queries provide the best balance between accuracy and efficiency; 100 slightly degrades AP and IoU, while 900 adds attention overhead and lowers accuracy.The study compares query count against computational overhead and forecasting accuracy.Source: Table 4 · p. 8, §4.4 · pp. 8–9
  • Raising query count from 200 to 900 improves GT-object coverage but increases invalid and duplicate predictions, weakening confidence-based discrimination.The 200-query model has a stronger ROC curve and captures more geometry-valid queries while admitting fewer invalid ones across confidence thresholds.Source: §4.4 · p. 9, Figure 6 · p. 9

5 CONCLUSION

LighTROcc uses compact learned queries, attention-guided forward lifting, and 3D Gaussian mixtures to forecast present and future occupancy in one pass. The authors report that it outperforms evaluated baselines in IoU and AP while achieving the highest inference speed among 4D forecasting methods.

5 CONCLUSION

  • Compact learned queries represent movable objects, which are localized by attention-guided forward lifting, modeled with 3D Gaussian mixtures, and propagated using predicted displacements.The framework predicts present and future occupancy in a single forward pass while retaining instance-specific representations.Source: §5 · p. 9

AI USE STATEMENT

  • AI was used only for grammar and style polishing, not research ideation, experimental design, or analysis.The authors state that they take full responsibility for the content.Source: AI USE STATEMENT · p. 10

REPRODUCIBILITY STATEMENT

  • The paper directs readers to Section 3 for architecture and training objectives and Section 4.1 for datasets and evaluation protocols.Appendix A covers dataset construction and metrics; Appendix B covers architecture and training implementation details.Source: REPRODUCIBILITY STATEMENT · p. 10

A.1 EVALUATION METRICS

The evaluation measures instance-level occupancy with threshold-averaged AP and voxel-level foreground overlap with IoU, aggregating results across present and future frames.

A.1 EVALUATION METRICS

  • Instance AP matches predicted occupancy masks to ground-truth instances by voxel IoU, averages AP over thresholds 0.1, 0.2, and 0.3, and reports each temporal step.Unmatched predictions count as false positives.Source: §A.1 · p. 14
  • AP_fut summarizes future-frame AP, while AP_avg summarizes AP across both present and future states.The per-step metric AP_k uses k = 0 for the present frame and k > 0 for future frames.Source: §A.1 · p. 14
  • Voxel-wise IoU compares merged binary foreground occupancy maps, with current, future, and average metrics aggregated over time like their AP counterparts.Dense baselines' instance masks are recovered from occupancy and flow only for instance-level evaluation; voxel IoU uses predicted foreground occupancy directly.Source: §A.1 · p. 14
  • Center localization reports confidence-ranked AP and active-query ground-truth recall using one-to-one matching by center distance at a 1 m threshold.Recall evaluates active queries after inference-time filtering.Source: §A.1 · p. 14

A.2 DATASET ANALYSIS AND GROUND TRUTH SUPPLEMENTATION

LiDAR-derived occupancy labels can be sparse and bias evaluation against plausible predictions; asset-based supplementation completes missing geometry while preserving observed annotations, and geometric checks assess its consistency.

A.2 DATASET ANALYSIS AND GROUND TRUTH SUPPLEMENTATION

  • Sparse LiDAR labels can leave distant or occluded objects fragmented, making plausible full-extent predictions appear false positive and weakening evaluation across visibility conditions.Figure 7 illustrates sparse annotations and this apparent false-positive effect.Source: §A.2 · p. 15, Figure 7 · p. 15
  • 3D ShapeNet and SMPL meshes are placed using nuScenes object pose and extent, then unioned with the original labels to fill missing volume without replacing observed geometry.Watertight meshes treat enclosed volume as occupied; the procedure is intended to reduce distance- and visibility-related evaluation imbalance.Source: §A.2 · p. 15
  • Supplementation raises overall occupancy density from 17.62% to 71.51% (4.06×), while original-label density falls below 14% beyond 30 m and reaches 8.11% beyond 50 m.Supplemented density ranges from 74.06% to 68.90% across distance, and density gaps across visibility levels are reduced.Source: §A.2 · pp. 15–16, Table 5 · p. 16

A.3 GEOMETRIC CONSISTENCY OF SUPPLEMENTED ANNOTATIONS

  • Fitted assets have mean and P90 nearest-surface distances of 0.20 m and 0.38 m, respectively, and a 2.4% free-space violation rate.At 0.2 m voxel resolution, the distances correspond to approximately 1.0 and 1.9 voxels; the results indicate increased completeness with limited conflict against observed sensor evidence.Source: §A.3 · p. 16, Table 6 · p. 17

A.4 ROBUSTNESS TO GROUND-TRUTH ANNOTATION PROTOCOL

A bbox-aware protocol avoids counting unlabeled in-box voxels as reliable negatives under sparse annotations; LighTROcc retains the highest aggregate performance under this evaluation.

A.4 ROBUSTNESS TO GROUND-TRUTH ANNOTATION PROTOCOL

  • The protocol excludes predicted voxels inside annotated 3D boxes but absent from sparse labels from false-positive counting, without treating them as true positives.It avoids treating unlabeled in-box regions as reliable negative evidence rather than assuming the whole box is occupied.Source: §A.4 · p. 17
  • Under bbox-aware evaluation of the original sparse labels, LighTROcc achieves the highest aggregate IoUavg = 9.16 and APavg = 13.93.It has the best future and average IoU and best AP across temporal aggregations, while SparseOcc has slightly higher current-frame IoU.Source: §A.4 · p. 17
  • BBox-aware evaluation still provides no positive target for unobserved regions, so missing predictions there are not counted as false negatives, especially for distant or occluded objects.The paper therefore adopts supplemented annotations as a more complete evaluation target.Source: §A.4 · p. 17

B.1 NETWORK ARCHITECTURE

The network uses a compact-query transformer with camera and temporal embeddings, then predicts per-instance Gaussian occupancy, future displacements, and confidence.

B.1 NETWORK ARCHITECTURE

  • A pretrained ResNet-18 feeds a transformer decoder with 200 learnable instance queries, embedding dimension 128, three layers, and four attention heads.Camera-parameter, 2D positional, camera-ID, and temporal embeddings encode image-token information.Source: §B.1 · pp. 17–18
  • A two-layer depth MLP predicts 64 categorical bins over 2–58 m, while a two-layer Gaussian head predicts 48 components per instance.The Gaussian components are converted to an occupancy field; each Gaussian has an occupancy weight.Source: §B.1 · p. 18
  • Lightweight two-layer MLP heads predict four future displacement vectors and each query's existence probability.The displacement vectors correspond to the four future forecasting steps.Source: §B.1 · p. 18

B.2 TRAINING DETAILS

Training combines supervised instance matching and occupancy objectives with trajectory learning from observed motion priors. The accompanying analysis explains how flow-based identity recovery in dense voxel-wise methods can fail in crowded scenes.

B.2 TRAINING DETAILS

  • LighTROcc trains for 20 epochs with AdamW, cosine-annealed learning rate, weight decay 0.01, and batch size one per GPU across eight GPUs.The initial learning rate is 3 × 10−4.Source: §B.2 · p. 18
  • Two observed displacements are concatenated with each query feature to predict four future displacement vectors, with teacher forcing reduced to zero by epoch 8.The training objective weights confidence, center, depth, attention, occupancy, and trajectory terms at 1.0, 0.3, 1.0, 1.0, 1.0, and 1.0, respectively.Source: §B.2 · p. 18

C.1 FLOW-BASED INSTANCE RECOVERY IN DENSE VOXEL-WISE MODELS

  • In crowded scenes, even small flow errors can assign voxels to incorrect identities, merging nearby objects or fragmenting one object.These association errors increase false positives and false negatives, degrading AP, which reflects occupancy overlap and instance separation.Source: §C.1 · p. 19, Figure 8 · p. 19

C.2 TEMPORAL SHAPE CONSISTENCY

LighTROcc propagates object geometry across future frames and shows stronger temporal shape consistency than a dense voxel-wise baseline. It also leads evaluated forecasting methods at each horizon, with learned trajectories outperforming constant-velocity extrapolation.

C.2 TEMPORAL SHAPE CONSISTENCY

  • Propagating Gaussian mixtures with displacement vectors while retaining component offsets, covariances, and occupancy weights preserves each instance's shape and extent across future frames.Dense voxel-wise methods instead recover identities indirectly from predicted flow, which can destabilize instance masks and extents.Source: §C.2 · p. 19
  • LighTROcc scores 91.32 TSC versus 65.67 for EfficientOCF, indicating more stable instance geometry across frames.TSC measures preservation of predicted object geometry across consecutive frames independent of global translation and orientation.Source: Table 8 · p. 19, §C.2 · p. 19

C.3 PERFORMANCE AT DIFFERENT FORECASTING HORIZONS

  • At 2.0 s, LighTROcc reaches 13.67 IoU and 27.70 AP, exceeding the strongest prior baselines by 0.89 IoU and 14.92 AP points.It achieves the best IoU and AP at every evaluated horizon; both metrics decline as the horizon increases.Source: Table 9 · p. 20, §C.3 · p. 20
  • From 0.5 to 2.0 s, learned trajectories decrease from 17.15 to 13.67 IoU (−20.3%) and from 39.35 to 27.70 AP (−29.6%), versus constant velocity's 16.30 to 11.18 IoU (−31.4%) and 35.13 to 23.19 AP (−34.0%).The comparison retains the same current instance predictions and Gaussian geometry while replacing learned future displacements with constant-velocity extrapolation.Source: §C.3 · p. 20

D ADDITIONAL QUALITATIVE RESULTS

Qualitative comparisons show LighTROcc preserving instance geometry while modeling future object evolution across diverse scenes. The baselines exhibit distinct limitations, including overestimated extents, merged instances, and occupancy that disappears at later steps.

D ADDITIONAL QUALITATIVE RESULTS

  • Across diverse scenes, LighTROcc preserves instance geometry while accurately modeling future object evolution.Cam4DOcc and EfficientOCF tend to overestimate object extents; EfficientOCF can merge nearby objects, while OccProphet's flow-based recovery can make occupancy progressively disappear at later steps.Source: D ADDITIONAL QUALITATIVE RESULTS · p. 21
Loading 2610.09444v1…