PaperlayerView exact revision on arXiv

Source-linked AI summary

TriO: Tri-Modal Unsupervised Occupancy World Model for Anything Perception

Quinlan Sykora, Sourav Biswas, Christopher Diehl, Andrew Cunningham, Thomas Gilles, Raquel Urtasun

arXiv:2609.32013v1cs.CVcs.AI

TL;DR

Dense occupancy labels are costly, finite semantic classes miss rare events, and existing self-supervised approaches leave sensor modalities uncombined. TriO uses camera, LiDAR, and RADAR for inputs and self-supervision, reporting state-of-the-art performance across several occupancy, flow, forecasting, and obstacle-segmentation tasks. The authors identify rare-object motion and larger-scale anything-perception benchmarks as remaining needs.

  • Problem

    Dense voxel labels are costly to produce, labeled data are much scarcer than unlabeled data, and finite semantic classes do not capture safety-critical rare events.

    Source: §1 · p. 2

  • Method

    TriO uses camera, LiDAR, and RADAR as both inputs and self-supervision to predict and forecast occupancy, obstacle segmentation, and flow.

    Source: §1 · pp. 2–3

  • Results

    TriO reports state-of-the-art performance across zero-shot anything-perception, LiDAR forecasting, and 3D/4D occupancy prediction, including rare classes relative to the drivable surface.

    Source: §5 · p. 15

  • Takeaways & Limitations

    TriO demonstrates robust perception of objects relative to the drivable surface, including rare classes, without additional costly human annotations.

    Source: §5 · p. 15

    Perceiving and forecasting rare-object movement remains understudied, and the authors see a need for larger-scale anything-perception benchmarks.

    Source: Limitations, Discussion, and Future Work · p. 15

Abstract

from arXiv · show

We present TriO, a multi-modal unsupervised world model that predicts 4D occupancy, obstacle segmentation, flow and LiDAR. In contrast to prior work, TriO utilizes three distinct sensor modalities (camera, LiDAR, and RADAR) as both inputs and sources of self-supervision, eliminating the need for additional human annotations. Thanks to its novel supervision, the model is able to segment any occupancy from the drivable surface, overcoming the limitations of existing open-set methods in handling long-tail objects. TriO achieves state-of-the-art results in multiple 3D and 4D tasks, including occupancy, flow, and LiDAR prediction, as well as zero-shot road obstacle segmentation across multiple datasets such as Argoverse 2, and Spotting the Unexpected.

1 Introduction

TriO integrates camera, LiDAR, and RADAR as inputs and self-supervision, and reports state-of-the-art performance across occupancy, flow, point-cloud forecasting, and obstacle segmentation. Its geometric supervision targets obstacles relative to the drivable surface without additional human labels in the training pipeline.

1 Introduction

  • Dense voxel labels are costly to create at high resolutions, labeled data are commonly orders of magnitude scarcer than unlabeled data, and finite semantic classes do not capture safety-critical rare events.LiDAR self-supervision offers another training signal, but LiDAR is susceptible to blooming, multipath returns, and limited range.Source: §1 · p. 2
  • The authors define anything-perception as segmenting any object above the road from the ground surface, rather than assigning objects to a closed set of semantic classes.They motivate this target by noting that open-vocabulary segmentation can struggle with rare objects, even when prompted with the ground-truth class.Source: §1 · p. 3
  • Its image/LiDAR geometric segmentation method generates obstacle pseudo-labels without additional human labels and relies on geometry rather than semantic annotations.The method uses image-depth representations from Metric3Dv2 and a LiDAR segmentation heuristic from Patchwork++.Source: §1 · p. 3
  • TriO reports state-of-the-art performance in point-cloud forecasting, obstacle segmentation, and 3D/4D occupancy and flow forecasting across three real-world datasets.The datasets are Argoverse2, Spotting the Unexpected, and Long Range RADAR.Source: §1 · p. 3
  • TriO uses camera, LiDAR, and RADAR both as inputs and as sources of self-supervision for its output representations.The paper presents multi-sensor fusion as improving forecasting capabilities.Source: §1 · p. 3
  • Here, “unsupervised” means not requiring direct human annotations, although Metric3Dv2 was partially trained on datasets with semi-supervised human annotations.The authors refer readers to the supplementary for further discussion.Source: §1 · p. 3

2 Related Work

Related work spans dense-label semantic occupancy, open-vocabulary anything-perception methods, and self-supervised occupancy models. TriO distinguishes itself by using camera, LiDAR, and RADAR for both inputs and self-supervision in 4D forecasting.

Semantic Occupancy

  • Semantic occupancy methods commonly predict 3D or 4D voxel grids with semantic labels, relying on costly dense voxel-level supervision or pseudo-labels from vision-language models trained on human annotations.Closed-set supervision restricts generalization in open-world scenarios.Source: §2 · p. 4

Anything Perception

  • Open-vocabulary methods extend prompt-based segmentation to 3D and occupancy, but depend on image-text or mask annotations and can degrade on rare classes.TriO instead distills fused obstacle estimates from LiDAR and monocular camera data and can forecast those segmentations in 4D.Source: §2 · p. 4

Unsupervised Occupancy

  • Self-supervised occupancy work ranges from classical filtering and LiDAR-only 4D voxel grids to continuous space-time models and cross-modal RADAR input with LiDAR supervision.Explicit 4D voxel representations can impose memory and runtime costs that force a trade-off between resolution and reconstruction quality.Source: §2 · pp. 4–5
  • Unlike prior approaches described here, TriO uses camera, LiDAR, and RADAR as both inputs and self-supervision for 4D occupancy, segmentation, and flow forecasting.The paper also reports an extensive tabular comparison in the supplementary material.Source: §2 · p. 5

3 Unsupervised Multi-Modal Occupancy-Flow Method

TriO learns continuous 4D occupancy, obstacle segmentation, and flow from complementary camera, LiDAR, and RADAR observations, using each modality as input and a source of supervision. Its method combines sensor fusion with image/LiDAR obstacle labels and modality-specific pseudo-labels for training.

3 Unsupervised Multi-Modal Occupancy-Flow Method

  • TriO combines camera, LiDAR, and RADAR to learn geometry, dynamics, and object segmentation in an unsupervised 4D world model.The method uses the sensors’ complementary properties, including LiDAR geometry, RADAR velocity, and camera semantics and texture.Source: §3 · p. 5

3.1 Unsupervised Task

  • At continuous space-time query points, TriO predicts occupancy, obstacle probability, and 3D flow, trained with sensor-derived pseudo-labels.The paper motivates obstacle segmentation as a way to distinguish path-blocking objects from traversable surfaces.Source: §3.1 · pp. 5–6

3.2 Unsupervised Multi-Modal Occupancy-Flow Model

  • Separate camera and LiDAR-RADAR encoders feed voxel-attention fusion and an implicit decoder that predicts occupancy, segmentation, and flow.The camera encoder uses pretrained ResNet50; LiDAR and RADAR measurements are encoded into sparse voxel features.Source: Figure 2 · p. 6, §3.2 · pp. 6–7

3.3 Obstacle Segmentation Model

  • Obstacle labels combine image and LiDAR probabilities, using image cues when LiDAR validity is low and image validity is higher.LiDAR segmentation captures objects distinctly above ground, while image segmentation can detect smaller near-ground obstacles.Source: §3.3 · p. 7, LiDAR Obstacle Segmentation · p. 8

3.4 Multi-Modal Self-Supervision

  • Training pseudo-labels use LiDAR rays for occupancy, combined segmentation probabilities at LiDAR points, and cleaned RADAR Doppler readings for flow.The flow labels include Doppler velocity, sensor velocity, and the RADAR point’s radial direction.Source: §3.4 · pp. 8–9

3.5 Training

  • TriO trains with occupancy, flow, and obstacle-probability losses; obstacle segmentation uses soft targets weighted by label confidence.For flow, predicted motion is converted to sensor-relative radial velocity and compared with Doppler measurements using an L1 loss.Source: §3.5 · p. 9

4 Experiments

Experiments test zero-shot obstacle perception on STU, forecasting on AV2, and fine-tuned downstream occupancy and flow on AV2 and LRR. STU is used only for zero-shot evaluation after training on AV2.

4 Experiments

  • The experiments compare against open-vocabulary segmentation and LiDAR baselines, and assess pseudo-label quality, semantic occupancy, forecasting, and fine-tuned downstream tasks.STU contains 22 labeled sequences and is used without fine-tuning; LRR contains 2,280 highway and urban sequences with labels up to 350 m in front of the vehicle.Source: §4 · p. 10, §4.1 · p. 11

4.1 Zero-Shot Anything Perception (Q1)

  • On STU, CSL has the best balance of obstacle recall and false positives, while ISL surpasses evaluated baselines in recall; distilled TriO performs best, with soft CSL outperforming hard CSL on AP and mIoU.TriO also improves over LiDAR-only methods after distillation.Source: §4.1 · pp. 11–12, Table 1 · p. 11, Table 2 · p. 11

4.2 Geometric Occupancy/ LiDAR Point Cloud Forecasting (Q2)

  • On the AV2 LiDAR forecasting benchmark, TriO is best on L1 and AbsRel and competitive on other metrics; its forecasts maintain coherent obstacle segmentation as agents move.The forecasting visualization includes a bicyclist and predicted future multimodality.Source: Figure 6 · p. 12, §4.2 · p. 13, Table 3 · p. 13

4.3 Fine-tuning for Downstream Occupancy Tasks (Q3)

  • TriO outperforms fine-tuning baselines, improves over the state of the art by 6.4 AP at range, and reaches similar performance with 102 labeled examples to baselines with 104.It also outperforms baselines at all tested supervision levels and is described as two-orders-of-magnitude more sample efficient.Source: §4.3 · p. 14, Table 4 · p. 14, Figure 7 · p. 15
  • Ablations show consistent gains from unsupervised pre-training; removing RADAR input degrades long-range metrics, while camera input improves far-range occupancy at the expense of EPE.Omitting RADAR loss degrades flow metrics across ranges; on AV2, TriO lacks pre-training flow supervision because RADAR data are absent.Source: §4.3 · p. 14, Table 5 · p. 15
  • TriO currently uses RADAR for flow self-supervision; adding RADAR occupancy supervision requires techniques beyond the line-of-sight assumption because of RADAR returns.The authors also call for larger-scale benchmarks focused on anything perception.Source: §4.3 · p. 14, Limitations, Discussion, and Future Work · p. 15

5 Conclusion

TriO is an unsupervised tri-modal occupancy model for predicting 4D occupancy, obstacle semantics, and flow. The authors report state-of-the-art results across several tasks and robust perception of rare obstacles relative to the drivable surface.

5 Conclusion

  • TriO predicts 4D occupancy, obstacle semantics, and flow using camera, LiDAR, and RADAR.The paper describes the model as unsupervised and independent of additional costly human annotations.Source: §5 · p. 15
  • The authors report state-of-the-art performance in zero-shot anything perception, LiDAR forecasting, and 3D/4D occupancy prediction.The conclusion also reports that TriO outperforms open-vocabulary methods in open-world scenarios.Source: §5 · p. 15
  • TriO robustly perceives rare classes relative to the drivable surface, which the authors describe as potential for scalable SDV deployment.Source: §5 · p. 15

A Extended Related Works

TriO combines camera, LiDAR, and RADAR for unsupervised 4D occupancy and flow, while its architecture and geometric pseudo-label pipeline support obstacle segmentation and downstream forecasting.

A Extended Related Works

  • TriO is the only compared method using camera, LiDAR, and RADAR for both input and self-supervision while remaining unsupervised and modeling continuous 4D occupancy.Source: A Extended Related Works · p. 25
  • TriO uses RADAR velocity for self-supervised flow learning and additionally predicts future 4D flow.Prior self-supervised flow methods primarily rely on multi-frame LiDAR point clouds.Source: Self-Supervised Flow Predictions · pp. 25–26

B.1 Network Architecture

  • Image-to-sparse-voxel attention fuses camera and LiDAR–RADAR features, while a deformable-convolution neck and feature pyramid expand the BEV representation’s receptive field.The authors connect the larger effective receptive field to better forecasting of fast-moving entities.Source: Feature Fusion · p. 27, Deformable Attention Neck · p. 27
  • The decoder attends to multiscale sparse and dense features at each 3D query point, then predicts occupancy, flow, and obstacle probability.Source: Decoder · pp. 27–28

B.2 Obstacle Segmentation Model

  • Image obstacle supervision uses Metric3D depth, confidence, and surface normals; projected soft labels provide supervisory signals on LiDAR points.The LiDAR pipeline estimates local ground patches and converts point-to-plane elevation into obstacle probability.Source: Image Obstacle Segmentation · pp. 28–29
  • Ground-versus-obstacle labels assume ground regions have smooth normals; confidence-based fusion uses image labels when LiDAR validity is low and image validity is higher.Ambiguous LiDAR geometric classifications receive lower validity, reducing their contribution to supervision.Source: Image Obstacle Segmentation · pp. 28, 30, Fused Obstacle Segmentation · p. 30

B.3 Occupancy World Model Loss Functions

  • Occupancy is trained with binary cross-entropy over query points, while obstacle segmentation uses a soft focal loss weighted by label validity.Source: §B.3 · pp. 30–31, §B.4 · p. 31

B.4 Training of Downstream Task

B.5 Training Hyperparameters

TriO uses a multi-GPU AdamW training setup, and its unsupervised claim applies to its own training pipeline rather than the complete system.

B.5 Training Hyperparameters

  • TriO is trained on 16 GPUs with AdamW and a cosine learning-rate schedule, warming up for 1,000 iterations to a peak rate of 8.0 × 10^-4.Baselines’ hyperparameters are also tuned for fair comparison.Source: §B.5 · p. 32

B.6 Extended Discussion of Unsupervised Framing

  • “Unsupervised” means TriO’s training pipeline needs no human-provided annotations, but this claim excludes supervision involved in pretraining Metric3Dv2.Some datasets used by Metric3Dv2 include human-in-the-loop CAD model alignment.Source: §B.6 · p. 32

C Experimental Setup

The experimental setup covers Argoverse 2, Spotting the Unexpected, and the unreleased Long Range Radar dataset.

C Experimental Setup

  • Additional experimental setup details are provided for Argoverse 2, Spotting the Unexpected, and the unreleased Long Range Radar dataset.Source: C Experimental Setup · p. 32

C.1 Occupancy World Model Pre-training (AV2/LRR)

The pretraining setup uses different sensor inputs, histories, forecast horizons, and regions of interest for Argoverse 2 and Long Range Radar.

C.1 Occupancy World Model Pre-training (AV2/LRR)

  • Argoverse 2 uses the official 750-train/150-validation split, with sequences spanning 15 seconds and 150 frames.Source: §C.1 · p. 32
  • For Argoverse 2, TriO trains with two front cameras and one LiDAR, using 3.0 seconds of history at 5 Hz to predict 3.0 seconds ahead.Using four cameras in an ablation led to slight performance gains.Source: §C.1 · p. 32
  • Long Range Radar uses five cameras, six LiDARs, and four RADARs, with 0.5 seconds of history at 10 Hz to predict 1.5 seconds ahead.Its 2,280 sequences cover urban and highway driving; 19 are used for evaluation.Source: §C.1 · p. 32

C.2 Zero-Shot Anything Perception Evaluation (STU)

The STU evaluation combines labeled train-set ground points with validation-set obstacle points to test zero-shot road-obstacle segmentation, then describes the protocols and metrics for STU, AV2 LiDAR forecasting, and downstream occupancy and flow.

C.2 Zero-Shot Anything Perception Evaluation (STU)

  • STU evaluation combines ground-labeled training points with obstacle-labeled validation points; the combined set is used only for evaluation, not for training or fine-tuning AV2-trained models.The validation set has 19 sequences and no ground labels; the combination focuses evaluation on segmenting rare obstacles from the road.Source: §C.2 · p. 33
  • STU predictions are evaluated on filtered LiDAR points, with TriO queried at timestamp zero and image-baseline labels obtained by projecting points onto segmentation images.A precision–recall curve selects the F1-maximizing operating threshold; mIoU, F1, recall, and AP are reported, while precision–recall curves are used only for occupancy baselines.Source: §C.2 · pp. 33–34
  • The STU comparison includes detect-then-segment and segment-then-classify methods, heuristic ground segmentation, and an oracle variant with manually added unknown-object prompts.Prompts map to ground, obstacle, or ignore classes; the oracle variant supplies an upper-bound comparison for open-vocabulary performance.Source: Baselines · pp. 34–35, Text Prompts · pp. 35–36, Table 9 · p. 35

C.3 LiDAR Forecasting (AV2)

  • AV2 LiDAR forecasting follows the leaderboard protocol and reports L1, AbsRel, CD, and NFCD, with NFCD restricted to a 70m × 70m range.L1 is the primary AV2 leaderboard ranking metric.Source: §C.3 · p. 36

C.4 Downstream Occupancy Fine-tuning (AV2/LRR)

  • Downstream BEV/3D semantic occupancy and flow forecasting are assessed with mAP, Soft IoU, and EPE, measuring occupancy against rasterized box labels and flow-vector error.EPE is the L2 distance between ground-truth and predicted flow vectors, computed in 2D or 3D as applicable.Source: Metrics · p. 36

D.1 Quantitative Results

The quantitative analyses examine runtime and memory, camera and RADAR contributions, forecasting behavior, and sensitivity to thresholds and loss weights.

Runtime and Memory

  • At 20,000 planner query points, runtime falls substantially across methods relative to the 2,048,000-point evaluation grid; TriO without camera achieves the best runtime and is described as real-time capable.TriO’s added camera and RADAR sensors increase runtime and memory versus LiDAR-only UnO and DIO, while camera encoding raises parameter count.Source: Runtime and Memory · p. 37, Table 10 · p. 37

Additional Camera Ablations

  • Camera inputs improve occupancy performance at range, while the authors hypothesize single-timestep camera input may slightly regress flow estimation; AV2 and LRR flow effects also differ with scene composition.The authors associate AV2’s flow improvement with more static vehicles, whereas LRR contains many high-velocity objects for which single-frame camera input is less informative; LiDAR remains the dominant geometric signal except at long range.Source: Additional Camera Ablations · pp. 37–38
  • Four-camera AV2 ablations yield marginal occupancy, flow, and LiDAR gains, but not enough to justify their added cost in the main experiments.The result aligns with the main-paper no-camera ablations.Source: Additional Camera Ablations · p. 37, Table 11 · p. 38

Unsupervised Flow Estimation

  • RADAR-supervised TriO provides the best unsupervised flow estimation against DIO, with especially more accurate predictions for dynamic obstacles.DIO predicts little flow inside occupied objects and performs well on static-flow EPE by predicting zero flow.Source: Table 12 · p. 38, Unsupervised Flow Estimation · p. 38

Breakdown of AV2 Prediction Metrics

  • AV2 LiDAR forecasting performs better close by and worse at range than the reimplemented UnO and DIO baselines in the reported breakdown.The reimplemented baselines have slightly better L1 but worse CD than their leaderboard results; the authors confirmed leaderboard L1 and AbsRel are implicitly near-field metrics.Source: Breakdown of AV2 Prediction Metrics · p. 38, Table 13 · p. 39

Sensitivity Analysis

  • The current 0.5 threshold balances recall and F1, with performance degrading only at extreme threshold values; loss-weight scaling leaves reported metric deviations small.Scaling each loss weight by 4× or 0.25× yields min/max deviations of 0.1/0.2% AP, 0.0/0.4% SoftIoU, and 0.03/0.44 m EPE.Source: Sensitivity Analysis · pp. 38–39, Figure 8 · p. 39

D.2 Qualitative Results

Qualitative results show TriO across varied AV2 obstacles, compare its pseudo-labels with open-vocabulary baselines, and illustrate zero-shot STU outputs and failure cases.

D.2 Qualitative Results

  • AV2 visualizations span varied scenes and obstacle types, including vulnerable road users, construction vehicles, animals, infrastructure, and snow.The examples are presented as evidence of generalization across scenarios and obstacle types.Source: §D.2 · p. 39
  • Pseudo-label comparisons cover uncommon obstacles and show representative failures of current VLM baselines, including with oracle ground-truth text prompts.The comparisons include debris, boxes, a toy car, and garbage among the depicted cases.Source: §D.2 · p. 39
  • AV2-trained TriO rollouts on STU depict zero-shot segmentation of rare obstacles, while an AV2 rollout shows a vulnerable road user on a scooter.The qualitative results also visualize forecasted LiDAR against ground truth in a challenging traffic scene.Source: §D.2 · p. 39
  • Failure visualizations show false positives on AV2 and STU, while additional examples depict TriO handling speed bumps and hilly roads.The reported AV2 and STU failure cases are summarized separately from the edge-case examples.Source: §D.2 · p. 40

E Extended Limitations and Future Work

The discussion identifies late-fused obstacle pseudo-labels, false-positive predictions in snow and exhaust, and unresolved flow representation issues as limitations or future-work directions.

E Extended Limitations and Future Work

  • Obstacle pseudo-labels are generated independently for image and LiDAR before late fusion; both approaches rely primarily on surface normals, motivating investigation of unified early fusion.The authors also identify more efficient camera encoding as future work because camera input increases runtime and memory.Source: E Extended Limitations and Future Work · p. 40
  • Snow and vehicular exhaust sometimes cause false-positive occupancy and obstacle segmentation, and open-loop metrics may not fully capture closed-loop driving performance.Direct use of TriO in downstream motion planning is beyond the paper’s scope; additional occupancy self-supervision from cameras and RADAR could help mitigate the false positives.Source: E Extended Limitations and Future Work · p. 40
  • RADAR-supervised flow is insufficient for standalone metric flow, and tangential failures likely persist because of highway-motion biases.The authors describe resolving 3D flow from 1D radial Doppler measurements through spatially and temporally diverse observations, but note remaining tangential errors.Source: E Extended Limitations and Future Work · p. 40
Loading 2609.32013v1…