PaperlayerView exact revision on arXiv

Source-linked AI summary

Foundation4D: Towards a unified perception and reconstruction model for autonomous vehicles

NVIDIA, :, Sérgio Agostinho, Maximilian Apodaca, Emmanuel Attia, Guillem Braso, George Dong, Ilya Fedorov, Michele Fenzi, Gang Huang, Jiahui Huang, Shengyu Huang, Bingxin Ke, Laura Leal-Taixé, Kuan-Ting Lee, Mingxuan Liu, Tian Liu, Tim Meinhardt, Aljoša Ošep, Naveen Kumar Rai, Kefei Ren, Cristiano Saltori, Keqiang Wu, Ning Xu, Fei Xue, Jin-Bey Yu, Lei Zhang, Qunjie Zhou

arXiv:2610.30340v1cs.CV

TL;DR

Driving perception and simulation reconstruction both need scene geometry, motion, and objects, but reconstruction also needs appearance and multi-camera video makes global attention costly. Foundation4D shares a feed-forward model across perception and Gaussian reconstruction, with video-level alignment and a dense-annotation benchmark. It achieves leading comparisons across depth, detection, and occupancy benchmarks, while its task-combination findings and remaining errors mark limits of the current recipe.

  • Problem

    Driving perception and simulation reconstruction require scene geometry, objects, motion, and— for novel-view simulation—scene appearance, while efficient use of multi-camera video observations across views and time remains a challenge.

    Source: §1 · p. 3

  • Method

    Foundation4D uses a shared multi-camera video model with task-specific decoders, factorized spatiotemporal attention, video-level alignment and tracking, and dense annotations released in PAI–Foundation4D.

    Source: §1 · pp. 3–4

  • Results

    Foundation4D attains the lowest depth error on Waymo and nuScenes, the highest detection accuracy on nuScenes, and the highest occupancy accuracy on OpenOccupancy among compared methods; task-combination benefits differ by task.

    Source: §1 · p. 3, §10 · p. 43

  • Takeaways & Limitations

    Predicted geometry and trajectories provide priors for lidarless NuRec, and the PAI–Foundation4D annotations support comparison of geometry, detection, and occupancy methods on common driving scenes.

    Source: §1 · p. 3

    Depth errors increase at long range and around object boundaries, while differences in camera capture times can complicate pose estimation because rig supervision may mix fixed calibration with ego motion.

    Source: Geometry · p. 43

Abstract

from arXiv · show

We introduce Foundation4D, a unified feed-forward model for geometry estimation, 3D detection, semantic occupancy, and Gaussian reconstruction from multi-camera driving video. A shared backbone uses factorized spatiotemporal attention to efficiently combine observations across cameras and time. Our video inference pipeline uses pose graph optimization to refine camera trajectories and assembles object tracks across input windows. We also develop a data annotation pipeline and release PAI-Foundation4D, a benchmark with dense depth and semantic occupancy labels for 1,253 driving clips. With benchmark-specific fine-tuning, Foundation4D achieves the lowest depth AbsRel on Waymo and nuScenes, the highest detection mAP on nuScenes, and the highest occupancy mIoU on OpenOccupancy among the compared methods. Our task-combination study shows benefits for both detection and geometry: co-training detection with geometry and occupancy improves top-class mAP by 1.2 points over detection-only training, while co-training geometry with detection reduces camera-to-rig translation error by $13.4\%$ relative to geometry-only training. We further use the predicted geometry and trajectories as priors for lidarless NuRec, supporting reconstruction of driving scenes for simulation.

1. Introduction

Unified perception and reconstruction could let one model understand driving scenes and recreate them for simulation, but multi-camera video makes efficient cross-view and temporal reasoning important. Foundation4D combines these capabilities and is evaluated with a dense-annotation benchmark and established benchmarks.

1. Introduction

  • Unified perception models share visual representations rather than repeatedly processing observations with separate networks, while simulation additionally requires scene appearance for novel-view rendering.Existing models connect perception with prediction and planning, or jointly predict depth, 3D objects, and semantic occupancy [3].Source: §1 · p. 3
  • Global attention over all image tokens grows quadratically, making longer multi-camera sequences costly to process.Recent feed-forward reconstruction models recover camera parameters and dense geometry or predict renderable driving scenes without per-scene optimization [5].Source: §1 · p. 3
  • Foundation4D uses a shared multi-camera video backbone with task-specific decoders and factorized spatiotemporal attention, then aligns window predictions into long-sequence geometry and camera and object trajectories.Its geometry and trajectories also provide priors for lidarless NuRec, while predicted ego poses preserve rendering quality close to ground-truth ego poses.Source: §1 · pp. 3–4
  • The PAI–Foundation4D benchmark provides 1,253 clips with dense depth and semantic occupancy labels and documented protocols for comparing geometry, detection, and occupancy methods on shared scenes.The annotation pipeline addresses alignment challenges from object motion, asynchronous sensing, and rolling-shutter exposure.Source: §1 · p. 3
  • Foundation4D achieves the lowest depth error on Waymo and nuScenes, the highest detection accuracy on nuScenes, and the highest semantic occupancy accuracy on OpenOccupancy among compared methods.With benchmark-specific fine-tuning, longer temporal context improves detection without retraining, and pose graph optimization with a motion prior improves camera trajectory accuracy over sequential stitching on PAI.Source: §1 · p. 3
  • The paper contributes a data annotation pipeline and benchmark alongside a shared perception-and-reconstruction model and a video-level inference pipeline for aligning geometry and assembling object trajectories.The annotation pipeline accounts for sensor timing, ego and actor motion, and visibility.Source: §1 · p. 4

2. Related work

Related work spans shared multi-task representations for driving and feed-forward methods for geometry, perception, and reconstruction. Foundation4D combines these directions while explicitly predicting ego motion and camera-to-rig calibration.

2.1. Multi-task learning for autonomous driving

  • Driving multi-task models share representations across detection, tracking, forecasting, segmentation, occupancy, or planning, and the benefits of joint learning depend on the tasks combined.Pretraining methods use rendering, depth reconstruction, temporal prediction, or geometric objectives to learn features for downstream tasks.Source: §2.1 · p. 4
  • Foundation4D combines geometric and semantic perception with Gaussian reconstruction in a shared feed-forward model, explicitly estimating ego motion and camera-to-rig calibration.Related approaches include GeoUP [3], which predicts depth, objects, and occupancy, and USR-Drive [24], which jointly denoises box and Gaussian latents.Source: §2.1 · p. 4

2.2. Perception and reconstruction tasks

  • Feed-forward geometry research covers pointmaps, correspondences, camera and depth priors, efficient multi-image inference, temporal extensions, and dynamic scene reconstruction.VGGT jointly predicts cameras and dense geometry from multiple images.Source: §2.2 · p. 5
  • Driving geometry methods exploit rig structure and include approaches that infer calibration, use rig metadata, or factorize rig geometry and ego motion.Examples include DVGT, Rig3R, DriveVGGT [42], and TRIG [43].Source: §2.2 · p. 5
  • Long-video reconstruction methods manage memory and drift through sliding windows, re-anchoring, chunk alignment, loop constraints, or submap optimization.Foundation4D separates camera-to-rig calibration from ego motion so overlapping windows can constrain a shared trajectory, with corrections propagated to each camera at its capture time.Source: §2.2 · p. 5

3D Object detection and tracking.

The paper places Foundation4D’s object detection and tracking alongside semantic occupancy and Gaussian scene reconstruction. Its shared multi-camera video features feed task-specific outputs, while detections and predicted motion support trajectory construction.

3D Object detection and tracking.

  • Foundation4D decodes PETR-style queries into 3D boxes and velocities through deformable attention, and predicts ego poses directly from multi-view images rather than requiring ground-truth poses.The related GeoUP approach uses a VGGT-initialized backbone and reports nuScenes results, but requires ground-truth ego poses.Source: 3D Object detection and tracking · pp. 5–6
  • Foundation4D associates detections offline using predicted velocities and Hungarian matching, reconnects fragmented tracks in a second pass, and refines trajectories with an RTS smoother using later observations.This combines a Kalman-filter-based association mechanism with offline track refinement.Source: 3D Object detection and tracking · p. 6

Semantic occupancy.

  • Foundation4D represents semantic occupancy with BEV at 20 cm resolution up to 100 m ahead, using a compact representation to limit memory use within the unified model.Occupancy methods in related work include plane-based, dense volumetric, sparse, and hybrid representations.Source: Semantic occupancy · p. 6
  • The shared architecture combines geometry, object, and occupancy outputs with Gaussian reconstruction; at inference, the Gaussian branch uses predicted ego motion and assembled actor tracks rather than requiring ground-truth poses or tracks.Its task-specific decoders also predict camera-to-rig calibration, ego motion, metric depth, objects, velocities, and semantic occupancy.Source: Semantic occupancy · p. 7, Figure 2 · p. 7

3. Foundation4D

Foundation4D estimates geometry, motion, objects, occupancy, and renderable Gaussians from multi-camera video through shared contextual features. Its branches combine calibrated geometric prediction, 3D detection, semantic occupancy, and reconstruction.

3.1. Problem formulation

  • Foundation4D estimates camera depth and pointmaps, ego motion, 3D boxes, semantic occupancy, and renderable 3D Gaussians from multi-camera video.Detection and occupancy share a feature pyramid; geometry and Gaussian branches decode contextual features into geometric predictions and a scene representation.Source: §3.1 · p. 7, §3.2 · pp. 7–8

3.3. Shared image and spatiotemporal representation

  • Factorized attention combines within-image, cross-camera, and temporal context without global attention over all video tokens.Camera and temporal groups alternate across layers, reducing the cost of aggregating observations across views and time.Source: §3.3 · p. 8

3.4. Geometry decoders

  • The geometry branch predicts camera poses and metric depth, then constructs pointmaps by unprojecting depth through calibrated camera rays.Rig transforms are fixed across a window, while ego motion is interpolated at each image's capture time; geometry training also uses pose, relative-motion, and auxiliary semantic supervision.Source: §3.4 · pp. 8–10

3.5. 3D object detection

  • The object branch predicts timestamped 3D cuboids with class, center, dimensions, yaw, and planar velocity, then an offline tracker assembles detections into clip trajectories.The detections are predicted in vehicle-rig coordinates, and public-benchmark classifiers are adapted to each benchmark's label set.Source: §3.5 · p. 10

3.6. Semantic occupancy

  • The occupancy branch predicts binary occupancy and semantic labels on a 0.2 m voxel grid, using BEV features to limit intermediate memory use.Its BEV projector aggregates multi-scale image features using four vertically spaced reference points per BEV cell, projected into calibrated camera views.Source: §3.6 · p. 11

3.7. Gaussian reconstruction

  • The Gaussian branch represents scenes with static and actor-associated Gaussians plus a sky cubemap, using predicted poses and trajectories at inference.Actor Gaussians are transformed along interpolated tracks for target-time rendering, while static Gaussians remain fixed; training uses annotated poses and cuboid tracks.Source: §3.7 · pp. 12–13

4. Training the unified model

Training uses three internal driving datasets with geometry, detection, and occupancy annotations, and proceeds from perception training to longer temporal context and then Gaussian reconstruction. The implementation scales from four- to eight-timestamp windows on H100 GPUs.

4.1. Training data

  • The three internal training collections contain approximately 68k, 76k, and 60k clips, with scene mixes that complement one another.NRE emphasizes highway driving, PlanNet emphasizes urban and residential scenes, and MLMCF has a more balanced urban/highway mix; clips average about 12 seconds across roughly 695 driving hours.Source: §4.1 · pp. 13–14, Table 1 · p. 14

4.2. Training strategy

  • Training proceeds from joint geometry, detection, and occupancy learning on short windows, to longer temporal refinement, then Gaussian reconstruction.The first two stages train the three perception tasks; the final stage warms up Gaussian heads with the existing model frozen before joint optimization with the aggregator and task heads.Source: §4.2 · p. 14

4.3. Implementation details

  • The implementation increases training windows from four timestamps per camera to eight, with representative updates taking 3.8 and 8.3 seconds, respectively, on 64 H100 GPUs.Gaussian training retains the eight-timestamp configuration and adds 10k joint updates after head warm-up.Source: §4.3 · pp. 14–15

5. Multi-camera video inference system

The video inference system aligns overlapping windows into a common reconstruction frame, then uses the assembled camera trajectory to associate detections across the full clip. It compares sequential stitching with global pose graph optimization and applies offline tracking to recover object trajectories.

5.1. Sequential window stitching

  • Sequential stitching aligns each new window to the preceding aligned window using one Sim(3) transform estimated from depth-derived correspondences across cameras and overlapping frames.Huber weighting excludes sky correspondences and weights matches by geometric-mean depth confidence; accumulated rotation, translation, and scale errors motivate global reconciliation.Source: §5.1 · p. 15

5.2. Global pose graph optimization

  • PGO jointly estimates a shared reference-camera trajectory across windows under a common metric scale, with an optional timestamp-aware prior penalizing changes in translation velocity.The prior is omitted for trajectories with fewer than three nodes.Source: §5.2 · p. 16
  • For each camera sample, the system transfers optimized reference-camera motion to its capture time while preserving its predicted relation to the source trajectory and leaving retained depth unchanged.Depth and semantic predictions are retained from the window where the sample lies closest to the temporal center.Source: §5.2 · p. 16

5.3. Offline multi-object tracking

  • An offline tracker transforms detections into a common world frame, associates them over time with gated Hungarian assignment, and smooths trajectories with a two-pass process and RTS backward smoothing.The first pass forms conservative tracks; interpolation and smoothing precede a longer-lifetime pass to reconnect fragments.Source: §5.3 · pp. 16–17

6. Physical AI Autonomous Vehicles Foundation4D

PAI–Foundation4D augments 1,253 test clips with dense depth and semantic occupancy annotations for evaluation. Its pipeline aggregates LiDAR while accounting for motion, then generates camera-view depth maps and vehicle-frame occupancy grids.

6. Physical AI Autonomous Vehicles Foundation4D

  • PAI–Foundation4D provides LiDAR-derived depth maps and occupancy grids with per-voxel semantic and velocity labels for the same 1,253 test clips and six cameras used across geometry, detection, and occupancy evaluations.Figure 4 summarizes the annotation pipeline combining temporally aggregated LiDAR, cuboid tracks, and image masks.Source: §6 · p. 17, Figure 4 · p. 18

6.1. Temporal LiDAR aggregation

  • Static LiDAR points are accumulated across the full clip, while dynamic points from up to ten scans are motion-compensated to actor poses at target timestamps, increasing coverage while preserving moving-object positions.Ego-motion compensation uses each return’s acquisition timestamp.Source: §6.1 · p. 17
  • Figure 5 presents RGB images overlaid with depth, followed by RGB-colored unprojected depth and side- and forward-view semantic occupancy examples.Its shared legend indicates depth in meters and semantic classes.Source: Figure 5 · p. 18

6.2. Depth annotation generation

  • Metric depth maps render accumulated static points and motion-compensated actors into calibrated camera views, accounting for rolling-shutter exposure and checking visibility and occlusion.Invalid pixels include missing depth, sky, and ego-vehicle regions; valid values are distances along the camera ray in meters.Source: §6.2 · p. 19

6.3. Semantic occupancy annotation generation

  • Semantic occupancy labels combine image-based semantic votes and cuboid classes, then voxelize static and dynamic points separately in the vehicle frame, with dynamic voxels generally taking precedence.The exported grid uses 0.1 m cubic voxels and stores world-relative velocity expressed in vehicle coordinates.Source: §6.3 · p. 19

7. Experiments

The experiments evaluate Foundation4D’s geometry reconstruction on public benchmarks and PAI, then ablate model design and video assembly. Fine-tuned Foundation4D leads the compared methods on public depth metrics, while PAI and ablation results characterize its calibration, depth, and assembly behavior.

7.1.1. Public benchmarks

  • Foundation4D achieves the lowest depth AbsRel and highest δ1.25 on Waymo and nuScenes, and reduces video-level trajectory ATE by 29–57% versus DVGT-1.Rig mAA@15 exceeds 83% on Waymo and 99% on nuScenes.Source: Table 2 · p. 21, Table 3 · p. 21, §7.1.1 · p. 21

7.1.2. PAI benchmark

  • On PAI, stitching reduces trajectory ATE by 29.8% versus internally fine-tuned VGGT, while PGO+LM lowers ATE from 78.06 to 52.75 cm.Rig translation error decreases by 86.7%, and rig mAA@15 rises from 42.50% to 88.72%.Source: Table 4 · p. 21, §7.1.2 · p. 22
  • On PAI, PGO+LM reduces AbsRel by 8.4% beyond 60 m versus VGGT, while stitching’s relative improvement grows from 2.2% within 10 m to 7.7% beyond 60 m.The depth-range comparison uses a shared full-clip Sim(3) scale without per-bin alignment.Source: Figure 6 · p. 22, §7.1.2 · p. 22

(a) City street

  • Video reconstructions accumulate six-camera observations across clips into a common 3D frame, including the overpass geometry rendered from a predicted camera pose.The detailed view includes geometry accumulated across the full clip using depth fusion with dynamic-point restoration.Source: §7.1.2 · p. 23, Figure 8 · p. 24, Figure 9 · p. 25

7.1.3. Ablations

  • Factorized global attention reduces observed runtime by 8.5% on one H100 GPU, with a trade-off in video-level and depth accuracy.The ablations use a reduced 16-block aggregator and a shared training setup.Source: §7.1.3 · p. 25
  • Metric-scale training approximately halves video-level ATE from 141.72 to 70.67 cm, while rig-ego factorization improves trajectory, rig, and depth accuracy.Relative-pose supervision improves local translation and rotation accuracy with a small increase in depth error; auxiliary semantic supervision yields the best depth accuracy and further improves trajectory translation, though rig accuracy does not improve uniformly.Source: §7.1.3 · pp. 25–26, Table 5 · p. 26
  • PGO nearly halves ATE versus sky-masked stitching, and a linear-motion prior further improves translation consistency with a smaller additional ATE gain.PGO revises earlier estimates using overlapping relative-pose constraints; on the example sequence it reduces large longitudinal deviations, while the prior suppresses short-interval oscillations.Source: §7.1.3 · pp. 26–27, Table 6 · p. 26, Figure 10 · p. 26

7.2. 3D object detection and tracking

Foundation4D is evaluated for camera-only detection and tracking on nuScenes and PAI, then tested through model, temporal-context, and tracking ablations. Occupancy evaluations on OpenOccupancy and PAI show benchmark results and the effects of architectural and training choices.

7.2.1. Public benchmarks

  • 60.9 mAP and 66.5 NDS make Foundation4D the best nuScenes detector, exceeding GeoUP-Joint by +1.7 mAP and +1.2 NDS.A larger generic backbone alone does not explain the gains: RayDN with DINOv3 ViT-L reaches 49.9 mAP, while both geometry-pretrained ViT-L models exceed 56 mAP.Source: §7.2.1 · p. 27, Table 7 · p. 28

7.2.2. PAI benchmark

  • 38.00% mAP and 34.76% HOTA over five main classes, and 29.11% mAP and 25.66% HOTA over all twelve, are the full PAI test results.For five main classes, mAP is 52.23% within 64 m, 40.43% within 108 m, and 11.16% at 108–200 m; these distance evaluations overlap and are scored independently.Source: §7.2.2 · p. 28, Figure 11 · p. 29

7.2.3. Ablations

  • Query denoising and auxiliary 2D supervision raise detection ablation performance to 34.7% mAP and 32.5% HOTA, while FPN features and deformable attention improve mAP but lower HOTA.Higher image resolution improves both metrics while the VGGT aggregator grid stays unchanged.Source: §7.2.3 · p. 30, Table 8 · p. 30
  • Increasing inference context from 2 to 32 timestamps adds 10.07 mAP and 6.89 HOTA points without retraining; 48 timestamps add 0.59 mAP but reduce HOTA by 0.32 points.Detection continues to improve over the tested range, while tracking peaks at 32 timestamps.Source: §7.2.3 · p. 30, Table 9 · p. 30
  • Kalman filtering raises HOTA from 41.4 to 42.1, and RTS smoothing adds 1.5 points to reach 43.6, a total 2.2-point gain over velocity-based tracking.With predicted ego-poses, HOTA drops from 43.6 to 40.5 for predicted detections; with ground-truth boxes and identities, it falls from 100.0 to 83.8.Source: §7.2.3 · p. 31, Table 10 · p. 31, Table 11 · p. 31

7.3.1. Public benchmark results

  • 18.1 mIoU and 27.4 IoU give Foundation4D the highest OpenOccupancy overall scores, versus 16.2 mIoU and 25.5 IoU for SPSC-temporal.The remaining largest gaps to SPSC-temporal concern barrier and traffic cone, possibly reflecting their absence from pre-training labels.Source: §7.3.1 · p. 32, Table 12 · p. 32

7.3.2. PAI benchmark

  • 31.5 IoU and 14.1 mIoU are Foundation4D’s full-ROI PAI occupancy scores; both are higher nearby and lower at 50–100 m, where scores are 22.9/6.9.Long-range errors are particularly pronounced for thin structures and small objects; qualitative examples show road layouts, vehicles, and roadside structures.Source: Figure 13 · p. 33, §7.3.2 · p. 34, Table 13 · p. 34

7.3.3. Component ablation

  • 32.35 IoU and 15.84 mIoU are the highest full-ROI occupancy scores in the component ablation, following gains from deformable attention, VGGT, auxiliary supervision, added data, and longer training.Replacing LSS with BEV deformable attention raises full-ROI IoU/mIoU from 24.03/10.37 to 27.08/11.44; adding PlanNet and extending training further improve scores.Source: §7.3.3 · p. 34

7.4. Gaussian reconstruction

Foundation4D predicts multi-camera Gaussian reconstructions without per-scene optimization, and its reconstruction system is evaluated against Instant NuRec under distinct inference protocols. The section also examines task-combination effects and the runtime and memory of the unified model.

7.4.3. Qualitative results

  • Foundation4D’s feed-forward Gaussian predictions render synchronized views from all six native cameras without per-scene optimization.The qualitative comparisons report visual improvements over Instant NuRec in road texture, distant background geometry, and sky appearance.Source: Figure 15 · p. 36, §7.4 · p. 37, §7.4.3 · p. 38

7.4.1. Evaluation protocol

  • The comparison measures reconstruction systems rather than isolating Gaussian decoders: Foundation4D uses predicted poses, while Instant NuRec’s reference protocol requires ground-truth ego poses.Foundation4D uses six input cameras and Instant NuRec five, with each model retaining its own context-window recipe.Source: §7.4.1 · pp. 37–38

7.4.2. Quantitative comparison

  • Without ground-truth poses or tracks, Foundation4D exceeds Instant NuRec’s overall PSNR by 1.28 dB (23.94 vs. 22.66 dB) and improves SSIM by 0.02.With matching ground-truth ego poses, Foundation4D reaches 24.02 dB; dynamic-region overall PSNR is slightly lower (19.98 vs. 20.01 dB), though SSIM improves (0.73 vs. 0.72).Source: Table 15 · p. 36, §7.4.2 · p. 38

7.5. Unified model

  • The scene-level visualization combines geometry, occupancy, and tracked objects across a 20-second clip, with object geometry and occupancy accumulated in tracked frames.The ego route uses optimized camera poses and fixed camera-to-rig calibration.Source: §7.5 · p. 38, Figure 17 · p. 39

7.5.1. Task-combination study

  • Co-training detection with geometry and occupancy raises top-class mAP from 40.94% to 42.14%; geometry co-training also raises occupancy mIoU from 11.00% to 11.82%.Detection alone does not give the same occupancy benefit: DET+OCC reaches 10.49% mIoU, compared with 11.53% when geometry is included.Source: §7.5.1 · p. 39, Table 16 · p. 40
  • Co-training geometry with detection reduces trajectory ATE from 5.00 to 4.38 cm and rig RTE from 7.53 to 6.52 cm, while achieving the highest rig mAA@3.GEO-only retains the lowest depth AbsRel and rig rotation error, so task combinations trade off some geometry metrics.Source: §7.5.1 · pp. 39–40, Table 16 · p. 40

7.5.2. Runtime analysis

  • Doubling the six-camera window from 24 to 48 timestamps roughly doubles forward time, while peak allocated memory rises from 32.47 to 58.41 GiB.The 3DGS heads account for about one third of forward time, followed by the image encoder and geometry heads.Source: §7.5.2 · p. 40, Table 17 · p. 40

8. Application to lidarless NuRec

Foundation4D’s predicted scene geometry, camera trajectories, and actor tracks provide priors to lidarless NuRec, which optimizes a separate Gaussian scene against recorded images. The reported workflow comparison shows faster turnaround and an example with more consistent tracks and more complete reconstruction of a crossing vehicle.

8. Application to lidarless NuRec

  • Foundation4D predictions supply priors to lidarless NuRec, which optimizes a separate Gaussian representation for each scene against recorded images.The application uses camera trajectories, dense geometry, and actor tracks.Source: §8 · p. 41

8.1. Reconstruction workflow

  • Video-level processing assembles predicted poses and depth into camera trajectories and a fused point cloud, while persistent actor tracks separate moving objects from the static scene.NuRec uses these outputs to optimize a dynamic Gaussian scene against RGB observations for novel-view rendering.Source: §8.1 · p. 41

8.2. Reconstruction quality and efficiency

  • End-to-end workflow turnaround falls from 16.0 to 3.7 hours per clip, a 4.3× speedup.The Figure 19 example shows more consistent tracks and a more complete reconstruction of the crossing white vehicle; the comparison does not isolate individual model changes.Source: §8.2 · p. 41, Figure 19 · p. 42, Table 18 · p. 42

9. Limitations and open questions

The paper identifies limits in joint-task training, geometry estimation, and semantic occupancy. Reported errors are more pronounced for certain metrics, ranges, classes, and scene structures.

Unified model.

  • Adding tasks does not always improve performance: GEO+OCC reaches 11.82% occupancy mIoU, compared with 11.53% for the three-task model.The authors have not systematically evaluated the effects of training geometry and rendering supervision together from the outset.Source: Unified model · p. 42

Geometry.

  • Depth errors increase at long range and around object boundaries, where distant objects and thin structures occupy few pixels.Asynchronous camera capture can also make rig supervision mix fixed calibration with ego motion between exposures.Source: Geometry · p. 43

Semantic occupancy.

  • Occupancy accuracy decreases at long range, while rare classes are often missed or assigned incorrect labels.Predictions also fluctuate between frames, particularly around object boundaries and thin structures.Source: Semantic occupancy · p. 43

10. Conclusion

Foundation4D unifies perception and reconstruction from multi-camera driving video, with video-level alignment and tracking extending predictions across long sequences. The conclusion highlights the PAI–Foundation4D benchmark, task-combination findings, and use of predicted geometry and trajectories as priors for simulation reconstruction.

10. Conclusion

  • Foundation4D combines geometry, objects, semantic occupancy, and Gaussian reconstruction from shared visual features, while video-level alignment and tracking extend predictions to long driving sequences.The annotation pipeline addresses sensor timing, motion, and visibility in generating dense depth and semantic occupancy labels.Source: §10 · p. 43
  • The PAI–Foundation4D benchmark makes dense depth and semantic occupancy annotations available for comparing geometry, detection, and occupancy methods on the same driving scenes.Source: §10 · p. 43
  • Task-combination results show that detection performs best with all three tasks, while occupancy performs best when paired with geometry.Source: §10 · p. 43
  • Predicted geometry and actor trajectories provide priors for lidarless NuRec reconstruction of driving scenes for simulation.Source: §10 · p. 43
  • The conclusion identifies balancing tasks and understanding interactions between geometric and rendering supervision as open directions.Source: §10 · p. 43

A. Author list

The author list assigns named contributors to geometry, detection, occupancy, Gaussian reconstruction, the unified model, data, product work, and leadership.

A. Author list

  • Qunjie Zhou, Shengyu Huang, Mingxuan Liu, Bingxin Ke, and Fei Xue are credited with geometry.Source: A. Author list · p. 44
  • Detection, occupancy, Gaussian reconstruction, and the unified model have separately listed contributors.Detection: Guillem Braso, Sérgio Agostinho, Tim Meinhardt, Ilya Fedorov, Cristiano Saltori, and Mingxuan Liu; occupancy: Cristiano Saltori, Tim Meinhardt, Bingxin Ke, Maximilian Apodaca, and Gang Huang; Gaussian reconstruction: Ning Xu, Shengyu Huang, and Jiahui Huang; unified model: Shengyu Huang, Guillem Braso, and Tim Meinhardt.Source: A. Author list · p. 44
  • Bingxin Ke, Aljoša Ošep, Michele Fenzi, Shengyu Huang, Mingxuan Liu, Sérgio Agostinho, and Jiahui Huang are credited with data.Source: A. Author list · p. 44
  • The product team and product leadership are listed separately from project leadership.Product team: Emmanuel Attia, George Dong, Jin-Bey Yu, Kefei Ren, Keqiang Wu, Kuan-Ting Lee, Naveen Kumar Rai, Ning Xu, and Tian Liu; product leadership: Lei Zhang; leadership: Laura Leal-Taixé.Source: A. Author list · p. 44

Related papers

Loading 2610.30340v1…