PaperlayerView exact revision on arXiv

Source-linked AI summary

Physical AI Smart Spaces: A Large-Scale Benchmark for Multi-Camera 3D Perception in Smart Spaces

Yuxing Wang, Yizhou Wang, Anqi Li, Shuo Wang, Sameer Satish Pusegaonkar, Haoquan Liang, Jiajun Li, Shenxin Jiang, Jianhe Yuan, Shangru Li, Tongwei Dai, Zihao Chen, David C. Anastasiu, Sujit Biswas, Xunlei Wu, Zheng Tang

arXiv:2610.02580v1cs.CV

TL;DR

Multi-camera 3D perception benchmarks face trade-offs in scale, calibration, object diversity, and reproducible evaluation. PAISS combines synthetic data, appearance augmentation, and a hidden real-world target with a 3D HOTA protocol. It broadens evaluation toward multi-class RGB-only Sim2Real generalization in indoor smart spaces.

  • Problem

    Multi-camera evaluation is difficult because deployments are privacy-sensitive, manual multi-view 3D annotation is expensive, and benchmarks trade off scale, calibration quality, object diversity, and reproducibility.

    Source: §1 · p. 1

  • Method

    PAISS combines synthetic supervision, Cosmos Transfer appearance augmentation, a hidden real-world target, and 3D HOTA for location- and box-based tracking evaluation.

    Source: §1 · p. 2, §4.2 · p. 7

  • Results

    PAISS broadens smart-space evaluation toward RGB-only Sim2Real generalization for multi-class indoor perception, with 3D HOTA evaluating location- and box-based tracking.

    Source: §1 · p. 2, §5 · p. 9

  • Takeaways & Limitations

    The benchmark distinguishes evaluation of geometry-aware learning on controlled data, appearance-transfer robustness, and transfer to represented real smart-space scenes without depth at inference.

    Source: §3.2 · p. 4

    The data covers static, pre-calibrated indoor cameras and is dominated by warehouses; its synthetic avatars are not balanced for demographic or fairness studies.

    Source: §5 · p. 9

Abstract

from arXiv · show

Physical AI Smart Spaces is, to the best of our knowledge, the first benchmark to simultaneously provide large-scale, multi-class, and multi-camera 3D perception data for indoor smart spaces. It contains over 280 hours of synchronized 1080p footage captured by nearly 1,800 cameras in warehouses, hospitals, retail venues, and similar settings, together with automatic annotations for multi-camera identities, 2D bounding boxes, 3D bounding boxes, camera calibration, and depth where available. The benchmark spans Isaac Sim synthetic generation, Cosmos Transfer appearance augmentation, and real-world Sim2Real evaluation. For the real-world target, we include two warehouse deployments with time-synchronized streams, automatic VGGT-based calibration, and a 3D labeling interface that projects world-frame 3D boxes into each view for cross-camera verification. We describe the dataset scope, annotation and calibration schema, generation workflow, benchmark protocols, and official evaluation system, which standardizes submission format, and leaderboard reporting. A central contribution is a 3D instantiation of Higher Order Tracking Accuracy (HOTA), extending the usual 2D box-based tracking evaluation to 3D locations and 3D boxes. We further report empirical baselines from the AI City Challenge leaderboards, showing how methods evolve from person-only 3D location tracking to multi-class 3D box tracking under realistic smart-space constraints. The release is available at https://huggingface.co/datasets/nvidia/PhysicalAI-SmartSpaces.

1 Introduction

PAISS addresses the difficulty of evaluating multi-camera perception by combining a large-scale dataset with geometry-aware supervision and controlled synthetic-to-real evaluation. Its contributions include a documented generation pipeline, appearance augmentation, a hidden real-world target, and 3D tracking evaluation.

1 Introduction

  • Multi-camera evaluation is difficult because real deployments are privacy-sensitive, manual multi-view 3D annotation is expensive, and existing benchmarks trade off scale, calibration, object diversity, and reproducibility.These settings require systems to detect objects, reason about 3D locations, and maintain identities across overlapping and non-overlapping camera networks.Source: §1 · p. 1
  • The benchmark comprises 139 multi-camera scenes, 282.5 hours of synchronized video, 1,799 camera streams, over 72 million 3D annotations or locations, and over 241 million 2D boxes.Annotations are generated primarily with NVIDIA Omniverse and Isaac Sim rather than manual labeling.Source: §1 · pp. 1–2
  • PAISS adds Cosmos Transfer 2.5 appearance-shifted RGB and a hidden real-world target to evaluate transfer from synthetic supervision under RGB-only inference.The augmented scenes preserve geometry, calibration, depth, and ground truth while changing RGB appearance.Source: §1 · p. 2
  • The benchmark combines multi-class synchronized 3D perception data, automatic annotations and calibration, Cosmos Transfer augmentation, a hidden real-world target, and a 3D HOTA protocol for location- and box-based tracking.The paper presents these as four benchmark contributions.Source: §1 · p. 2

2 Related Works

Prior multi-target multi-camera benchmarks established cross-camera association and expanded in scale, coverage, and realism, but leave gaps for physical AI perception. PAISS occupies a complementary position through its combination of metric 3D supervision, identity consistency, multi-class mobile objects, and Sim2Real evaluation.

2 Related Works

  • Earlier MTMC benchmarks established cross-camera association as a reproducible task, while later datasets expanded scale, resolution, synthetic coverage, or real-world diversity; gaps remain in multi-class mobile objects, dense 3D supervision, depth, calibration, and controlled synthetic-to-real evaluation.Examples include PETS2009, DukeMTMC [Ristani et al., 2016], and SCOUT.Source: §2 · p. 2
  • PAISS is larger in camera count and video duration than most prior MTMC resources, and distinguishes itself through metric 3D localization, identity consistency, calibration, and multi-class mobile-object coverage.The 2026 release adds paired appearance-transfer data and a hidden real-world target without changing the task definition or annotation schema.Source: §2 · p. 3
  • Real-world datasets often reflect person tracking, while synthetic resources such as MTA and MultiviewX lack the combination of scale, 3D boxes, depth, mobile-object classes, and Sim2Real stress testing sought for physical AI.PAISS complements rather than replaces real-world datasets, supplying geometry-rich supervision and a standardized 3D evaluation protocol.Source: §2 · p. 3
  • The benchmark exposes systems to overlapping and non-overlapping camera topologies, non-human mobile objects, and visual domain shift as explicit evaluation dimensions.This separates exploiting simulator geometry from maintaining accuracy under real camera appearance, lighting, and texture statistics.Source: §2 · p. 3

3 Dataset

PAISS combines synchronized indoor camera networks and dense world-coordinate annotations with explicit evaluation assumptions and a pipeline spanning synthetic generation, appearance augmentation, and real-world data. Its evaluation claims are bounded to the benchmark’s represented settings.

3.1 Scope

  • The release covers 139 scenes, 1,799 cameras, and 282.5 hours of synchronized 1080p video, with over 72 million 3D annotations and 241 million 2D boxes across seven object classes.Camera coverage includes both overlapping and non-overlapping views; annotations include calibration and identities in a shared world coordinate system.Source: §3.1 · pp. 3–4

3.2 Evaluation Claims and Assumptions

  • The benchmark evaluates joint 3D detection, localization, and association across seven classes, camera-network conditions, and transfer from synthetic and appearance-augmented data to a hidden RGB-only real target.These claims assume simulator geometry, calibration, and automatic labels are sufficiently accurate; strong results support claims only for the benchmark’s represented settings, not arbitrary surveillance or outdoor deployments.Source: §3.2 · p. 4

3.3 Synthetic Generation and Curation

  • Synthetic actors follow random, waypoint-based, or triggered behaviors designed to exercise blind spots, occlusions, and changing camera overlap.IRA operates human and humanoid actors, while IAR operates wheeled platforms.Source: §3.3 · p. 5
  • ISP places cameras to target full ground coverage or a configurable coverage ratio, and ISC produces calibration parameters in a shared metric world coordinate system.Each rendered scene also yields synchronized RGB/depth streams and automatic identities, 2D and 3D boxes, and calibration metadata, followed by synchronization, alignment, and identity-continuity checks.Source: §3.3 · p. 5

3.4 Synthetic Augmentation

  • Cosmos Transfer 2.5 broadens visual conditions while retaining source geometry, calibration, and labels, allowing evaluation differences to be attributed to appearance rather than changed trajectories or annotations.It uses rendered RGB, a depth-derived edge map, and a text prompt to produce an appearance-augmented stream aligned with the original calibration and labels.Source: §3.4 · p. 6

3.5 Real World Target

  • The real-world target comprises two warehouse deployments, with four cameras in one and seven in the other, all time-synchronized for multi-view association and geometry.The target reflects changing lighting, clutter, and sensor variability.Source: §3.5 · pp. 6–7
  • VGGT-based calibration estimates camera parameters from synchronized video, then fits a ground plane and normalizes coordinates; annotators project each world-frame 3D box into every synchronized view for verification.The resulting camera parameters support multi-view projections, visibility estimation, and 3D box prediction.Source: §3.5 · p. 7

4 Evaluation

The section defines synchronized multi-camera 3D detection and tracking, evaluates it with world-frame 3D HOTA, and describes a standardized server and leaderboard. It also reports AI City Challenge reference systems as the task shifts from person-location tracking to multi-class 3D boxes.

4.1 Task Definition

  • The task requires detecting relevant objects across synchronized scene cameras and maintaining identities over time and across views, with world-frame coordinates, meter dimensions, and yaw.The 2026 taxonomy includes persons, named humanoid and mobile-robot classes, forklifts, and pallet trucks; 2024 evaluated 3D locations, while 2025–2026 evaluated full boxes using 3D IoU.Source: §4.1 · p. 7

4.2 3D HOTA Evaluation

  • PAISS uses 3D HOTA to balance detection, association, and localization, matching in metric 3D locations or boxes rather than image-plane overlap.The 2024 protocol used deduplicated 3D locations and a 2-meter distance cutoff; 2025–2026 used 3D IoU, with class- and scene-based aggregation.Source: §4.2 · p. 7

4.4 Reference Results from AI City Challenge Track 1

  • In 2024, person-tracking systems commonly combined 2D detection, ReID, pose- or footpoint-based projection, and cross-camera association; MCBLT used BEV detections and hierarchical GNNs.Yachiyo’s future-frame access and global association were reported to substantially improve identity consistency in long multi-camera sequences; online systems remained important for deployment.Source: Table 3 · p. 8, §4.4 · pp. 8–9

4.3 Evaluation Server and Leaderboard Protocol

  • The server checks submissions, runs the official metric, and returns scores; fixed evaluation and hidden test data support consistent ranking and separate public-data development from generalization measurement.The protocol also sets data-use and audit expectations for award contenders and compares detector-first, geometry-first, ReID-first, and hybrid systems.Source: §4.3 · p. 8

4.4 Reference Results from AI City Challenge Track 1

  • 2025 shifted to multi-class 3D boxes: ZV fused depth with 3D detection and hybrid association, while Metropolis-Sparse4D was the only online RGB-only method and scored very highly against methods using RGB and depth.The leading offline approaches also exploited depth, calibration, and sparse 3D reasoning when full sequences were available.Source: Table 4 · p. 9, §4.4 · p. 9

5 Conclusion

PAISS combines synthetic supervision, dense annotations, calibration, appearance augmentation, and hidden real-world evaluation to assess multi-camera 3D perception. Its conclusions are bounded by an indoor, static-camera setting and should not be read as evidence of deployment readiness.

5 Conclusion

  • PAISS links synthetic supervision and Cosmos Transfer appearance augmentation with hidden real-world Sim2Real evaluation for multi-camera 3D perception.The 2024 and 2025 leaderboards provide reference performance, while the 2026 benchmark targets RGB-only Sim2Real generalization for multi-class indoor perception.Source: §5 · p. 9
  • The benchmark covers indoor smart spaces captured by static, pre-calibrated 1080p cameras and is dominated by warehouses.Cosmos Transfer 2.5 is single-view, and the synthetic avatar library is finite and unbalanced across body type, skin tone, age, and mobility.Source: §5 · p. 9
  • Strong leaderboard performance is evidence of geometry-aware tracking under benchmark conditions, not deployment readiness.Real-site use requires privacy review, access controls, and validation in the target environment.Source: §5 · p. 9

Related papers

Loading 2610.02580v1…