PaperlayerView exact revision on arXiv

Source-linked AI summary

Holi-World: Building Holistic World Intelligence from Human-Ego Experience

Hao Li, Xiyuan Zheng, Yuanyuan Gao, Yalun Dai, Xinhao Ji, Xuan-Li Chen, Jing Zhao, Fangzhou Hong, Zhaoxi Chen, Ziwei Liu

arXiv:2610.26549v1cs.CV

TL;DR

Egocentric recordings capture environments, objects, and activities, but scaling detailed supervision while preserving spatial and temporal information remains a challenge. HOLIWORLD connects automated static and dynamic perception to build grounded learning resources and train a vision-language model. Its experiments report improved perception, tracking, and spatial and temporal reasoning, including higher Token F1 on HOLIWORLD-Bench.

  • Problem

    Scaling high-quality supervision from egocentric recordings while retaining the spatial and temporal information needed to understand environments, objects, and interactions remains challenging and often requires substantial human annotation and curation.

    Source: §1 · pp. 1–2

  • Method

    HOLIWORLD-Foundry combines static and dynamic perception to build world records linking scene geometry, object semantics, object states, interaction histories, and supporting observations for learning and evaluation.

    Source: §1 · pp. 2–3

  • Results

    HOLIWORLD-Foundry improves static perception and dynamic tracking, while HOLIWORLD-VL-4B reaches Token F1 78.02% versus 16.35% for the original Qwen3.5-4B on HOLIWORLD-Bench and also improves reasoning on EgoDynamic4D.

    Source: §1 · p. 3, §5 · p. 11

  • Takeaways & Limitations

    HOLIWORLD provides a practical foundation for expanding grounded learning resources with new recordings.

    Source: §5 · p. 11

    Dynamic object perception assumes at most one moving object at each time.

    Source: §3.2.3 · p. 6

Abstract

from arXiv · show

Human egocentric experience offers a rich foundation for learning how the world is organized and how it changes through interaction. Realizing this potential at scale requires turning growing volumes of recordings into learning resources that connect scene structure, object semantics, and interaction histories across viewpoints and time, while reducing dependence on costly manual annotation. We introduce HOLI-WORLD, an integrated ecosystem connecting experience capture, world representation, automated data construction, model learning, and independent evaluation. To sup- port learning at scale, HOLIWORLD-Foundry combines complementary static and dynamic perception to automatically construct world records that connect persistent scene structure and evolving object states with interaction histories and supporting observations. HOLIWORLD-Data organizes these records from approximately 100 self-collected scenes into grounded learning resources, including approximately 80K QA pairs used to train HOLIWORLD-VL-4B. Independent human annotations of heldout recordings form HOLIWORLD-Bench, supporting evaluation of both annotation quality and learned spatial and temporal understanding. Experiments on ScanNet, ScanNet++, Aria Digital Twin, and HOLIWORLD-Bench demonstrate strong performance in static perception and dynamic tracking, while training on HOLIWORLD-Data substantially improves HOLIWORLD-VL-4B's spatial and temporal reasoning on HOLI- WORLD-Bench and EgoDynamic4D. HOLIWORLD provides a practical foundation for expanding learning resources from the continuous flow of human experience and advancing machine understanding of the dynamic world.

1. Introduction

HOLIWORLD addresses the challenge of scaling egocentric learning supervision while retaining the spatial and temporal information needed to understand an evolving world. It connects capture, automated world-record construction, learning, and independent evaluation, with results showing improvements in annotation and downstream reasoning.

1. Introduction

  • Human experience reveals enduring spatial structure through movement and changing object states and relationships through interaction.The paper frames the challenge as scaling high-quality supervision from incoming egocentric recordings without losing the spatial and temporal information needed to understand environments, objects, and interactions.Source: §1 · pp. 1–2
  • HOLIWORLD-Foundry combines static and dynamic perception to connect scene geometry, object semantics, and interaction histories to supporting observations.Static detections are lifted into 3D and consolidated; dynamic objects are tracked across interaction segments with 3D states and trajectories in shared world coordinates.Source: §1 · pp. 2–3
  • HOLIWORLD-Data organizes approximately 100 scenes and 444 videos into grounded resources, including approximately 80K evidence-linked QA pairs for spatial relations, temporal changes, and interactions.HOLIWORLD-VL is trained on this supervision, while independently annotated held-out recordings form HOLIWORLD-Bench for evaluating record accuracy and learned spatial understanding.Source: §1 · p. 3
  • HOLIWORLD-Foundry achieves AP25 0.68 on ScanNet versus 0.40 for the strongest baseline, HOTA0.1 54.31% on 20 Aria Digital Twin sequences versus 36.00% for IT3DEgo, and Token F1 78.02% versus 16.35% on HOLIWORLD-Bench and 66.39% versus 18.29% on EgoDynamic4D.The tracking comparison uses IT3DEgo with ground-truth-guided initialization; the Token F1 comparisons are against the original Qwen3.5-4B.Source: §1 · p. 3
  • HOLIWORLD is an integrated ecosystem connecting experience capture, world representation, automated data construction, model learning, and independent evaluation.Its shared representation is intended to describe an evolving world.Source: §1 · p. 3

2. Related Work

Prior egocentric datasets offer broad activity coverage, but detailed supervision remains costly to extend; static scene methods and temporal tracking methods provide complementary capabilities. HOLIWORLD connects these capabilities and the supervision pipeline to support spatial and temporal reasoning from new recordings.

2.1. Egocentric Datasets and Annotations

  • Detailed supervision in egocentric datasets such as HD-EPIC requires further annotation effort to extend to new recordings; HOLIWORLD instead automatically recovers scene structure, object states, and interaction annotations for learning supervision.Prior datasets span diverse people, places, and activities, and some connect actions, object movements, and gaze to 3D digital twins.Source: §2.1 · p. 3

2.2. 3D Scene Reconstruction and Object Tracking

  • HOLIWORLD-Foundry connects static scene reconstruction with egocentric tracking through shared world coordinates and persistent identities, linking geometry, evolving object states, and interaction evidence.Static consolidation does not retain successive object states needed to describe motion and interactions; tracking methods maintain object identities and locations over time.Source: §2.2 · p. 4

2.3. Spatial and Temporal Reasoning in Vision–Language Models

  • HOLIWORLD addresses the upstream task of automatically constructing annotations from new recordings, connecting capture, annotation, supervision generation, and model training with independent human-annotated evaluation.It follows EgoDynamic4D QA patterns, while evaluating both recovered-record accuracy and learned model capabilities.Source: §2.3 · p. 4

3. Method

HOLIWORLD turns egocentric recordings into grounded learning resources by connecting scene geometry, object semantics, and interaction histories. Its pipeline reconstructs and organizes observations, builds static and dynamic object records, and derives traceable spatial-temporal supervision.

3.1. HoliWorld Overview

  • The Eagle Capture Device records synchronized stereo RGB and IMU measurements; Foundry reconstructs scenes and annotates static objects and dynamic tracks, while independent annotations of held-out recordings form Bench.Bench evaluates Foundry’s annotation quality and HoliWorld-VL’s spatial understanding.Source: §3.1 · p. 4

3.2.1. Geometric Preparation and Shared Semantic Cues

  • Visual-inertial SLAM estimates camera poses and sparse points, then depth estimation and 3D Gaussian scene optimization produce dense geometry and reconstructed observations.A VLM extracts categories, attributes, motion cues, and interaction descriptions from RGB keyframes to route objects to static or dynamic perception.Source: §3.2.1 · pp. 4–5

3.2.2. Static Object Perception

  • Image-based and point-cloud-based detectors generate complementary 3D object proposals, which geometric checks merge and filter before VLM validation.For each selected box, the pipeline retains a fallback when available; the VLM checks evidence, can refine or relabel boxes, and retries a rejected box’s fallback once.Source: §3.2.2 · pp. 5–6

3.2.3. Dynamic Object Perception

  • Hand regions and VLM interaction descriptions associate object identities across interaction segments, guiding target-mask selection and propagation within each segment.Tracked masks and completed dynamic depths are back-projected using camera poses; filtered points are fit to per-frame OBBs and linked by identity into 4D tracks.Source: §3.2.3 · pp. 6–7

3.2.4. Unified World Representation

  • The unified representation combines observations, detected boxes and masks, scene geometry, static objects, time-indexed dynamic states, and associated labels, identities, interaction semantics, confidence scores, and evidence links.This structure lets supervision refer to specific objects, times, and supporting observations.Source: §3.2.4 · p. 7

3.3. HoliWorld-Data

  • HoliWorld-Data derives grounding, captioning, and spatial, temporal, and interaction QA from world records, linking examples to source observations and annotations.It contains roughly 100 scenes and 444 videos, with about 4K 3D grounding annotations, 80K captions, and approximately 80K QA pairs.Source: §3.3 · pp. 7–8

4. Experiments

The experiments evaluate static perception, dynamic tracking, downstream spatial and temporal reasoning, and the contributions of key pipeline components. HOLIWORLD-Foundry outperforms comparison methods on the reported perception and tracking benchmarks, while its generated supervision supports improved reasoning across two benchmarks.

4. Experiments

  • The evaluation examines object-centric 3D annotations for static scenes and moving objects, with controlled ablations of key pipeline components.Detailed experimental settings are provided in Appendix Sections A.1 and A.2.Source: §4 · p. 8

4.1. Static 3D Object Perception

  • HOLIWORLD-FOUNDRY achieves the highest AP25 and AP50 on all three datasets; its AP50 is 0.55 on ScanNet and 0.13 on ScanNet++, versus 0.12 and 0.05 for the strongest baselines.Figure 7 shows more complete object coverage and better-aligned box extents across representative scenes.Source: §4.1 · p. 8, Table 1 · p. 8, Figure 7 · p. 9

4.2. Dynamic 3D Object Perception and Tracking

  • HOLIWORLD-FOUNDRY achieves HOTA scores of 54.31% on ADT and 43.14% on HoliWorld-Bench, versus strongest-baseline scores of 36.00% and 15.13%.Higher AssA and PCL indicate more consistent identities and broader trajectory coverage on both datasets.Source: §4.2 · p. 9, Table 2 · p. 9, Figure 8 · p. 10

4.3. Downstream Spatial and Temporal Understanding

  • Token F1 rises from 16.35% to 78.02% on HoliWorld-Bench and from 18.29% to 66.39% on EgoDynamic4D after fine-tuning Qwen3.5-4B on HoliWorld-Data.The resulting model also surpasses zero-shot Qwen3.8-27B on all four metrics across both benchmarks.Source: §4.3 · p. 9, Table 3 · p. 10

4.4. Ablation Studies

  • Image-based lifting captures more small objects, point-cloud detection recovers larger extents, and stereo fusion combines observations from both views.Caption guidance recovers missed entities; VLM verification modestly changes geometric metrics while correcting some labels and unsuitable candidates.Source: §4.4 · p. 10
  • Cross-interaction label continuity reduces identity fragmentation, while SAM association supplies temporally consistent masks that reduce trajectory deviations seen with per-frame detection.Together, they preserve object correspondence and spatial continuity; corresponding HOTA and PCL gains on both datasets are reported in Table 5.Source: §4.4 · p. 10

5. Conclusion

HOLIWORLD integrates complementary static and dynamic perception to build world records linking scene geometry, object semantics, interaction histories, and supporting observations. These records form learning resources and support evaluation of annotation quality and learned capabilities.

5. Conclusion

  • HOLIWORLD-Foundry constructs world records that connect scene geometry, object semantics, and interaction histories to supporting observations.The records form HOLIWORLD-Data and support HOLIWORLD-VL training; independently annotated HOLIWORLD-Bench evaluates annotation quality and learned capabilities.Source: §5 · pp. 10–11
  • Experiments demonstrate improved static perception and dynamic tracking, alongside gains in spatial and temporal reasoning on HoliWorld-Bench and EgoDynamic4D.The paper presents HOLIWORLD as a practical foundation for expanding grounded learning resources with new recordings.Source: §5 · p. 11

A.1. Static Perception Evaluation Details

The static evaluation compares methods on ScanNet, ScanNet++, and in-house sequences, using official or manually annotated 3D ground truth. HOLIWORLD-FOUNDRY’s AP gains persist at stricter geometric matching, while matched-box mIoU should be interpreted alongside AP.

A.1. Static Perception Evaluation Details

  • Evaluation uses 20 sequences each from ScanNet and ScanNet++, plus 20 self-collected in-house sequences with manually annotated ground truth.ScanNet and ScanNet++ use their official 3D annotations.Source: §A.1 · p. 16
  • The comparison includes model-based, 3DGS-based, and spatial-pipeline baselines, including SpatialLM, M3-Spatial, and Holi-Spatial.The listed spatial pipelines are Boxer and Holi-Spatial.Source: §A.1 · p. 16
  • At IoU 0.50, HOLIWORLD-FOUNDRY reaches AP50 of 0.55 on ScanNet and 0.13 on ScanNet++, compared with strongest-baseline scores of 0.12 and 0.05.These gains persist under stricter geometric matching.Source: §A.1 · p. 16

A.2. Full Dynamic Tracking Results

The full dynamic evaluation compares tracking methods on ADT20 and HoliWorld-Bench under a shared trajectory-scoring protocol. Inputs and binding rules differ for box-output methods, while trajectory evaluation uses common distance-based scoring.

A.2. Full Dynamic Tracking Results

  • HOTA0.1, DetA0.1, and AssA0.1 measure joint detection and association, detection accuracy, and identity-association consistency; PCL0.1 scores correctly localized GT instance-frames under a sequence-global assignment.Scoring uses a 0.1 m world-coordinate center-distance tolerance; TP-L2 is mean center error among successful matches.Source: §A.2 · p. 16
  • The compared inputs include GT first-visible mask initialization for SAM-based baselines and GT category queries for EgoSeg3D.For IT3DEgo, each target’s first-visible GT mask is converted to a bounding box for its native enrollment interface.Source: §A.2 · p. 16, Table 4 · p. 18
  • Trajectory evaluation uses common distance-based scoring across methods, while box-output methods additionally require maximum 3D IoU greater than 0.5 for candidate-track binding.Binding does not filter trajectory evaluation; missing predictions remain failures, and sequence scores are averaged equally.Source: §A.2 · p. 18
  • Table 4 reports full dynamic tracking results on ADT20 and HoliWorld-Bench, with TP-L2 in meters and other metrics in percentages.Ours uses no annotation-derived test-time inputs; box metrics use bound objects.Source: Table 4 · p. 18

A.3. Additional Ablation Results

The ablation results cover component-wise comparisons, while Figure 12 presents qualitative comparisons of the static perception branches across three scenes.

A.3. Additional Ablation Results

  • Table 5 reports component-wise ablations for static AP25/mIoU and dynamic HOTA/PCL at 0.1 m.Checkmarks and crosses indicate enabled and disabled components; shaded rows S6 and D3 enable all components.Source: §A.3 · p. 18, Table 5 · p. 19
  • The supplied passages describe the ablation comparisons but do not state the measured results or identify a winning configuration.The table caption specifies the metrics and component encoding, but provides no cell values.Source: Table 5 · p. 19, Figure 12 · p. 19
  • Figure 12 compares static perception without 3D lifting, without point-cloud detection, and with both branches across three scenes.C1–C3 identify camera viewpoints and corresponding image projections.Source: §A.3 · p. 18, Figure 12 · p. 19

A.4. Holi-World Static and Dynamic QA

Holi-World defines 30 fine-grained QA types spanning static spatial reasoning and dynamic trajectory and caption reasoning, with questions grounded in visual and temporal evidence.

A.4. Holi-World Static and Dynamic QA

  • The QA set contains 30 fine-grained types, divided broadly into static and dynamic reasoning.Figure 13 illustrates examples across static mapping, dynamic trajectory, and dynamic caption reasoning.Source: §A.4 · p. 19, Figure 13 · p. 20
  • The 11 static types evaluate camera motion and object-level spatial relations, including distance, depth, height, and relative position in single- and multi-view settings.Source: §A.4 · p. 19
  • The 19 dynamic types comprise seven trajectory types and 12 caption-based types covering motion, actions, interactions, temporal grounding, event order, and state changes.Questions combine observations, 3D boxes, trajectories, and temporally aligned captions; answers include labels, numbers, choices, time intervals, and free-form descriptions.Source: §A.4 · pp. 19–20

B. Reproducibility and Responsible Use

The paper describes the pipeline and evaluation details intended to support reproducibility, and specifies responsible-use requirements and boundaries.

B. Reproducibility and Responsible Use

Loading 2610.26549v1…