Source-linked AI summary
DeepInsight II: One Trace from Benchmark to Robot
Siyi Li, Yuchen Kang, Wuliang Wang, Zhengjie Zhang, Jiangpin Liu, Jianhao Yao, Jie Chen
TL;DR
Physical AI evaluation remains fragmented and least standardized in the embodied layers that determine deployment, while DeepInsight v1’s quantitative evidence centered on foundation models. DeepInsight II preserves v1’s task–resource–result substrate, evaluates native embodied benchmarks, standardizes whole-body control across simulation and matched robot trials, and extends trace-based diagnosis to repair-oriented handoffs. The result is empirical continuity from benchmark execution to matched robot evidence and physical repair testing.
Problem
Embodied navigation, manipulation, and whole-body control remain evaluated through fragmented benchmark-specific simulators, embodiments, interfaces, and per-lab harnesses, while physical execution was outside v1’s empirical scope.
Method
The paper preserves the task–resource–result substrate, runs native System 1 benchmarks, compares four System 0 controllers under MotionBench, matches simulation with real-robot traces, and diagnoses handoffs using evidence-grounded labels and repairability.
Results
Across 8 physical episodes, automated labels matched reference task outcomes in every episode and handoff attribution in 2 of 3 failed-or-partial episodes.
Takeaways & Limitations
Matched parent-trace identities make simulation–hardware correspondence and repair-oriented diagnosis part of one evaluation runtime rather than separate toolchain reconciliations.
Takeaways & Limitations
StarVLA-OFT is excluded because its released Bridge checkpoint was confirmed incorrect and no corrected artifact was available.
Abstract
from arXiv · showhide
Across a Physical AI stack, evaluation maturity is inversely aligned with deployment risk: foundation models enjoy mature, standardized harnesses, while the embodied layers on which deployment actually turns remain fragmented across benchmark-specific simulators, embodiments, and interfaces. The first DeepInsight report (v1) unified evaluation across this stack behind three abstractions---task, resource, and result---but its quantitative evidence centered on the foundation-model layer; navigation and manipulation (System 1) and whole-body control (System 0) remained simulation case studies, and physical execution was outside its empirical scope. DeepInsight II keeps that substrate fixed and quantifies the embodied half. First, it reproduces released-checkpoint references across two navigation and four manipulation benchmarks under their native protocols. Second, MotionBench places four released whole-body controllers under one workload and metric contract, then carries a qualified within-family cohort from parallel simulation to matched real-robot trials in which simulated and physical rollouts share a parent trace identity while retaining execution-domain-specific records, making the sim-to-real gap a native reduction rather than a reconciliation across toolchains. Third, a composed System 2--1--0 study extends trace localization into five evidence-grounded handoff labels, each mapped to a concrete repair action, with a measured repairability criterion and physical episodes testing the same attribution under hardware-observable state. The contribution is therefore not a new evaluation architecture, but empirical continuity from benchmark execution to matched robot evidence and repair-oriented diagnosis.
1 Introduction
DeepInsight II addresses the structural imbalance in Physical AI evaluation by extending a shared substrate from mature foundation-model harnesses to embodied execution. It preserves native benchmark protocols while adding matched simulation–robot evidence and repair-oriented diagnosis.
- Motivation: Evaluation infrastructure is least standardized at the navigation, manipulation, and whole-body-control layers where deployment failures become physical.Embodied episodes are closed-loop stateful rollouts, creating heterogeneity in task semantics, simulators, embodiments, and policy interfaces.
- Scope: DeepInsight II keeps the existing substrate fixed and closes its empirical gap by quantifying embodied layers and physical execution.The scope expands beyond v1’s foundation-model-centered evidence without replacing its architecture.
- System 1: Six System 1 benchmarks retain native protocols while sharing execution, resource management, trace identity, and cross-domain reduction.Released checkpoints are evaluated across two navigation and four manipulation benchmark settings.
- System 0: Four released System 0 whole-body controllers are compared under one contract, with a qualified cohort carried from simulation into matched real-robot trials.The study standardizes the workload and evaluation conditions without introducing new control policies.
2 Related Work
DeepInsight II treats fragmented embodied benchmarks as workloads to carry rather than alternatives to replace. It preserves their protocols while adding shared coordination, runtime, trace identity, and matched simulation–hardware evaluation.
- Embodied benchmarks: Navigation and manipulation workloads span R2R, VLN-CE, RxR, LIBERO, LIBERO-Plus, and LIBERO-PRO across established benchmark ecosystems.The paper carries these workloads within the Habitat and manipulation simulator families rather than redefining their benchmark semantics.
- Shared substrate: DeepInsight preserves each benchmark’s suites, splits, embodiment, and success criteria while unifying the runtime, resource layer, and trace beneath them.Benchmark fragmentation becomes the workload specification for the shared substrate.
- Infrastructure: The shared runtime adds one episode driver, one result identity across benchmarks, and a real robot behind the same resource protocol.This extends coordination beyond containerized model-server federation.
- Sim-to-real evaluation: Prior sim-to-real work compares paired rollouts, model orderings, and matched errors, whereas DeepInsight II applies that measurement line to whole-body motion-tracking controllers.Matched simulation and hardware branches share one trace identity and evaluation contract, so correspondence is produced by the runtime.
3 Same Abstractions, New Backends
DeepInsight II brings embodied simulators, policies, and a physical robot onto the unchanged task–resource–result substrate. Matched simulation and real-robot rollouts share parent trace identity while retaining domain-specific evidence.
- Inherited invariants: The embodied extension changes backends rather than architecture, preserving the reset/step interface, acquire/release control plane, and append-only hierarchical trace.These invariants remain independent of what backend sits behind a handle.
- Simulator backends: Simulators run as sandboxed container backends, isolating conflicting dependencies while retaining shared lease, parallelism, and accounting controls.Habitat, MuJoCo/robosuite, and SAPIEN/ManiSkill are attached behind the sandbox handle.
- Core abstractions: Embodied evaluation uses the same three abstractions as v1: task, resource, and result.A simulator, policy, and real robot attach as resources behind one handle, while each rollout writes one identity-addressed trace.
- Policy backends: Policy servers expose observation-in/action-out interfaces and validate benchmark bindings at configuration time.Interface mismatches fail during binding instead of during rollout execution.
- Robot backend: The physical robot is leased per episode and writes events into the shared trace, while serial capacity, human-gated resets, interventions, and emergency stops remain backend-specific.Physical asymmetries are absorbed as backend behavior rather than treated as retryable infrastructure errors.
- Cross-domain matching: Matched simulation and physical episodes appear as sibling branches under one parent trace identity, with calibrated initial states and execution-domain-specific records.They share task and metric contracts, enabling cross-domain reduction as a join on the parent identity.
4 System 1: Benchmark Fidelity and Common-Protocol Evaluation
DeepInsight II evaluates released System 1 checkpoints through a shared runtime while preserving native benchmark protocols. Across navigation and manipulation, unified execution largely reproduces published behavior, with localized discrepancies tied to specific references or implementation differences.
- Coverage: Six benchmarks—two navigation and four manipulation—are evaluated through one substrate using released checkpoints and trace-derived execution diagnostics.The evaluation retains benchmark-specific task splits, interfaces, success criteria, and aggregation rules while sharing runtime and resource bindings.
- Navigation: R2R-CE and RxR-CE use identical episode lists, egocentric monocular inputs, and 500-step horizons under the shared episode driver and trace schema.Privileged simulator state remains confined to environment and metric implementation rather than policy input.
- Navigation: Published navigation references remain close to unified execution: R2R-CE differences reach at most 3.81 points and 0.18 meters, while RxR-CE bounded-metric differences reach at most 1.27 points.The reported R2R-CE maxima vary by checkpoint, while RxR-CE StreamVLN matches reported SR exactly and NE differences reach at most 0.63 meters.
- Interpretation: Residual navigation gaps reflect unmatched source aggregates, simulator stochasticity, software revisions, and adapter-level choices that can compound during closed-loop execution.The paper identifies preprocessing, action decoding, collision handling, and stopping implementation as possible sources while holding episode lists, observations, and evaluation budgets fixed.
- Manipulation: Manipulation coverage spans clean-suite, perturbation, cross-embodiment, and mobile-compositional settings across LIBERO, LIBERO-Plus, SimplerEnv, and RoboCasa365.The evaluation scales from 2,000 LIBERO episodes per model to 10,030 LIBERO-Plus instances, 384 SimplerEnv episodes, and 2,500 RoboCasa365 episodes.
- Manipulation: Across standard LIBERO, SimplerEnv, and RoboCasa365, unified aggregates track published references without a systematic shift across model families.Most LIBERO-Plus pairs follow the same pattern, except π0 under Sensor Noise, where DI obtains 18.7 versus a published 79.0; this unresolved reference is excluded from ranking claims.
5 System 0: Scalable WBC Evaluation from Simulation to Real-Robot Evidence
MotionBench standardizes whole-body-controller evaluation by fixing the workload and metric contract, then extends qualified simulation evidence to matched real-robot trials. Four released models are compared in simulation, while a qualified SONIC cohort preserves trace-linked correspondence across domains.
- Standardized evaluation: MotionBench fixes the workload, failure semantics, trace schema, and reducers while allowing only the WBC model binding to change.Its contract separates task, WBC binding, trace, and metric components.
- Simulation comparison: Four released WBC models are evaluated under the same MotionBench snapshot, with SR and MPJPE-l measuring executability and local fidelity.The common contract supports heterogeneous architectures and runtime interfaces.
- Simulation comparison: HoloMotion 1.3.2 leads the full benchmark with 96.7% SR and 4.53 cm MPJPE-l, also achieving 93.1% progress-soft.These are full-benchmark aggregates under the shared simulation protocol.
- Capability analysis: Capability decomposition shows greater separation in locomotion, dynamic, and low-posture motion than in nearly saturated upper-body progress-soft.SONIC variants reach 90.8% and 89.2% hard-failure SR, but 72.7% and 67.4% progress-soft, respectively.
- Matched physical evidence: A qualified SONIC cohort uses 25 shared motion clips, 100% simulation SR, trace-derived checks, and three hardware trials per pair.Qualification checks include trajectory completion, tracking, posture, and actuator/contact behavior.
- Matched physical evidence: Matched simulation and hardware records share a parent trace identity and metric definitions while retaining domain-specific episode records.Hardware MPJPE-l uses synchronized state estimates and the same root-translation subtraction as simulation.
- Simulation–real correspondence: SONIC release remains lower than SONIC low-latency across all seven readings in both domains, while hardware raises tracking errors and tail peaks.The reported gaps include larger low-latency differences for MPJPE-l, joint velocity, acceleration-P95, and jerk-P95.
6 Full-System: Handoff Attribution and Repairability in Physical Execution
The full-system study turns trace-localized handoff failures into five evidence-grounded labels mapped to repair actions, then tests the attribution in simulation and on a physical robot. A measured feasible-region-to-step ratio distinguishes interface-repairable boundary errors from action-space limitations.
- Attribution framework: DeepInsight II adds automatic handoff attribution that identifies the broken cross-module assumption or coordination design and maps it to a repair action.Each decision must be supported by a concrete trace interval.
- Attribution framework: The five labels distinguish boundary-definition problems from executability problems and downstream skill-internal failures.The labels are defined by verified evidence patterns and associated repair actions.
- Representative handoffs: H2 is a boundary-definition failure because navigation terminates on position while greeting requires facing the user; 8 of 10 isolated attempts failed before orientation was added.The repair is to add the missing interface convention.
- Representative handoffs: H5 is an executability failure: registered conditions pass, but the feasible entry region is too narrow for reliable correction under the available action step.No interface convention repairs a region narrower than one step; the action space must change.
- Simulation evaluation: In 20 simulated composed episodes, failure mass and hidden failures with both adjacent modules reporting success concentrate on H2 and H5.A negative control assigns a mid-execution safety-axis failure to skill-internal execution rather than a boundary.
- Physical execution: Across 8 physical episodes, automated labels match task outcome in every episode and handoff attribution in 2 of 3 failed-or-partial episodes.Every physical failure cause maps to a class identified in simulation.
- Physical execution: Hardware-observable evidence resolves an H2 failure despite both modules self-reporting success, while odometry drift reduces the simulated H5 slack margin.The cited H2 interval is [t=39.3 s, t=49.1 s], and bounded drift is 0.10–0.20 m against 0.20 m simulated slack.
7 Conclusion
DeepInsight II extends the shared evaluation substrate to embodied layers and physical execution without adding a fourth abstraction. Matched traces connect standardized whole-body-control comparison with repair-oriented diagnosis across simulation and robot evidence.
- Scope: The paper extends DeepInsight from foundation-model-centered simulation to navigation, manipulation, whole-body control, and physical execution.The same three abstractions absorb embodied heterogeneity.
- Trace continuity: Matched simulated and physical rollouts share a parent trace identity while retaining execution-domain-specific evidence.This supports native simulation–real reduction rather than reconciliation across toolchains.
- System 0 evidence: The two SONIC variants preserve their numerical ordering across seven matched readings, while hardware raises tracking error and shifts dynamics toward larger upper-tail peaks.The conclusion ties standardized WBC comparison to matched physical evidence.
- Repair-oriented diagnosis: Five-way attribution maps full-system failures to concrete repair actions, while the feasible-region-to-step ratio separates interface-repairable errors from action-space limitations.The criterion is measured in the simulated case study.