Source-linked AI summary
ObjectTwin: Measuring the Asset Correspondence Gap for Capture-Anchored Digital Twins
Zhenghe Guo
TL;DR
Generated meshes used as proxies for captured objects must reproduce calibrated observations of the same instance, which isolated-asset scores and pose estimates do not jointly establish. ObjectTwin measures this through fixed-state replacement against scan-derived visible geometry; across tested domains and backends, generated assets lose 0.302–0.497 instance-macro MaskIoU. The results place capture correspondence as a distinct interface before physical or task-level evaluation.
Problem
Isolated-asset scores and pose estimates do not jointly establish whether a generated proxy reproduces calibrated observations of the particular captured instance it replaces.
Source: §1 · pp. 1–2Method
ObjectTwin freezes the instance, cameras, scan-derived visible support and depth, metric anchor, and denominator, then compares scan and generated geometry at that same state.
Source: §1 · p. 2, §3.1 · p. 3Results
Takeaways & Limitations
Capture correspondence is a separately measurable interface between generated-asset quality and stronger physical or task-level evaluation.
Source: §6 · p. 8, §5 · p. 8The study covers selected, author-adjudicated indoor instances and two backends; its uncertainty intervals do not establish population prevalence or a universal model ranking.
Source: §5 · p. 8
Abstract
from arXiv · showhide
Object-wise reconstruction increasingly inserts generated meshes into captured scenes as editable proxies for observed objects. This use creates a requirement absent from isolated-asset scores and pose estimates: at the captured metric state, the proxy should reproduce calibrated observations of the same instance. We present ObjectTwin, a scan-grounded measurement protocol that freezes the instance, cameras, scan-projected support and recorded depth, metric anchor, and evaluation denominator, then replaces only the object representation. The resulting asset correspondence gap measures lost observation agreement relative to scan-derived visible geometry without treating the scan as complete physical ground truth. Across two indoor RGB-D capture domains and two generation backends, visible, anchored generated assets lose 0.302-0.497 MaskIoU under this replacement. Every tested scene-level estimate is positive, while same-input generator preference reverses across capture domains. Coverage and overlap-depth analyses locate disagreement in both visible support and surface placement; the evaluated conservative pose refinement does not systematically recover it. These results establish capture correspondence as a separately measurable interface between generated-asset quality and stronger physical or task-level evaluation.
1. Introduction
Object-wise digital twins need generated proxies to reproduce the calibrated observations of the particular captured instance, a requirement not answered by isolated-asset scores or pose estimates. ObjectTwin measures this correspondence through a fixed-state replacement contrast and finds a positive gap across tested domains and generators.
1. Introduction
- The estimand concerns agreement with visible capture evidence, not unseen geometry, watertightness, contact parameters, or robot task success.Those stronger physical and task-level evaluations require other references.Source: §1 · p. 2
- Existing asset, orientation, and pose interfaces do not jointly measure observation loss from replacing scan geometry with a crop-generated proxy at the captured state.A canonically upright, well-placed chair can still have incorrect parts or visible depth.Source: §1 · p. 2
- The asset correspondence gap is the observation loss when generated geometry replaces scan-derived geometry at the same captured state.The protocol fixes views, scan-projected support and depth, metric anchor, and denominator; the scan is a reference, not complete physical ground truth.Source: §1 · p. 2
- Every tested scene-level estimate is positive, but same-input generator preference reverses between Replica and ScanNet.Coverage and depth analyses find disagreement in both missing visible support and overlapping surfaces; conservative pose refinement does not systematically recover the loss.Source: §1 · p. 2
- ObjectTwin contributes a reproducible protocol and cross-domain, multi-scene, cross-generator evidence for a persistent visible-surface gap.Its protocol separates representation, output availability, and pose adaptation while preserving prediction-independent views and instance-level statistical units.Source: §1 · p. 2
2. Related Work
Prior work evaluates generated-asset quality, canonical orientation, model correspondence, and broader scene or task outcomes through distinct interfaces. ObjectTwin isolates the object-level test of whether a generated proxy reproduces the observations of its claimed captured instance.
2. Related Work
- Asset benchmarks, canonical-orientation methods, and CAD-to-scan or pose methods address distinct criteria, but do not jointly measure instance observation loss under fixed-state replacement.Canonical orientation or stable placement can leave instance proportions, parts, support, and visible depth inconsistent with capture.Source: §2 · p. 2
- ObjectTwin fixes instance, state, cameras, and scoring views to identify representation-replacement loss that flexible scene-editing loops otherwise leave unresolved.Agent-selected edits and views cannot define the measurement, though an agent can use its diagnosis.Source: §2 · pp. 2–3
- ObjectTwin begins after persistent instance identification, making the front end replaceable because the measured intervention starts with fixed instances and source observations.Mask3D and OpenIns3D are examples of persistent 3D proposal and recognition systems.Source: §2 · p. 3
- ObjectTwin defines the controlled link between generated outputs and captured-object proxies, without replacing asset QA, canonical orientation, CAD retrieval, pose estimation, or task validation.An asset can be plausible and stably posed yet fail to reproduce observations tied to its claimed identity.Source: §2 · p. 3
3. ObjectTwin Measurement Protocol
ObjectTwin measures visible observation loss by comparing scan-derived and generated geometry while holding the captured instance and evaluation state fixed. Its auditable instrument freezes inputs and views, reports complementary mask and depth measures, and treats pose adaptation as a separate control.
3.1. Fixed-state replacement estimand
- The fixed-state estimand contrasts scan and generated observation scores while sharing the instance, cameras, reference support and depth, metric anchor, and view denominator.This identifies observation-space replacement loss relative to captured scan evidence; unobserved physical surfaces leave complete mesh error unidentified.Source: §3.1 · p. 3
- The scan endpoint uses projected instance-labeled scan geometry and recorded RGB-D depth to define visible support, serving as a protocol reference rather than complete physical ground truth.The masks are scan-derived rather than independent manual annotations, and rasterization or incompleteness can keep scan scores below one.Source: §3.1 · p. 3
3.2. Auditable object-wise instrument
- Persistent identities, frozen prediction-independent views, and distinct endpoint records prevent object identity, scoring views, or later pose updates from changing the primary contrast.The instrument records crop provenance, filtering, completion input, generated asset, anchor, and rendered endpoints.Source: §3.2 · p. 3
- The primary anchor uses the scan-derived centroid, uniform scale matching scan extent, and upright yaw initialized from scan PCA, without ICP before measurement.This favorable initialization reduces scale and placement ambiguity but is not independent 6D pose ground truth; Tripo and Hunyuan receive the same completed input in paired comparisons.Source: §3.2 · p. 3
- The evaluation contracts distinguish instance-linked scores and fixed state, then identify what each contract leaves unresolved for a generated proxy.The table caption describes these dimensions but does not report a winning contract.Source: Table 1 · p. 4
- Figure 2 shows scan and generated geometry rendered from the same metric anchor and frozen cameras before a conservative verifier may change pose.Its PASS/FAIL belongs only to the pose-evidence gate; correspondence itself has no acceptance threshold.Source: Figure 2 · p. 4
3.3. Observation measures and pose controls
- Mask IoU is averaged across views within each instance and then across instances; visible coverage aggregates pixels within an instance before the instance-macro average.Coverage guards against deceptively small depth error from nearly empty renders, while overlap-depth L1 tests surface placement where silhouettes overlap.Source: §3.3 · pp. 4–5
- Generated-at-anchor is the primary endpoint; HYBRID, CAST-orient, and Any6D are auxiliary pose-compatibility controls with distinct assumptions, not complete native-system evaluations.HYBRID uses scan-geometry pose adoption and bounded yaw and planar-footprint refinement; translation and scale stay fixed during global-yaw search, with an evidence gate controlling updates.Source: §3.3 · p. 5
4. Experiments
Across two indoor RGB-D domains and two generators, fixed-state replacement reveals a persistent loss of agreement with visible scan evidence. The experiments test its robustness, sources, and whether evaluated pose refinements recover it.
4.1. Questions, data, and statistical protocol
- The protocol uses 74 Replica and 63 ScanNet instances, averages views within each instance, and bootstraps scenes before instances.Frozen views are selected before endpoint predictions are inspected; missing or out-of-view renders remain in the denominator, and paired generator tests require common inputs and visible outputs from both backends.Source: §4.1 · p. 5
4.2. Replacement loss and scene robustness
- 0.302–0.497 MaskIoU is lost under representation replacement, with every hierarchical 95% CI excluding zero.The comparison conditions on visible generated assets at the scan anchor and fixes the instance, cameras, reference evidence, metric anchor, and denominator.Source: §4.2 · p. 5, Table 2 · p. 6
- All 20 scene–generator estimates are positive; leave-one-scene-out estimates remain positive across the four domain–generator cells.Replica gaps range from 0.209 to 0.418 and ScanNet gaps from 0.179 to 0.553; the conditional comparison excludes unavailable outputs, while Replica–Hunyuan’s complete-inventory gap rises from 0.302 to 0.534.Source: §4.2 · p. 5
4.3. Generator-specific explanation
- Hunyuan exceeds Tripo by 0.110 MaskIoU on Replica, but trails by 0.186 on ScanNet, so their capture-conditioned ordering reverses.Same-input paired assets show instance-dependent outcomes, including opposite rankings and cases where both proxies omit most captured support; the examples illustrate patterns, not their frequency.Source: §4.3 · pp. 5–7, Figure 3 · p. 6
4.4. Where observation agreement is lost
- Generated assets lose 0.254–0.396 visible coverage, while generated depth L1 increases by 0.122–0.283 m on overlapping masks.Every interval excludes zero; the diagnostics distinguish missing projected support from surface disagreement where support is shared.Source: §4.4 · p. 7
4.5. Can pose refinement substitute for the test?
- HYBRID changes mean MaskIoU by −0.057 to −0.002, and CAST-orient has a negative mean change in every cell; neither recovers the replacement loss.No evaluated HYBRID interval recovers even the lower end of the 0.302–0.497 gap; Any6D changes Replica–Tripo MaskIoU by −0.292, but is excluded from the cross-domain adapter conclusion because equivalent endpoints are unavailable in other cells.Source: §4.5 · pp. 7–8, Table 3 · p. 8
5. Discussion and Limitations
ObjectTwin interprets MaskIoU through a controlled replacement contrast that fixes the instance, cameras, metric placement, and scoring views. The resulting claim is limited to visible capture correspondence; it does not establish physical identity or task-level performance.
5. Discussion and Limitations
- Fixing the instance, state, cameras, and scoring views makes the MaskIoU contrast identify observation agreement lost when generated geometry replaces scan-derived geometry.Coverage and depth provide complementary diagnostics, while hierarchical and paired analyses test scene and backend explanations.Source: §5 · p. 8
- The scan-rendering reference measures compatibility with visible scan evidence, not complete physical identity or agreement on unseen surfaces.Physical identity would additionally require unseen-surface scans, material and contact properties, articulation, and task-specific tolerances.Source: §5 · p. 8
- Alignment estimates where a chosen model belongs, whereas correspondence tests whether that model explains the target instance once placed; the evaluated updates do not recover the gap.A stronger correspondence-aware deformation or reconstruction method may improve future endpoints, which ObjectTwin can test.Source: §5 · p. 8
- The evaluation covers selected, author-adjudicated indoor instances, and its intervals quantify uncertainty within the tested scene–instance hierarchy rather than population prevalence.The loss is conditioned on visible surfaces, recorded cameras, scan-derived anchors, and generated assets reaching the rendering endpoint.Source: §5 · p. 8
- Visible-surface correspondence and stable capture anchoring are prerequisites for later physical and task contracts, not evidence that collision, contact, dynamics, or robot success are satisfied.Those later contracts require assets, physical parameters, and task protocols absent from this evaluation.Source: §5 · p. 8
6. Conclusion
ObjectTwin measures capture correspondence when generated meshes replace observed objects. Across tested domains and backends, generated proxies lose visible observation agreement, while the tested pose controls do not recover the gap.
6. Conclusion
- 0.302–0.497 instance-macro MaskIoU is lost by generated proxies across the tested domains and backends when instance, cameras, metric anchor, and denominator are fixed.The loss survives every scene and leave-one-scene-out check; same-input generator preference reverses, and both coverage and visible depth degrade.Source: §6 · p. 8
- The evaluated pose controls do not recover the measured gap, and same-input generator preference reverses across capture domains.The reported disagreement includes both visible support and visible depth.Source: §6 · p. 8
- Capture correspondence is presented as a separately measurable interface between generated-asset quality and stronger physical or task-level evaluation.Source: §6 · p. 8
A. Frozen-view protocol and data
The protocol freezes calibrated views and scan-derived reference evidence before evaluating generated endpoints. It compares scan-derived geometry with generated assets at the same metric anchor, retaining missing views in the denominator.
A. Frozen-view protocol and data
- The benchmark uses Replica and ScanNet, selecting up to five views per instance before inspecting generated endpoints and treating the instance as the statistical unit after view averaging.Replica has five views for each of 74 instances; ScanNet has 298 views across 63 instances.Source: A. Frozen-view protocol and data · p. 10
- Every endpoint shares calibrated cameras, scan-derived reference support, the recorded-depth occlusion rule, metric anchor, and frozen-view denominator; missing or out-of-view renders receive zero MaskIoU and coverage.The geometric comparison conditions on generated proxies reaching anchored rendering and being visible in at least one frozen view; output availability is summarized separately.Source: A. Frozen-view protocol and data · p. 10
- 0.358 (Replica–Tripo), 0.302 (Replica–Hunyuan), 0.310 (ScanNet–Tripo), and 0.497 (ScanNet–Hunyuan) are the scan-reference minus generated@anchor losses.Replica–Hunyuan contributes 30 visible, anchored proxies; the other cells contribute 74, 63, and 63.Source: A. Frozen-view protocol and data · p. 10
B. Uncertainty, scene robustness, and pairing
Uncertainty is estimated by resampling scenes and then instances while keeping each instance’s frozen views grouped. Leave-one-scene-out estimates remain positive, and paired generator preference changes direction between domains.
B. Uncertainty, scene robustness, and pairing
- The 4,000-replicate hierarchical bootstrap gives 95% intervals of [0.294, 0.433], [0.221, 0.380], [0.222, 0.388], and [0.426, 0.558] for Replica–Tripo, Replica–Hunyuan, ScanNet–Tripo, and ScanNet–Hunyuan.Each replicate samples scenes with replacement, then instances within sampled scenes, keeping frozen views grouped.Source: B. Uncertainty, scene robustness, and… · p. 10
- Leave-one-scene-out loss ranges are 0.342–0.385, 0.293–0.317, 0.294–0.351, and 0.483–0.519 in Replica–Tripo, Replica–Hunyuan, ScanNet–Tripo, and ScanNet–Hunyuan order.Source: B. Uncertainty, scene robustness, and… · p. 10
- Hunyuan minus Tripo is +0.110 [0.036, 0.188] on Replica and −0.186 [−0.250, −0.105] on ScanNet in the paired comparison.The corresponding Hunyuan/Tripo win counts are 25/5 on Replica and 14/49 on ScanNet.Source: B. Uncertainty, scene robustness, and… · p. 10
C. Endpoint and adapter configuration
The evaluation compares fixed-state scan and generated endpoints, with a bounded post-anchor refinement and auxiliary pose controls. Rendering and aggregation are fixed across paired contrasts, while incomplete endpoint provenance and coverage constrain interpretation.
C. Endpoint and adapter configuration
- The evaluated endpoints include scan geometry, Tripo or Hunyuan meshes at scan-derived anchors, HYBRID refinement, CAST-orient, and an Any6D compatibility control limited to a fully documented cell.Source: C. Endpoint and adapter configuration · p. 10
- The primary anchor transfers scan location and uniform extent scale without ICP, then supplies upright PCA yaw; HYBRID subsequently applies gated real-view pose and planar-footprint refinement.Source: C. Endpoint and adapter configuration · p. 10
- Tripo and Hunyuan receive the same completed image, but results describe frozen backend outputs rather than variability across repeated service calls.Source: C. Endpoint and adapter configuration · pp. 10–11
- Table 4 reports scene-level Tripo and Hunyuan scan-reference-minus-generated@anchor MaskIoU gaps and conditioned counts; Table 5 summarizes endpoint configurations and observation evidence.Source: Table 4 · p. 11, Table 5 · p. 11
- Rendering uses recorded capture dimensions, calibration, trajectories, and depth scale; MaskIoU is averaged across frozen views, while coverage and overlap-depth errors are aggregated within instances.Source: C. Endpoint and adapter configuration · p. 11
- −0.292 MaskIoU: Any6D changes Replica–Tripo relative to generated@anchor; other domain–generator cells are unreported, and this auxiliary result is excluded from the factorial claim.Source: C. Endpoint and adapter configuration · pp. 11–12
D. Evidence boundary
The evidence boundary is visible-surface agreement under recorded cameras and scan-derived anchors, not complete physical or task-level validation. Experimental measurements come from frozen-view renders and instance-level records, not the explanatory schematics.
D. Evidence boundary
- The scan reference is not independent annotation or complete physical ground truth; the measured loss is limited to visible surfaces under calibrated views and scan-derived anchors.Source: D. Evidence boundary · p. 12
- The evaluation does not identify hidden geometry, independent 6D pose ground truth, contact stability, articulation, dynamics, synchronization, or robot task success.Source: D. Evidence boundary · p. 12
- Figures 1–2 are explanatory schematics, while numerical claims derive from frozen-view renders and instance-level records summarized in Figures 3–4 and accompanying tables.Source: D. Evidence boundary · p. 12