FACT: Fidelity-Aware Construction of Articulated Twins

1 INTRODUCTION

• Visual plausibility alone is insufficient because assets determine what a robot observes, where it can establish contact, which motions are feasible, and how objects move.The paper distinguishes geometry, contact, and dynamic fidelity as complementary requirements.Source: §1 · p. 2
• FACT links geometric discrepancies to editable features, free-space and motion violations to collision partitions, and trajectory residuals to response models and physical parameters.The agent selects measurements and edits using semantic understanding and quantitative feedback; numerical tools execute and validate updates.Source: §1 · p. 2
• FACT reconstructs editable articulated geometry from images through feature planning, targeted measurements, and diagnostic refinement.The agent plans parametric features and selects measurements to constrain their dimensions and placement.Source: §1 · p. 2
• FACT repairs task-specific contact failures through local collision-partition revisions, then compresses proxies under fidelity constraints.The collision-refinement stage diagnoses task-specific failures before simplifying proxies.Source: §1 · p. 2
• FACT constructs and revises response models from interaction observations and directs constrained system identification.Its dynamic-calibration contribution uses residuals to guide model revision and parameter fitting.Source: §1 · p. 2

Articulated object reconstruction and generation

• Articulated reconstruction recovers part geometry and kinematics from observations, while generative and image-conditioned methods produce articulated assets from visual cues (Jiang et al., 2022; Liu et al., 2023; Weng et al., 2024; Liu et al., 2025b; Ai et al., 2026; Qiu et al., 2025; Wang et al., 2026a; Li et al., 2026; Lei et al., 2023; Liu et al., 2024; Gao et al., 2025; Su et al., 2025; Chen et al., 2024; Liu et al., 2025a).Code-based and agentic pipelines express articulated assets as executable models (Le et al., 2025; Zhou et al., 2026).Source: §2 · pp. 2–3

Programmatic mesh reconstruction and agentic 3D modeling

• Programmatic 3D modeling spans structured primitives, shape programs, shape-to-code reconstruction, and agentic pipelines that combine planning, execution, validation, and feedback (Ye et al., 2025; Fedele et al., 2025; Tavernini et al., 2026; Jones et al., 2020; 2021; 2023; 2026; Dai et al., 2026; Rukhovich et al., 2025; Kolodiazhnyi et al., 2025; Zhang et al., 2026; Zhou et al., 2026; Lin et al., 2026; Kabisov et al., 2026; Hu et al., 2026).In FACT, feature plans guide measurements, and local diagnostics determine whether to revise a feature representation or its parameters.Source: §2 · p. 3

Collision geometry for interactive simulation

• Collision-proxy methods use approximate convex decomposition, with some approaches adapting decompositions to navigable space or animation poses (Wei et al., 2022; Fokin & Savva, 2026; Yang et al., 2026a).FACT uses CoACD with task and motion diagnostics to guide local repartitioning and fidelity-constrained compression.Source: §2 · p. 3

Physical digital twins and system identification

• Physical-twin research estimates parameters from visual simulation, observed interactions, or combinations of visual priors, torques, and poses (Jatavallabhula et al., 2021; Li et al., 2023; Zhong et al., 2024; Jiang et al., 2025; Pfaff et al., 2025; Chen et al., 2025; Lou et al., 2026; He et al., 2026).FACT’s agent constructs and revises response models from observed motion and directs constrained numerical fitting.Source: §2 · p. 3

3.1 OVERVIEW

• FACT represents a twin as A = (G, K, C, θ) and applies an evidence–diagnosis–revision loop, with the agent selecting revisions and numerical tools returning quantitative feedback.Geometry and articulation establish (G, K); contact refinement repairs C on fixed (G, K), and dynamic calibration fits θ on fixed (G, K, C).Source: §3.1 · p. 3, Figure 2 · p. 4

3.2 GEOMETRY FIDELITY: FEATURE PLANNING, MEASUREMENT, AND REFINEMENT

• Feature plans specify part types, relations, parameters to measure, and image regions, while feature-conditioned measurements constrain dimensions, placement, and editable profile controls.The executable program P(ϕ) generates meshes from editable features rather than copied source triangles.Source: §3.2 · pp. 3–4
• Geometry refinement compares reconstructed meshes with the image-generated mesh using surface distances, matched-view silhouettes, renders, and targeted views to localize errors and trigger replanning, remeasurement, or program correction.Acceptance requires automated geometric validation and agent visual inspection.Source: §3.2 · p. 4
• After validation, functional structures are added or refined, parts moving together are grouped into rigid bodies, and sampled poses reveal interferences that can prompt geometry or joint-placement corrections.The resulting geometry and kinematics form the articulated reference for contact refinement.Source: §3.2 · pp. 4–5

3.3 CONTACT FIDELITY: TASK-AWARE COLLISION REFINEMENT

• Task-aware contact diagnostics assess volume errors, task-region occupancy, false collisions, and sampled motion across the articulated range, identifying responsible hull pairs and source regions.The default decomposition initializes the proxy and sets its total hull budget; reference geometry and articulation remain fixed.Source: §3.3 · p. 5
• Local repartitioning, splitting, parameter adjustment, or merging repairs diagnosed proxy failures before compression seeks fewer hulls under motion-validity and fidelity constraints.A repaired proxy becomes the quality reference; compression accepts edits only when they improve the lexicographic objective while bounding error increases.Source: §3.3 · p. 5

3.4 DYNAMIC FIDELITY: VIDEO-BASED CALIBRATION

• With geometry, joints, and proxies fixed, the agent estimates joint trajectories and initial states from videos, constructs a mechanism-specific response model, and selects bounded parameters or initial-state components for fitting.Physical parameters can include damping, friction, and state-dependent restoring or holding terms; parameters are shared across sequences of the same joint and configuration.Source: §3.4 · pp. 5–6
• Residual patterns guide revisions to the response-model form, optimization variables, and bounds; forward simulation evaluates the revised fit before the model and parameters are fixed for held-out clips.Speed-dependent discrepancies prompt examination of damping, while incorrect stopping or holding behavior prompts examination of resistance terms.Source: §3.4 · p. 6

4.1 GEOMETRY FIDELITY

• With oracle inputs, FACT leads all six geometry metrics and reduces CD by 83.8% relative to Codex; both image-based variants also outperform Codex and Articraft.Figure 3 adds matched-view comparisons showing gains in local structures and dimensions, beyond overall shape agreement.Source: §4.1 · p. 7, Figure 3 · p. 7
• Adding feature planning, targeted measurement, and diagnostic refinement progressively reduces CD and SD-P95; refinement further reduces SD-P95 by 23.3%.The cumulative ablation adds workflow stages sequentially to Codex on oracle meshes, ending with full FACT.Source: §4.1 · p. 7, Table 2 · p. 7

4.2 CONTACT FIDELITY

• FACT achieves 99.47% ISR versus 42.28% for DiagSearch, with 29.7% fewer hulls and 36.5% lower physics-step time.It also improves IoU and TFSO, with no self-collisions in the articulated motion space.Source: Table 3 · p. 8, §4.2 · p. 8
• Increasing hull count generally improves volumetric fit but does not ensure reliable interaction: FACT reaches 99.47% ISR with 229.2 hulls on average, versus 58.33% ISR with 2,081.5 hulls for the finest Component configuration.The sweep compares Body, Part, and Component CoACD thresholds with DiagSearch hull budgets; failed settings are omitted.Source: §4.2 · p. 8, Figure 4 · p. 8

4.3 DYNAMIC FIDELITY

• FACT achieves the lowest error across all three held-out motion metrics, reducing TMSE by 85.3% and OTR@10% by 59.2% relative to Gemini-3.8-Flash.Figure 6 shows FACT capturing oven and fridge door motion reversals that several baselines miss.Source: Table 4 · p. 9, Figure 6 · p. 9, §4.3 · p. 10

• FACT improves geometric reconstruction, enables more reliable interaction with simpler proxies, and better reproduces held-out motion than direct parameter inference.Source: §5 · p. 10
• Its shared representation supports geometric reconstruction, task-aware collision refinement, and residual-guided dynamic calibration.Source: §5 · p. 10
• Later refinement fixes geometry and kinematics, while validation is limited to selected interactions, sampled poses, and observed passive responses.Future work includes selective cross-stage revision and active acquisition of informative interactions.Source: §5 · p. 10

• Generative AI tools assisted with writing, language polishing, and literature retrieval and discovery.Source: AI USE DISCLOSURE · p. 10
• The authors reviewed and verified all AI-assisted content and references.Source: AI USE DISCLOSURE · p. 10
• The authors take full responsibility for the final content of the work.Source: AI USE DISCLOSURE · p. 10

A DATA AND EXPERIMENTAL SETTINGS

• Geometry evaluation uses all 21 reference assets, while contact evaluation uses 19 and excludes the grand piano and oven 6 333 because non-manifold geometry prevents reliable closed-material references.The collection comprises 13 ArtVIP assets and eight artist-created commercial assets; geometry uses complete reference surfaces, including interiors.Source: A DATA AND EXPERIMENTAL SETTINGS · p. 15, §A.1 · p. 15

A.2 REAL-OBJECT RECORDINGS

• Dynamics evaluation uses six real objects reconstructed through the geometry and contact stages before calibration and evaluation.One or two clips per object are held out exclusively for evaluation; the remaining clips support observation extraction, response-model selection, parameter fitting, and stopping decisions.Source: A DATA AND EXPERIMENTAL SETTINGS · p. 15, §A.2 · p. 15

A.1 REFERENCE ASSETS

• Contact evaluation uses reviewed closed material units, excludes visual-only geometry, and omits single-kinematic-body objects from self-collision rate evaluation.Repairs to open boundaries and material solids, along with reviewed body assignments and joints, are shared across contact methods.Source: §A.1 · p. 15

A.2 REAL-OBJECT RECORDINGS

• A measured exterior dimension sets s = Lreal/Lmesh, and uniform scaling applies to geometry, joint locations, and prismatic travel but not revolute angles.Recordings are 1280×720 at approximately 60 FPS.Source: §A.2 · p. 15

B.1 IMPLEMENTATION AND VALIDATION

• Reconstructed meshes are generated from feature parameters rather than copied source triangles, while exported identifiers preserve assembly correspondences.Feature plans associate editable features with source regions while preserving node identities, and parameter records track measurements, constraints, adjustments, defaults, and user inputs.Source: §B.1 · p. 15
• Automated validation requires per-axis dimension errors below 5%, center error and directed global P95 surface distances below 0.02DM, and silhouette IoU of at least 0.90.Checks use source assembly coordinates without alignment; unsegmented inputs use recovered regions only for visual diagnosis, not source-part ground truth.Source: §B.1 · p. 16

B.2 INPUTS AND BASELINE CONFIGURATIONS

• For image-input comparisons, Codex and Articraft receive only the input image, while FACT incorporates Rodin generation and freezes the resulting asset for reconstruction.Source: §B.2 · p. 16
• For oracle-mesh evaluation, learned baselines receive whole-object geometry without part grouping, while agentic methods retain the scene hierarchy and may inspect individual parts.All agentic geometry methods use GPT-6 Astra with High reasoning level; Codex uses Blender MCP without FACT's reconstruction workflow, and Articraft follows its native pipeline.Source: §B.2 · p. 16

B.3 GEOMETRY EVALUATION METRICS

• Candidate surfaces are uniformly aligned by their AABB centers and longest side lengths, without rotation search, ICP, nonuniform scaling, or per-part alignment.The scores therefore capture normalized shape and relative assembly dimensions, not absolute scale or global translation.Source: §B.3 · p. 16
• Distances use 4,096 area-weighted samples in each direction, with symmetric Chamfer distance, reconstruction-to-reference P95 distance, and F-scores reported from unsquared point-to-surface distances.Missing reference geometry contributes to reverse Chamfer distance and F-score recall; SD-P95 measures reconstruction-to-reference error only.Source: §B.3 · pp. 16–17
• Silhouette IoU averages or takes the minimum across three orthographic projections rasterized as 512×512 binary masks.Each candidate–reference pair shares projected bounds, an isotropic pixel scale, and a 31-pixel margin.Source: §B.3 · p. 17

B.4 CASE STUDY: BAKING-CHAMBER HANDLE REFINEMENT

• The handle was measured from a 19-vertex source-coordinate section and revised as lofted, height-varying half-superellipse sections while preserving its open mating face.The fitted parameters were cx ≈0.02538, hx ≈0.06353, yf ≈−0.48697, d ≈0.06631, and e⋆≈2.6593.Source: §B.4 · p. 17
• After whole-object validation passed, part-level diagnostics exposed an inaccurate handle profile and spurious closure surfaces; revision brought all eight parts within the local threshold.The global reconstruction-to-source P95 distance increased slightly after revision.Source: §B.4 · p. 17, Table 8 · p. 17

C.1 PROXY INITIALIZATION, REPAIR, AND COMPRESSION

• The initial proxy separates closed, consistently oriented face-connected components within each source part, groups residual components, and sets the hull budget to that decomposition's count.Repartitioning preserves source-face ownership and stays within rigid-body boundaries.Source: §C.1 · p. 17
• Repair accepts only candidates within the hull budget that pass sampled motion checks and bounded volumetric and free-space error increases, then ranks feasible candidates by contact P95, development loss, and complexity.Compression is evaluated against the fixed repaired anchor, with per-body, ROI, and source-normal fidelity guards.Source: §C.1 · p. 18

C.2 CONTACT DIAGNOSTICS AND METRICS

• Contact diagnostics measure proxy volume fidelity, false interbody collisions at reference-free poses, and occupancy of task-specific access and motion-clearance regions.Free-space probes are defined from reference material independently of the candidate proxy, with eligibility thresholds for sampled regions.Source: §C.2 · p. 18

C.3 INTERACTION PROTOCOL

• Each object receives a prescribed gripper or pressing-pad task, and passive joints move only through contact; ten trials share controller settings and perturbation offsets across methods.Success requires sustained articulation progress or lifting criteria, alongside penetration and tracking limits and no forbidden contacts or solver warnings.Source: §C.3 · p. 19

C.4 SIMULATION AND TIMING

• Contact simulations use fixed MuJoCo settings and object properties across methods, while physics-step timing excludes control, rendering, loading, and geometry audits.Timing is measured on an Intel Core i9-14900K after a warm-up and averaged across steps, seeds, and objects.Source: §C.4 · p. 19

C.5 MICROWAVE REFINEMENT TRACE

• The final microwave proxy removes 82 hulls within the prescribed fidelity bounds and succeeds in all ten interaction trials.Further local edits either violate fidelity guards or do not reduce hull count; the result is a local stopping point.Source: §C.5 · p. 19, Table 9 · p. 19

D DYNAMIC FIDELITY

• Passive-response prediction is evaluated on six reconstructed real objects, with one or two clips per object held out exclusively for evaluation.FACT and baselines use the same calibration/evaluation split.Source: D DYNAMIC FIDELITY · p. 20

D.1 TRAJECTORY EXTRACTION

• Video trajectories are mapped to reconstructed joint coordinates using image features and geometry-aware mappings, with uncertain detections filtered, interpolated only when supported, or masked.Evaluation uses accepted unsmoothed position measurements, and all methods share references, mappings, masks, and windows.Source: §D.1 · p. 20

D.2 RESPONSE MODELS

• Agent-constructed models capture varied passive effects across six mechanisms, including dry and viscous resistance, state-dependent restoring terms, piecewise spring–damper behavior, and impact-like closure responses.The listed examples include Coulomb lid resistance, drawer viscous/Coulomb resistance, and cabinet piecewise response.Source: §D.2 · p. 20, Table 10

D.3 PARAMETER FITTING

• After fitting, residual diagnosis distinguishes parameter or initial-condition errors from inadequate model form; the agent revises variables, bounds, or structure and refits, retaining the simplest model meeting per-sequence error criteria.Geometry, kinematics, inertial estimates, camera mappings, and time offsets remain fixed during fitting.Source: §D.3 · p. 21

D.4 REPLAY AND EVALUATION

• Direct-inference baselines use calibration observations only, then replay their fixed inferred models or parameters on held-out clips.Because input modalities and parameterizations differ, the comparison measures end-to-end motion prediction rather than identical intermediate representations.Source: §D.4 · pp. 21–22
• Replay shares reconstructed geometry, joint coordinates, observation-derived initial states, and numerical settings within each object, with no external input after release.Initial states come from a short release window and are not fitted to evaluation trajectories.Source: §D.4 · p. 22
• Observed and simulated trajectories are interpolated onto a common approximately 60 Hz grid, with shared smoothing, masking, and reference-determined scoring windows.Position metrics use accepted unsmoothed observations; velocities use the shared smoothing and numerical-differentiation procedure.Source: §D.4 · p. 22