ArtVIP: Articulated Digital Assets of Visual Realism, Modular Interaction, and Physical Fidelity for Robot Learning

Motivation

• Simulation offers low-cost data, virtual environments, rapid deployment, and standardized testing without hardware-damage or safety concerns.These properties motivate simulation as an efficient alternative for advancing robot learning.Source: §1 · p. 2
• High-quality articulated-object assets are needed because existing datasets provide limited visual realism, physical fidelity, or simulator flexibility.The paper identifies asset quality, rather than quantity, as the central bottleneck and specifies visual realism, modular interaction, physical fidelity, and simulation friendliness as requirements.Source: §1 · pp. 2–3

ArtVIP

• ArtVIP embeds interaction semantics within assets and supplies pixel-level affordance annotations, enabling modular reuse and scalable behavior modeling.These features are designed to support manipulation skills including rotating, clicking, pulling, and pressing.Source: §1 · p. 3

Applications

• The dataset includes configured indoor scenes and supports imitation-learning, reinforcement-learning, and 3D-construction experiments.Assets are distributed in USD and can be converted to legacy formats such as URDF or MJCF for established robotics workflows.Source: §1 · p. 3

Simulation Platforms

• Simulation platforms combine physics and rendering engines, but differing platform capabilities make broadly compatible, ready-to-use assets valuable.The paper contrasts robotics-oriented simulators such as MuJoCo and Webots with game engines that do not natively support ROS.Source: §2 · p. 3

Datasets for Robot Simulation

• Indoor-scene datasets support navigation but lack GUI-based editing, while ShapeNet, Objaverse, and related assets generally function only as rigid bodies.Rigid-body-only assets cannot support articulated manipulation tasks.Source: §2 · pp. 3–4

Articulated Objects

• Existing articulated-object datasets and construction methods do not consistently combine visual realism, physical fidelity, simulator compatibility, and scalable production.The related work discusses both dataset limitations and generation methods that remain reliable mainly for simple joints.Source: §2 · p. 4

Construction and Generation

• Public-repository assets often have inconsistent modeling quality, disorganized part hierarchies, and non-standardized coordinate systems requiring manual preprocessing.Generative and reconstruction methods scale more easily but are not yet mature enough to ensure quality.Source: §2 · p. 4

3.1 VISUAL REALISM

• Professional modelers build each object hierarchically from assembly to module to mesh under unified modeling, texture, material, and coordinate guidelines.The top-down mechanical approach organizes rigid-body modules and their mesh parts before bottom-up assembly and joint integration.Source: §3.1 · pp. 4–5, Figure 2 · p. 4

3.2 PHYSICAL FIDELITY

• ArtVIP balances contact fidelity and computational efficiency by selecting convex hulls, convex decompositions, or fine-tuned collision meshes for each object.The collision representation is chosen according to geometry complexity and affordance requirements.Source: §3.2 · p. 5
• The joint model extends basic stiffness-and-damping dynamics by making stiffness and target terms functions of joint position, with velocity dependence when needed.This parameterization represents variable dynamics in complex joints such as door closers and light switches.Source: §3.2 · p. 5

3.3 MODULAR INTERACTION

• ArtVIP embeds customizable behaviors directly in assets, covering five canonical interaction primitives across 394 assets and more than 900 joints.The primitives include latching or magnetic closure, damping, cross-asset effects, within-asset effects, and hover or hold position.Source: §3.3 · pp. 5–6, Figure 3 · p. 5
• Binding behaviors at design time lets users import a USD file and obtain interaction affordances without writing additional code.The modular design is intended to reduce development overhead and accelerate algorithm iteration.Source: §3.3 · pp. 5–6

Evaluation Scope

• The evaluation combines digital-twin comparisons with quantitative visual and physical tests to assess ArtVIP’s realism and interaction fidelity.The visual tests include triangle counts, reconstruction performance, and feature distributions, while physical testing tracks joint motion against real-world trajectories.Source: §4 · p. 6, §4.1 · p. 6, §4.2 · p. 7

Visual Realism Evaluations

• ArtVIP assets show richer geometric detail than comparison datasets, with dense triangular meshes preserving surface smoothness and minimizing faceting.The comparison covers object categories appearing in all three datasets.Source: §4.1 · p. 6, Figure 4 · p. 6
• ArtVIP feature distributions align more closely with real-world data than OmniGibson features in the CLIP-based comparison.The analysis samples 100 ArtVIP models and corresponding or semantically similar OmniGibson and real-world objects using matched viewpoints.Source: Figure 5 · p. 7, §4.1 · p. 7
• ArtVIP produces reconstructions with higher structural fidelity and finer detail preservation than OmniGibson under identical multi-view sampling.The evaluation uses VGGT on matched asset views.Source: §4.1 · p. 7, Figure 5 · p. 7

Physical Fidelity and Interaction Evaluations

• Simulated and real drawer displacement trajectories closely agree across applied forces, demonstrating physical fidelity of ArtVIP joints.The real-world trials apply horizontal pulling forces of 1 N, 1.5 N, 2 N, and 2.5 N, while simulation compares default and optimized joint parameters.Source: §4.2 · p. 7, Figure 6 · p. 8

Experimental Setup

• The imitation-learning study uses four tasks, 100 successful real trajectories and 100 simulated trajectories per task, and ACT and DP baselines.The tasks are PullDrawer, OpenCabinet, SlideShelf, and CloseOven, with RGB observations from four viewpoints and proprioceptive states.Source: §5.1 · pp. 8–9

Imitation Learning in Real World Environments

• Simulation-trained policies achieve zero-shot real-world success, while mixing real and simulated data improves performance across articulated-object manipulation tasks.For example, ACT reaches 39% on PullDrawer with simulation-only training, and DP on SlideShelf rises from 44% to 59% with mixed data.Source: §5.1 · p. 9, Table 1 · p. 9
• Equal-volume real-world training still outperforms simulation, with DP reaching 49% versus 10% on OpenCabinet, showing persistent sim-to-real challenges.The experiments compare real-only, sim-only, and mixed datasets for ACT and DP across four tasks.Source: §5.1 · p. 9

Digital-Cousin Comparison

• Higher-quality ArtVIP assets produce stronger zero-shot sim-to-real transfer and higher mixed-data success than PartNet-Mobility on the microwave door-pull task.The digital-cousin comparison uses five microwaves from each dataset and evaluates real-only, sim-only, and mixed settings.Source: §5.2 · pp. 9–10, Table 2 · p. 10

Reinforcement Learning in High-Fidelity Simulators

• Simulation-to-real reinforcement-learning performance has a Pearson correlation coefficient of 0.9886 across checkpoints, supporting ArtVIP’s physical and perceptual fidelity.A policy trained in Isaac Sim is deployed unchanged in the real world and evaluated across five checkpoints.Source: §5.3 · p. 10, Table 3 · p. 10

Conclusion

• ArtVIP provides visual realism, accurate physical properties, and modular interaction capabilities for articulated-object robotic manipulation.The dataset includes 992 objects across 9 categories and 37 subcategories, plus six physically interactive indoor environments.Source: §6 · p. 10, A ARTICULATED OBJECTS · p. 17, B SCENES · p. 17

Limitations

• Scaling remains limited by the intensive human labor required for asset modeling, motivating future generative synthesis methods.The stated future direction is to reduce manual effort while broadening object diversity.Source: §6 · p. 10

Annotations

• ArtVIP’s annotations describe functional object parts to support task-appropriate interaction behavior.The annotation design depends on consistent modeling standards that preserve functionally distinct components.Source: C ANNOTATIONS · p. 17, Table 5 · p. 20

Visual modeling standards

• High-quality meshes, textures, and materials support realistic appearance and reliable collision-based interaction in simulation.ArtVIP uses manifold meshes, high-resolution UV-aligned textures, and physically based rendering with RTX Renderer.Source: D MODELING STANDARDS · p. 19

Physical fidelity

• ArtVIP models complex joint behavior by extending basic drive dynamics with position- and velocity-dependent stiffness, targets, damping, and friction.The friction model covers static, maximum static, and dynamic friction, while latch and door-closer mechanisms use state- and position-dependent transitions.Source: E PHYSICAL FIDELITY OF JOINTS · pp. 19–20

Chart comparison with existing datasets

• Table 6 provides a detailed comparison of ArtVIP with existing articulated-object datasets.Source: Table 6 · p. 21, F CHART COMPARISON WITH EXISTING… · p. 21

Visual realism comparison

• ArtVIP uses more triangular faces than BEHAVIOR-1K and PartNet-Mobility for categories such as toilets and refrigerators, improving geometric detail at a frame-rate cost.Profiling was conducted to optimize simulation frame rate for each object.Source: G VISUAL REALISM COMPARISON · p. 21
• Table 7 reports category-wise averages for triangle count, active joints, and FPS across ArtVIP, PartNet-Mobility, and BEHAVIOR-1K.Source: G VISUAL REALISM COMPARISON · p. 21, Table 7 · p. 22
• Approximately 100k triangles and up to 20 active joints have negligible FPS impact for a single object, while both factors reduce FPS in complex scenes.The kitchen scene contains 65 actuated joints, making scene complexity an important performance condition.Source: G VISUAL REALISM COMPARISON · pp. 21–22

Physical fidelity evaluation

• Microwave button-press experiments compare real-world and simulated door-opening trajectories to validate modular interaction behavior.Real motion is captured with optical tracking, while simulation uses a virtual marker.Source: H PHYSICAL FIDELITY AND INTERACTION… · p. 22, Figure 12 · p. 24
• Door-closer evaluation analyzes linear and angular velocities as the door moves from a threshold state to full closure.Source: H PHYSICAL FIDELITY AND INTERACTION… · p. 22, Figure 13 · p. 24

Task design

• Four manipulation tasks—PullDrawer, OpenCabinet, SlideShelf, and CloseOven—require rotation, angled pushing, and horizontal translation.Source: I IMITATION LEARNING APPLICATION · p. 22, Figure 14 · p. 25

Imitation learning application

• ACT and Diffusion Policy receive multi-view RGB images and proprioceptive states, then output robot control signals for end-to-end task execution.Implementation details are provided in Tables 8 and 9.Source: I IMITATION LEARNING APPLICATION · p. 25, Table 8 · p. 25, Table 9 · p. 26
• The full imitation-learning experiment results are presented in Table 10.Source: I IMITATION LEARNING APPLICATION · p. 25

Task and Training Setup

• The CloseTrashcan task requires an agent to approach the lid and close it smoothly within a time limit.Source: J REINFORCEMENT LEARNING APPLICATION · p. 26

Training Details

• Stage 1 trains a PPO teacher from privileged low-level states, while Stage 2 distills that teacher into a visuomotor student.Source: J REINFORCEMENT LEARNING APPLICATION · p. 26

Reward Functions

• The multi-objective reward combines gripper–lid proximity, directional alignment, closure progress, and action smoothness.The reported weights are λ1 = 0.5, λ2 = 0.125, λ3 = 10, and λ4 = −0.01.Source: J REINFORCEMENT LEARNING APPLICATION · p. 26

Baseline Comparison

• EAGLE achieves a 98% success rate after 500k training iterations, whereas the vision-based PPO baseline performs poorly on CloseTrashcan.The comparison attributes the baseline’s poor performance to high computational complexity and low data diversity.Source: J REINFORCEMENT LEARNING APPLICATION · p. 27, Table 12 · p. 28

• r = 0.9886 indicates a strong linear relationship between simulated and real-world success rates across checkpoints.The coefficient is computed from corresponding simulation and real-world success rates in Tab. 3.Source: K PEARSON CORRELATION COEFFICIENT… · p. 28

• Generative outputs exhibit self-collisions, mesh distortions, incorrect joint parameters, unrealistic materials, and missing interior details, unlike ArtVIP digital-twin assets.The reported failures lead the authors to conclude that current generative baselines are not simulation-ready for articulated assets.Source: L COMPARISON TO GENERATIVE PIPELINES · p. 28, Figure 16 · p. 29
• On real-world cabinet and fridge inputs, generative reconstruction performance degrades markedly compared with synthetic inputs, revealing a substantial sim-to-real gap.The comparison uses reconstruction metrics reported in Tab. 13 for the same object categories.Source: L COMPARISON TO GENERATIVE PIPELINES · p. 28, Table 13 · p. 30
• Generated cabinet and fridge assets contain approximately 100× more triangles than ArtVIP counterparts, reducing simulation frame rate from approximately 90 fps to 70 fps.Source: L COMPARISON TO GENERATIVE PIPELINES · p. 28
• Generative pipelines commonly rely on synthetic training data with calibrated cameras and controllable materials, limiting their robustness on real images.Source: L COMPARISON TO GENERATIVE PIPELINES · p. 29
• Real-image reconstruction accuracy degrades when camera extrinsics are estimated rather than known.Source: L COMPARISON TO GENERATIVE PIPELINES · p. 29
• Generalization is constrained by PartNet-Mobility category coverage and articulation priors, while low-texture planar structures may appear warped.Source: L COMPARISON TO GENERATIVE PIPELINES · p. 29
• Reconstruction outputs may lack physically consistent materials and textures, while spherical-harmonic colors require conversion or baking for simulator rendering.Source: L COMPARISON TO GENERATIVE PIPELINES · p. 29
• Current metrics do not capture mesh efficiency or collision performance relevant to simulation frame rate.Source: L COMPARISON TO GENERATIVE PIPELINES · p. 29