Lucid-XR: An Extended-Reality Data Engine for Robotic Manipulation

1 Introduction: From Atoms to Bits

• Real-world datasets rarely contain mission-critical events, while creating realistic virtual worlds at the scale needed for generalization is infeasibly expensive.Source: §1 · pp. 1–2
• Lucid-XR turns human demonstrations in low-fidelity virtual environments into diverse, realistic training data using language- and text-to-image-guided generation.The pipeline amplifies a small number of simple designs into millions of multiview images; the results use basic handcrafted 3D scenes and author-collected data.Source: §1 · p. 2, Figure 2 · p. 2
• Lucid-XR moves physics simulation onto XR devices and retargets human poses to virtual robots without requiring a custom program.Source: §1 · p. 2

2 A Touch of Physics in Extended Reality (XR)

• Browser compute and memory are constrained by operating-system throttling, while newer XR devices receive stronger browser support; the implementation uses WebAssembly, web-XR, and webGL.Source: §2 · p. 2

2.1 Multi-Physics Simulation in Vuer

• The browser supports flexible-material contact, SDF collisions for non-convex shapes, wind and fluid interactions, and soft-skin contact; SDF collision avoids convex decomposition but costs more compute and memory.Source: Figure 4 · p. 3, §2.1 · p. 3
• More complex particle, deformable-vertex, or liquid physics is a future possibility when WebGPU supports compute shaders and hardware acceleration.Source: §2 · p. 3
• On-device simulation runs at 90 fps on Apple Vision Pro, with rendering at 90 fps and data collected at 25 fps; local physics avoids network delays and supports interactive deformable objects.Each simulation step takes under 12 ms, so latency is negligible unless overloaded.Source: §2.1 · p. 3, Figure 3 · p. 3

2.2 Precise Interactions at A Distance: Hitchhiking Controllers

• Remote interaction applies an SE(3) transform in the mocap site's local frame, enabling distance activation by gaze and click before natural grasping gestures.The approach addresses the poor experience of naive remote grasping, where tracking error grows with distance; examples include balancing blocks, throwing a basketball, and folding cloth.Source: §2.2 · pp. 3–4, Figure 5 · p. 4, Figure 6 · p. 4

2.3 On-Device Retargeting for Dexterous Hand Control

• Binding mocap sites to finger tips and using their relative pose to the wrist and action space supports retargeting across different human and robot kinematics.A schema-based interface lets users specify bindings between mocap sites or geometric landmarks and hand gestures.Source: §2.3 · p. 4

2.4 Porting Existing Environments

• Existing environments from RoboHive, RoboCasa, RoboSuite, and MuJoCo Menagerie can be extracted and loaded into Vuer as asset bundles.Source: §2.4 · p. 4

3 Synthesizing Diverse Manipulation Data from Virtual Demonstrations

• Lucid-XR’s generative engine converts virtual human demonstrations into diverse, realistic-looking multiview image datasets for robot training.This is the third component of the system.Source: §3 · p. 4
• Examples pair a MuJoCo Menagerie Spot robot with RobotCasa scenes, illustrating robot and scene assets used with LucidXR.Source: Figure 7 · p. 5

3.1 Generating Realistic Images from Virtual Demonstrations.

• Diverse text prompts and simulation semantic masks guide image generation, with varied prompts reported as key to producing images.Figure 8 depicts the pipeline’s semantic masking and depth conditioning.Source: §3.1 · p. 5, Figure 8 · p. 5

3.2 Demonstration Augmentation

• Procedurally generated MuJoCo XML scenes let users compose LucidXR environments and control initial scene configurations during data collection.Source: §3.2 · p. 5
• Camera poses can be changed after demonstrations, while ground-truth depth and camera calibration support image warping and optical flow for nearby poses.The authors describe this as important for preventing sensitivity to camera pose in sim-to-real transfer.Source: §3.2 · p. 5
• Trajectory warping moves selected keypoints with pose and rotation interpolation to create new demonstrations and vary object, robot, and initial-scene positions.The authors use it to improve robustness to object-position variation.Source: §3.2 · p. 5

4 Results

• The contact-rich task suite spans block stacking, pouring particles, ball sorting, knot tying, kitchen-sink placement, and mug placement, with randomized initial states.Figure 9 depicts granular, deformable, and rigid-contact examples.Source: §4 · pp. 5–6, Figure 9 · p. 6

4.1 Data Collection and Learning Setup

• Demonstrations record embodiment-agnostic SE(3) mocap poses at 25 Hz; policies use proprioception and wrist or fixed RGB views, with ACT and diffusion variants.ACT uses 15k updates and chunk size 25; the diffusion model uses 1000 denoising steps. Kitchen-clearing scores allow four points per run across two pick-and-places.Source: §4.1 · p. 6, Table 1 · p. 7

4.2 Comparing Data Collection Speed in Virtual and the Real-world

• Participants collected roughly 2× more demonstrations in Lucid-XR than with real-world teleoperation, and augmentation raised effective dataset size to about 5× the real-world baseline.They collected 30 minutes of demonstrations for each of three tasks in both settings; Figure 10 compares the data amounts.Source: §4.2 · p. 6, Figure 10 · p. 6

4.3 Real-to-Sim Evaluation

• In small numbers of cluttered kitchen staging environments modeled as 3D Gaussians, the unseen-environment Lucid-XR policy outperformed the policy trained on simulated data.Figure 11 presents real-to-sim transfer results and per-task simulation performance.Source: §4.3 · p. 7, Figure 11 · p. 7

4.4 Sim-To-Real Evaluation

• On the same evaluation environment, policies trained on purely synthetic data performed comparably to policies trained on real data.The pick-and-place datasets used 10, 20, and 30 minutes of collection; Figure 12 presents the results.Source: §4.4 · p. 7
• Under changed lighting, color, and tabletop conditions, LucidXR-trained policies maintained high success rates and outperformed the real-world-trained baseline, which failed to generalize.The evaluation replaced the wooden tabletop with textured or black tablecloths.Source: §4.4 · p. 7

5 Related Works

• XR teleoperation methods include gaze-controlled virtual avatars, hand overlays [20], mobile AR [24] [25], and physical-robot AR/VR frameworks [10] [26].The paper says throughput and environment diversity remain limited in many frameworks.Source: §5 · p. 7
• LucidXR’s hitchhiking controller builds on gaze-directed virtual-avatar interaction, while its on-device simulation avoids cloud latency associated with DART and IRIS.Action Chunking Transformer architecture foundations are attributed to [14].Source: §5 · pp. 7–8
• LucidXR builds on LucidSim’s generated-image approach [11], adding semantic classes and randomized object geometry, while other generation methods [31] [32] [34] are described as lacking realistic textures.Works [35] [36] [37] [38] instead generate scenarios on real data while maintaining semantic consistency with robot objectives.Source: §5 · p. 8
• Large demonstration collections such as Open X-Embodiment [39], RH20T, and DROID [41] remain embodiment-specific or limited in diversity across robots, environments, and tasks.The paper frames that diversity as falling short of what is needed for generalist policies.Source: §5 · p. 8

6 Conclusion

• The pipeline produces visual manipulation policies that generalize across object instances, appearance, and lighting conditions.The authors also present VR-controller and hand-gesture interaction designs for virtual robots.Source: §6 · p. 8
• Cross-embodiment transfer is limited by inverse-kinematics and mobility.The authors observe that full-embodiment data supports transfer, and that embodiment-free data collected with a floating gripper often transfers readily.Source: §6 · p. 8

7 Limitations

• Consistent views rely on using the same text prompt, while future iterations could use paired text labels for additional supervision.The authors note that generated visual data already comes with paired text labels.Source: §7 · p. 9

Appendix

• The appendix gathers additional simulated environments, physics-engineering and generative-workflow details, virtual demonstrations, and synthetic training-image examples.It also includes a live presentation in the accompanying video.Source: Appendix · p. 13

7.1 Examples of Dexterous Virtual Demonstrations

• A virtual dishwasher scene demonstrates dexterous manipulation around articulated doors, a plate, and interior racks.Source: Figure 13 · p. 13

7.2 Examples of Policy Unroll in Simulation

• In the simulated kitchen unroll, the robot stacks a cup on a bowl and then places both in the sink.Source: Figure 14 · p. 13

7.3 Examples of Learned Re-try Behavior

• The mug-tree policy retries after contact during both mug pickup and an attempt to hang it on the rack arm.Source: §7.3 · p. 14, Figure 15 · p. 14

7.4 Additional Examples of Synthetic Data

• Additional generated images illustrate control over lighting, geometry, and content diversity.Source: Figure 16 · p. 15, §7.4 · p. 15

7.5 The Vuer Scene Description Language

• The vuer-mujoco scene-description language represents simulations as nested scene components; its example includes a fishing toy whose magnetic hook is simulated with adhesion actuators.The code example composes cameras, lighting, a table, a cube, a robot, and mocap points.Source: §7.5 · p. 15, Figure 17 · p. 16

7.6 Tips on Engineering Synthetic Environments

• The fishing-toy simulation combines a flexible rope with magnetic adhesion tuned to lift wooden fish pieces yet release them after a slight rod shake.The hook interacts with a ferromagnetic nail head embedded in each fish; the simulation models this with adhesion actuators.Source: §7.6 · p. 16
• Generated 3D meshes are simplified, centered, and rescaled because they often have excessive faces and dimensions off by two orders of magnitude.Source: §7.6 · p. 16

7.7 Real-to-sim Evaluation Setup

• Clean and messy kitchens provide controllable proxies for real-world experiments, and the PolyCam mesh is manually aligned to MuJoCo.The clean kitchen contains no clutter, with 3D object assets placed programmatically.Source: §7.7 · p. 16, Figure 18 · p. 17, Figure 19 · p. 17

7.8 Image Generation Workflow

• Two object masks and a normalized inverse-depth image control generated-image geometry, lighting, and composition.The workflow is provided as JSON that can be loaded into ComfyUI.Source: Figure 20 · p. 17, §7.8 · p. 17

7.9 Training Details

• The policy uses an Action-Chunking transformer with multiple backbones, one for each input camera feed; its hyperparameters are listed in Table 2.Source: §7.9 · p. 18, Table 2 · p. 18

• One kitchen prompt pairs a minimalist acrylic cup and an antique-style bowl with scattered baking supplies and reflective lighting.Source: §7.10 · p. 18
• An alternate kitchen prompt describes a glossy, multicolored bowl and patterned cup against a cluttered countertop with fruit and cracked tiles.Source: §7.10 · p. 18
• Mug-tree prompts vary the mug and rack descriptions while specifying a bar counter with dim lighting, stains, and scattered bar items.Source: §7.10 · pp. 18–19
• Another mug-tree example describes a scuffed novelty mug and transparent rack amid droplets, salt grains, aged beer mats, and bar-counter clutter.Source: §7.10 · p. 19