Source-linked AI summary
RoboCousin: Build Your Own Simulation Playground for Robust Bimanual Robotic Manipulation
Jingxuan Zhu, Jingyi Li, LiangLiang Chen, Zhiyuan Jing, Jidong Zhang, Hongming Li
TL;DR
Bimanual manipulation needs broad training distributions, but physical demonstrations are costly and existing simulators make new objects and environments laborious to integrate. RoboCousin converts observations into interaction-ready assets and structured digital-cousin scenes, then generates executable manipulation and navigation data. Its experiments show comparable contact annotations, effective sim-to-real supervision from generated assets, and improved or competitive transfer from tabletop cousins.
Problem
Existing simulation pipelines rely on closed asset libraries and predefined scenes, making newly observed objects and workspaces costly to reconstruct, annotate, and integrate for scalable bimanual data generation.
Method
RoboCousin converts observations into simulator-ready assets with geometry, metadata, and grasp-contact candidates, then varies compatible objects, backgrounds, layouts, language, and navigation contexts while preserving task structure.
Results
Across evaluated experiments, generated contact points are comparable to curated annotations, generated assets match or outperform RoboTwin 2.0 assets, and tabletop cousins improve transfer on three of four tasks while remaining competitive on the fourth.
Takeaways & Limitations
RoboCousin provides a practical route to expanding both the scale and coverage of synthetic bimanual manipulation data beyond a single reconstructed scene.
Abstract
from arXiv · showhide
Bimanual manipulation policies require large and diverse training datasets, yet collecting demonstrations on physical robots is expensive and difficult to scale. Simulation can generate data efficiently, but existing pipelines typically operate within closed asset libraries and predefined scenes: adding a newly observed object or environment still requires substantial effort to reconstruct geometry, specify physical and semantic properties, annotate interactions, and integrate the result into executable tasks. We present RoboCousin, an extensible simulation-based data-generation platform that turns user-provided observations into reusable assets, scenes, and expert trajectories for bimanual manipulation. Built on RoboTwin~2.0, RoboCousin converts object images into simulation-ready assets with visual and collision geometry, semantic and physical metadata, and automatically generated grasp-contact candidates. It further constructs digital cousins that vary compatible objects, backgrounds, layouts, and language instructions while preserving task-relevant affordances and spatial relations. The same asset system supports tabletop and room-level scene construction, with collision-aware base control for interaction beyond a fixed workspace. We release RoboCousin-OBD, containing more than 3,000 annotated object instances and 50 background environments, and use RoboCousin to generate over one million expert trajectories across 50 tasks. Simulation and real-robot experiments show that the automatically generated interaction annotations are comparable to curated annotations, generated assets provide effective sim-to-real supervision, and tabletop cousins can improve transfer beyond training on a single reconstructed scene. RoboCousin therefore provides a practical path for expanding both the scale and coverage of synthetic bimanual manipulation data.
1 Introduction
RoboCousin addresses the limited diversity and manual integration burden of closed simulation pipelines by turning user observations into interaction-ready assets, varied scenes, and executable bimanual data. It supports scalable asset, cousin, and navigation-augmented trajectory generation.
- Motivation: Physical-robot demonstrations are expensive, slow, and difficult to scale, while robust bimanual policies require diversity across objects, layouts, environments, instructions, and embodiments.Simulation offers automated supervision and controlled variation at lower cost.
- Problem: Existing systems generate data inside closed worlds, so adding a new object or workspace still requires manual geometry, collision, physical-property, interaction, and task integration.More rollouts can increase coverage within the original simulator distribution without expanding its underlying object and environment distribution.
- Contribution: RoboCousin converts visual and textual inputs into simulator-ready objects and backgrounds with visual and collision geometry, semantic and physical metadata, and grasp-contact candidates.The same asset system supports object-level and environment-level expansion.
- Contribution: Digital cousins vary compatible objects, backgrounds, layouts, and language instructions while preserving task-relevant semantic relations and support structures.The framework connects cousin scenes directly to automated bimanual expert trajectory collection.
- Scale and evaluation: Room-level collision-aware routes can be prepended to tabletop demonstrations, extending fixed-workspace manipulation data into longer navigation-and-manipulation trajectories.Routes vary initial poses, environments, and approach paths while ending at the pose expected by the existing manipulation pipeline.
- Scale and evaluation: More than 3,000 annotated object instances, 50 background environments, and over one million expert trajectories across 50 bimanual tasks are released or generated.The platform supports asset-centric, tabletop-cousin, and navigation-augmented data-generation workflows.
2 Method
RoboCousin builds simulation-ready assets from observations, organizes them into structured household and tabletop cousin scenes, and connects those scenes to executable manipulation and navigation data collection. Its randomization expands visual, spatial, semantic, and linguistic variation while retaining task-relevant structure.
- 2.1 Digitalize Anything: The pipeline uses 3D reconstruction, 3D Gaussian Splatting, and language models to synthesize interactable objects and background environments from images.These outputs form an object-background asset library for user-defined instances and workplace environments.
- 2.1 Digitalize Anything: The real-to-sim asset pipeline converts visual and textual inputs into object, interaction, and scene-level assets required by the simulator.It is designed to address large-scale simulation data-generation bottlenecks.
- 2.1 Digitalize Anything: Objects are segmented, reconstructed as textured meshes, converted to URDF assets, and assigned semantic and physical attributes from multi-view renderings.The estimated attributes include category, scale, mass, friction coefficients, color, and descriptive metadata.
- 2.1 Digitalize Anything: Contact-point synthesis normalizes geometry, extracts AABB and OBB representations, selects grasp strategies by symmetry, and filters infeasible candidates.Mixed grasping targets approximately rotationally symmetric objects, whereas OBB grasping targets asymmetric or box-like objects.
- 2.2 Generate Digital Cousins: Room prompts become structured JSON layouts specifying furniture, anchor objects, relative placements, spacing, rotations, and static attributes before concrete assets are loaded into the simulator.Separating room planning from asset selection supports consistent instantiation and later randomization.
- 2.2 Generate Digital Cousins: Digital-cousin randomization expands training diversity semantically while preserving geometric and semantic affordances, including object identity, support relations, and tabletop layouts.Tabletop cousins derive from real desktop images and use structured layouts with object labels, relative positions, candidate instances, and support relations.
- 2.3 Robots Should Move: Navigation replay serializes routes as planar poses and moves the complete robot through alternating rotations and translations to reach the tabletop collection station.This adds room-scale episodes with varied starting poses, path lengths, obstacle contexts, and visual observations to existing manipulation tasks.
3 RoboCousin Data Generator, User Interface and Large Scale Dataset
RoboCousin-OBD combines annotated object and background assets with three integrated data-generation modes for scalable bimanual manipulation. Its interface supports multimodal inputs, quality checks, simulation monitoring, and human-in-the-loop verification while preserving automated trajectory generation.
- RoboCousin-OBD: RoboCousin-OBD contains over 300 object categories, 3,000 object instances, and 50 background environments across 6 household scenarios.Objects include dimensions, mass, friction coefficients, and interaction-oriented annotations; backgrounds support varied household simulation settings.
- RoboCousin-OBD: The dataset encodes graspable regions, placement surfaces, and task-relevant functional parts rather than treating assets as static visual meshes.Automated annotations combine geometric structure and semantic priors with a CoT-based URDF generator.
- Scene construction: Users can load backgrounds through JSON configurations or GLB binaries and combine object and room libraries to construct new digital-cousin environments.The scene generator also supports automatic JSON scene-configuration generation from high-level room descriptions.
- Data-generation modes: The platform provides interactive asset-centric generation, tabletop digital-cousin randomization, and scene-scale mobile manipulation as complementary operating modes.These modes cover object interaction, varied tabletop configurations, and navigation before upper-limb manipulation in household layouts.
- User interface: A unified interface accepts images and text prompts, displays reconstruction checks, monitors execution, and supports human verification during asset, scene, and trajectory generation.The pipeline pre-collects over 1,000,000 dual-arm manipulation trajectories in RoboCousin.
4 Experiment
The experiments test whether RoboCousin’s automatically generated contacts, reconstructed assets, and tabletop cousins support real-world bimanual manipulation. Across the evaluations, generated contacts are competitive with curated ones, generated assets match or exceed RoboTwin assets, and cousins improve transfer on most tasks.
- Experimental design: The experiments address executable simulated grasps, sim-to-real transfer from reconstructed assets, and robustness from multiple semantically consistent tabletop cousins.The evaluation uses contact-point, generated-asset, and cousin-training comparisons.
- Automatic contact-point generation: The automatic contact-point pipeline achieves a 76% success rate versus 72% for original annotations across 100 sampled trajectories.The comparison keeps the asset, robot configuration, and evaluation protocol unchanged while testing a simplified grasp-and-lift task.
- Sim-to-real generated assets: RoboCousin assets match or outperform RoboTwin 2.0 assets on all four real-world tasks.Success rises from 40% to 80% on Pick_Up_Bottle, remains 75% on Place_Microphone_in_Basket, rises from 30% to 40% on Place_Mouse_Pad, and rises from 50% to 65% on Place_Marker_Box.
- Tabletop digital cousins: Tabletop cousin training improves transfer on three of four tasks and remains competitive with single-scene twin training on the remaining task.Cousins improve Pick_Up_Eyeglass_Case from 0% to 55%, Adjust_Bottle from 50% to 70%, and Place_Mouse_Pad from 20% to 25%, while scoring 35% versus 40% for Twin on Place_Spray_Bottle_in_Basket.
5 Related Work
Prior work provides physics simulation, demonstration generation, asset construction, domain randomization, and digital-cousin methods, but generally leaves user-provided observations separate from interaction-ready manipulation data. RoboCousin targets this bottleneck by connecting extensible asset generation, semantic scene variation, and automated expert trajectories for bimanual manipulation.
- Simulation infrastructure: Simulation platforms such as SAPIEN, RLBench, ManiSkill2, ManiSkill3, Habitat 2.0, and BEHAVIOR-1K provide physically grounded interaction, standardized tasks, diversity, or household-scale environments.These systems establish infrastructure for large-scale robot learning across articulated objects, heterogeneous embodiments, and realistic scenes.
- Demonstration generation: MimicGen, DexMimicGen, RoboTwin, and RoboTwin 2.0 reduce demonstration costs through adaptation, bimanual generation, digital twins, and annotated asset libraries.Their released benchmarks remain generally centered on curated assets and predefined scene or task templates.
- Generative simulation construction: Generative systems automate task programs, long-horizon reasoning, asset and task construction, or propose–generate–learn loops, while asset methods improve visual coverage or physical grounding.Examples include GenSim, GenSim2, Gen2Sim, RoboGen, Objaverse, EmbodiedGen, RialTo, and DreMa.
- RoboCousin’s distinction: RoboCousin connects visual observations to reusable assets by producing visual and collision representations, semantic and physical attributes, grasp-contact candidates, and functional asset roles.The same pipeline handles manipulable objects and background components for expert trajectory collection.
- Digital cousins: Unlike retrieval-only cousin pipelines, RoboCousin can generate new interaction-ready objects and vary compatible objects, backgrounds, and layouts while preserving task-relevant semantics.This extensible cousin representation couples semantic scene variation with automated expert trajectories for real-world bimanual manipulation.
6 Conclusion
RoboCousin links real-to-sim asset generation, interaction annotation, cousin construction, and expert trajectory collection into an extensible bimanual manipulation data pipeline. Experiments indicate that this extensibility yields useful real-robot supervision rather than only increasing synthetic-data volume.
- Platform contribution: RoboCousin turns user-provided observations into reusable data by combining asset generation, interaction annotation, digital-cousin construction, and expert trajectory collection.Its cousin generation varies assets, layouts, backgrounds, and language instructions while preserving affordances and spatial relations.
- Scale and scope: The framework supports tabletop and room-level environments, including base movement beyond a fixed workspace.RoboCousin-OBD contains more than 3,000 annotated object instances and 50 background environments, and the pipeline generates over one million expert trajectories.
- Empirical findings: Automatically generated contacts are comparable to curated annotations, generated assets match or outperform RoboTwin 2.0 assets, and tabletop cousins improve transfer on selected tasks.The conclusion also reports competitiveness with single-scene twin training on the longer spray-bottle placement task.
- Future work: Future work will broaden evaluation across tasks, environments, and robot embodiments while improving physical calibration and task-aware cousin generation.The stated targets include contact-sensitive and long-horizon manipulation.