RH20T-P: A Primitive-Level Robotic Dataset Towards Composable Generalization Agents

• Out-of-distribution generalization remains difficult because imitation- and reinforcement-learning systems cannot feasibly cover all real-world tasks in training data.Source: §1 · p. 2
• VLM-based task planners decompose compounded tasks into primitive sequences, enabling novel skills to be composed from familiar skills.Source: §1 · p. 2
• Existing CGAs lack reliable primitive-level spatial information and sufficient dataset support for training specialized controllers or motion planners.Source: §1 · p. 2
• RH20T-P provides about 38k manually segmented clips across 67 tasks, while RA-P demonstrates feasibility and generalization, including novel skills, in real-world demonstrations.Source: §1 · p. 2

VLMs

• VLM research includes 2D and 3D visual-language understanding, with 3D-LLM introducing 3D semantics by rendering point clouds into 2D images.Source: §2 · pp. 2–3

VLMs as Task Planners

• VLM task planners can generate primitive skills, but PaLM-E has variable action granularity and GPT-4V-based systems are proprietary and inflexible for observations such as depth.Source: §2 · p. 3

• Existing robotic manipulation datasets often lack textual annotations or describe only whole tasks, while free-form clip descriptions are too coarse for low-level controllers.Source: §2 · p. 3

Primitive-level Robotic Manipulation Datasets

• RH20T-P responds to the absence of datasets with fine-grained primitive skills needed for composable generalization agents.Source: §2 · p. 3, Table 1 · p. 3

3.1 Preliminary of RH20T and Data Sampling

• RH20T-P samples visually reasoning-suitable RH20T tasks while retaining complex activities such as wiping tables and arranging chessboard pieces.Source: §3.1 · p. 3

3.2 Hierarchical and Scalable Primitive Skills

• Primitive skills are defined by robot-arm state changes and divided into motion-based and gripper-based categories rather than coarse free-form language.Source: §3.2 · p. 4
• Motion-based skills include spatial information such as destinations, directions, or trajectories, reducing ambiguity and granularity inconsistency for low-level controllers.Source: §3.2 · p. 4
• Hierarchical skills build specialized operations such as pull and press on the general move skill, supporting tailored motion planning and future expansion.Source: §3.2 · p. 4
• Primitive templates such as “pick the {attribute} {object}” leave object and attribute composition to annotation and planning instead of enumerating every combination.Source: §3.2 · p. 4

3.3 Hindsight Primitive-level Annotation

• Annotators segment episodes into clips with frame boundaries and primitive labels, then derive spatial information from 7-DoF teleoperation records and add inspected GPT-4V captions.Source: §3.3 · p. 5

4 Plan-execute CGA Paradigm

• The standardized CGA paradigm uses task decomposition, motion planning, and primitive-level controllers in an alternating plan-execute loop.Source: §4 · p. 5, §4.1 · p. 5

4.1 Overall Pipeline

• At each state, the task planner predicts a primitive, the motion planner supplies spatial information when needed, and the controller generates actions for the next state.Source: §4.1 · p. 5
• Training associates each state transition with an RH20T-P clip, using RGB observations, robot position, annotated primitives, and recorded 7-DoF controls.Source: §4.1 · p. 5

4.2 RA-P: A Baseline Implementation

• RA-P uses LLaVA as task planner and fine-tunes it with a robotic-manipulation instruction dataset derived from RH20T-P.Source: §4.2 · pp. 5–6
• RA-P uses Deformable DETR to localize destination coordinates, converts them to a calibrated 3D point, and activates motion planning through a special <pos> token.Source: §4.2 · p. 6
• Hard-coded controllers execute basic motions, while individually trained ACT policies handle object-interaction primitives including pick, push, pull, and press.Source: §4.2 · p. 6

5.1 Experimental Setup

• Eight unseen tasks test incremental generalization to novel scenes, novel objects or compositions, and novel skills, with higher levels including the preceding ones.None of the evaluation tasks, scenes, objects, or skills appears in RA-P’s training distribution.Source: §5.1 · pp. 6–7
• The evaluation measures whole-system execution success over 10 trials per task and separately records task-planning accuracy.Execution success includes task and motion planning, while motion-planning assessment is left for the execution phase.Source: §5.1 · p. 7, Table 2 · p. 7

5.2 Experimental Results

• ACT performs poorly even on basic tasks, whereas RA-P improves picking by decoupling motion planning from subsequent execution.Poor low-level control can limit agents that delegate motion planning to controllers, especially on composite tasks such as Stack Block.Source: §5.2 · p. 7
• RA-P achieves higher execution success across all three generalization levels than GPT-4V-based motion-planning agents, especially on novel skills.The authors attribute this to the designed primitive skills and corresponding spatial information in RH20T-P.Source: §5.2 · pp. 7–8

5.3 Qualitative Analysis and Discussion

• RA-P distinguishes a target garbage object from unrelated objects and successfully performs Throw Garbage, demonstrating robustness to object distractions.Figure 6 visualizes RA-P executions, including this distraction scenario.Source: §5.3 · p. 8, Figure 6 · p. 8
• Observed failures arise from low-level controller errors, DETR localization deviations under distractions, and VLM perception errors that cause incorrect positioning.These failure sources qualify the system’s otherwise robust qualitative behavior.Source: §5.3 · p. 8
• Data scaling significantly improves task planning and motion planning, but both remain far from saturation.The authors identify further method design and data accumulation as opportunities.Source: §5.3 · p. 8, Table 3 · p. 8
• Using a 7B language model as RA-P’s decision-making backend contributes approximately 8% of inference time, leaving room for larger models and asynchronous optimization.The deployed system’s inference timing is shown in Figure 7.Source: §5.3 · p. 8, Figure 7 · p. 8

• RH20T-P provides meticulously defined primitive skills and diverse forms of primitive-level spatial knowledge for robotic manipulation.Source: §6 · p. 8
• An agent built with the plan-execute CGA paradigm demonstrates feasibility and robust generalization in real-world robotic manipulation tasks.Source: §6 · p. 9
• Further data accumulation and limited computing resources constrain current systems, including the LLaVA-7B task planner and DETR motion planner.The authors propose more sophisticated planning systems and robust motion-planning strategies as future directions.Source: §6 · p. 9

A More Details about RA-P

• Chain-of-Thought inference first captions the scene, then reuses that description for subsequent decisions to reduce perceptual errors.The approach adds minor inference overhead and remains vulnerable to hallucinated descriptions and multiple-object interference.Source: A More Details about RA-P · p. 13
• RA-P’s deployment collects RGB images and robot-arm position, then returns observations to the agent through a communication module.Source: A More Details about RA-P · p. 14
• The planner selects predefined primitive skills, while motion-based choices invoke coordinate prediction and camera-calibrated 3D transformation.Source: A More Details about RA-P · p. 14
• Controllers map decisions into action sequences, execute them, and repeat planning until the planner issues a done decision.Source: A More Details about RA-P · p. 15

B GPT-4V Execution Setup

• RA-P uses LLaVA as its task planner, while GPT-4V can alternatively provide task planning or scene descriptions through in-context learning.Source: B GPT-4V Execution Setup · p. 15

C More Results

• RA-P trained on RH20T-P provides more reasonable spatial priors, whereas GPT-4V failures often result from insufficient localization and inaccurate target positions.Source: C More Results · p. 15

D All Tasks in RH20T-P

• Table 7 lists the tasks included in RH20T-P.Source: D All Tasks in RH20T-P · p. 15

E Potential Social Impact

• RH20T-P and RA-P demonstrate effectiveness and generalization, especially for novel physical skills, supporting future CGA development.Source: E Potential Social Impact · p. 15
• Violent elements such as knife use may create negative social impacts through related knowledge learned by robots, although the authors describe these impacts as controllable.Source: E Potential Social Impact · p. 16