PaperlayerView exact revision on arXiv

Source-linked AI summary

ManiUnit: A Manipulation Skill Dataset and Benchmark for Long-Horizon Tasks

Guoting Wei, Dawei Yan, Xia Yuan, Gengming Zhang, Yelin He, Guodong Du, Jiaquan Ye, Heng Zhang, Xinming Wei, Xianbiao Qi, Chunxia Zhao, Haokui Zhang, Rong Xiao

arXiv:2610.12089v1cs.RO

TL;DR

Long-horizon mobile manipulation requires navigating multi-room environments and executing successive skills under one instruction, but skill selection, inherited starting states, and task-level evaluation pose challenges. ManiUnit provides instructed manipulation segments and independent skill tests with controlled starting-state variations. Its evaluations expose per-skill differences and starting-state sensitivity, while ManiUnit-trained skills improve local manipulation success on two activities.

  • Problem

    Long-horizon mobile manipulation requires successive skills under one instruction, while similar observations can obscure skill selection, inherited robot states can differ from demonstrated starts, and task-level metrics limit skill-specific diagnosis.

    Source: §1 · p. 2

  • Method

    ManiUnit segments demonstrations from 50 BEHAVIOR-1K activities into instructed operations and evaluates skills from restored intermediate states using local goals and controlled base or joint perturbations.

    Source: §1 · pp. 2–3

  • Results

    Representative VLA policies show hidden per-skill differences, and joint perturbations reduce full-benchmark success by approximately 56% relative to demonstrated starts.

    Source: §1 · p. 3

  • Takeaways & Limitations

    On two activities, coordinating task and ManiUnit-trained skill policies through a planner achieves 18.0% full-task success versus 4.0% for the task policy alone.

    Source: §1 · p. 3

    Reliable skill execution remains challenging under perturbed starts, and complete-task performance depends on coordination decisions about subtask instructions and policy switching.

    Source: §5 · p. 10

Abstract

from arXiv · show

Long-horizon mobile manipulation requires a robot to navigate multi-room environments and execute a sequence of manipulation skills under a single natural language instruction. Learning and evaluating these skills present three challenges: similar observations under a fixed task instruction may make skill selection ambiguous; even when a preceding skill succeeds, the robot state inherited by the next skill may deviate from its demonstrated starting states and affect execution; and task-level metrics hinder skill-specific diagnosis, while early failures leave later skills untested. We therefore introduce ManiUnit, a manipulation skill dataset and benchmark built from 50 BEHAVIOR-1K activities. Its dataset contains 137,899 segments across 21 skill types and 417 subtasks, and its benchmark contains 1,260 test instances. Correspondingly, ManiUnit pairs each segment with an explicit subtask instruction; measures sensitivity to perturbations of the robot's starting base position or joint configuration; and restores intermediate simulator states and defines local success conditions so that each skill can be evaluated without executing preceding stages. Evaluations of representative vision-language-action (VLA) policies show that similar aggregate scores can hide substantial per-skill differences. The tested starting-state perturbations also degrade execution: on the full benchmark, joint perturbations reduce success rates by approximately 56% relative to those from demonstrated starting states. On two long-horizon activities, a skill policy trained on ManiUnit segments achieves 78.7% local manipulation success, compared with 49.3% for a task policy trained on complete demonstrations. The trained skills further support complete-task execution on these activities, as coordinating the task and skill policies through a planner raises full-task success from 4.0% to 18.0%.

1 INTRODUCTION

ManiUnit addresses skill ambiguity, starting-state mismatch, and limited skill-specific diagnosis by making individual manipulation operations units of training and evaluation. Its evaluations reveal both substantial differences across skills and performance gains from ManiUnit-trained skills on two long-horizon activities.

  • ManiUnit segments demonstrations from 50 BEHAVIOR-1K activities into 137,899 segments, each paired with explicit subtask instructions, and organizes the dataset across 21 skill types and 417 subtasks.The benchmark contains 1,260 test instances from restored intermediate states with local success conditions; each skill is evaluated under demonstrated-state, base-position-perturbation, and joint-configuration-perturbation conditions.Source: §1 · pp. 2–3
  • 18.0% full-task success for the combined system versus 4.0% for the task policy alone on the same activities.The planner coordinates task and skill policies, supplying the appropriate subtask instructions.Source: §1 · p. 3
  • 78.7% local manipulation success for the ManiUnit-trained skill policy versus 49.3% for the task policy trained on complete demonstrations.The policies were initialized from the same pretrained weights and compared on Turning On Radio and Picking Up Trash.Source: §1 · p. 3
  • Joint perturbations reduce success rates by approximately 56% relative to demonstrated starting states on the full benchmark.Source: §1 · p. 3
  • Similar aggregate scores among representative VLA policies can conceal substantial per-skill performance differences.Source: §1 · p. 3

2 MANIUNIT DATASET AND BENCHMARK

ManiUnit reorganizes BEHAVIOR-1K demonstrations into instructed manipulation segments and builds independent skill tests from restored operation starts. Its construction combines curated segment boundaries and instructions with local goals and controlled starting-state variants.

2.1 OVERVIEW

  • The overview organizes ManiUnit-Full into 21 skill types, 417 subtasks, and 137,899 segments, with a 10,588-segment Lite subset across 7 skill types.A skill type names an operation category, a subtask specifies the operation, objects, and spatial context, and a segment records one demonstrated execution.Source: §2.1 · p. 3

2.2 DATASET CONSTRUCTION

  • Segment construction extracts annotated manipulation intervals, merges supporting motions into complete operations, refines boundaries, and manually reviews explicit subtask instructions against videos.Boundary refinement uses arm deviation from a lowered reference posture and gripper events to reduce surrounding navigation and inactivity while retaining grasp and release motions.Source: §2.2 · pp. 4–5

2.3 BENCHMARK CONSTRUCTION

  • Each benchmark instance pairs a restored operation-start state with its instruction and local goal, enabling independent tests under original, base-perturbed, and joint-perturbed states.Local goals encode operation-specific object states and relations; for Sweep, the goal concerns only the mud particles targeted by that segment. Pick Up variants also test target position, color, or a nearby distractor while holding other specified factors fixed.Source: §2.3 · pp. 5–6

3 POLICY COMPOSITION FOR LONG-HORIZON TASKS

ManiUnit separates manipulation learning from task-level control: a task policy can handle navigation and stage progression while a skill policy executes selected operations. A planner coordinates when to delegate and which subtask instruction to provide.

3 POLICY COMPOSITION FOR LONG-HORIZON TASKS

  • The task policy remains the default controller, with selected manipulation stages delegated to a skill policy.This composition separates manipulation execution from navigation and decisions about which operation comes next.Source: §3 · p. 6
  • A planner uses robot-state and visual histories to choose when to delegate control and which subtask instruction to supply.Delegation requires both stronger execution of the selected operation and an appropriate stage-specific instruction.Source: §3 · p. 6
  • Navigation and planner coordination remain distinct targets for future improvement alongside focused manipulation training and diagnosis.The paper identifies task-progress estimation, subtask selection, and policy switching as planner-design directions.Source: §3 · p. 7

4 EXPERIMENTS

The experiments evaluate representative VLA policies on per-skill performance and starting-state sensitivity, compare skill- and task-level training, and test planner-mediated complete-task execution.

4.1 EXPERIMENTAL SETUP

  • Policies include StarVLA-PI and StarVLA-GR00T on Full and StarVLA-Cosmos2 and π0.5 on Lite, with each model trained for five epochs.The evaluation covers 1,260 Full instances across 21 skill types and 186 Lite instances across seven skill types.Source: §4.1 · p. 7
  • Tests restore operation starts and use three conditions: Original, Base Perturbation, and Joint Perturbation; success is averaged equally across skill types and conditions.A trial succeeds when the local goal holds for ten consecutive frames within three times the subtask's mean demonstration duration.Source: §4.1 · pp. 7–8

4.2 SKILL-LEVEL EVALUATION

  • On ManiUnit-Full, StarVLA-PI and StarVLA-GR00T achieve 45.9% and 47.6% overall success; on Lite, π0.5 leads with 66.3%.Lite scores are 57.1% for StarVLA-GR00T, 54.7% for StarVLA-PI, and 45.3% for StarVLA-Cosmos2.Source: §4.2 · p. 8
  • Per-skill profiles differ despite similar aggregate scores: GR00T leads PI on Full Pick Up, 54.4% versus 44.4%, while PI leads on Lite Place, 81.9% versus 70.5%.On Lite Pour, GR00T scores 40.0% versus 12.4% for PI.Source: §4.2 · p. 8
  • On Full, Joint Perturbation reduces success by 55.9% for PI and 55.6% for GR00T; target-position changes reduce Lite Pick Up success by approximately 19%–72%.Target-position changes produce the largest drop among the tested scene variations.Source: Table 1 · p. 8, §4.2 · p. 8

4.3 SKILL-LEVEL VS. TASK-LEVEL LEARNING

  • 78.7% overall success for the skill policy versus 49.3% for the task policy; the skill policy also leads on all five subtasks, including Press radio, 80.0% versus 13.3%.The skill policy's success under Joint Perturbation is 76.0%, compared with 32.0% for the task policy.Source: Figure 4 · p. 9, §4.3 · p. 9

4.4 LONG-HORIZON TASK EXECUTION

  • 18.0% overall full-task success with the combined system versus 4.0% for the task policy alone; mean Q-score rises from 11.0% to 25.0%.The combined system uses a planner to coordinate task and skill policies, averaging results across Turning On Radio and Picking Up Trash.Source: Table 2 · p. 9, §4.4 · p. 9

5 CONCLUSION

The conclusion presents ManiUnit as a dataset and benchmark for training and evaluating manipulation skills independently within long-horizon tasks. The related work situates it among approaches to demonstration organization, skill evaluation, robustness, and hierarchical control.

5 CONCLUSION

  • ManiUnit organizes long-horizon operations as training and evaluation units, using demonstration segments, subtask instructions, restored states, and local goals.Controlled base-position and joint-configuration variations measure their effects on skill performance.Source: §5 · p. 9, A RELATED WORK · p. 13
  • On two activities, skill-level training improves local manipulation success, while policy composition improves full-task success.ManiUnit evaluations also support skill-level diagnosis and reveal policy sensitivity to robot starting states.Source: §5 · p. 10
  • Reliable skill execution remains challenging under perturbed starts, and complete-task performance depends on coordination decisions about instructions and policy switching.The conclusion identifies task-progress tracking, subtask selection, and switching times as directions for planner development.Source: §5 · p. 10

A RELATED WORK

  • Related work spans demonstration organization, skill evaluation and robustness, and hierarchical control; ManiUnit contributes instruction-paired segments and controlled starting-configuration variation.The cited approaches include cross-activity demonstration data and skill annotations, independent or staged evaluation, and intermediate policy guidance.Source: A RELATED WORK · pp. 12–13

B DATASET CONSTRUCTION AND CURATION

This appendix details ManiUnit’s skill inventory and the rules used to extract, merge, and subset its manipulation segments. It describes organizing data by skill type and subtask to support pooled or targeted selection.

B.2 SKILL SEGMENT EXTRACTION AND MERGING

  • The appendix describes extraction and merging rules for constructing manipulation segments from BEHAVIOR-1K annotations.For 50 source activities, the authors define 312 extraction rules and 48 merging rules; extraction uses object information and execution order where needed, and merging can combine supporting actions with their main operation.Source: B DATASET CONSTRUCTION AND CURATION · p. 13, §B.2 · pp. 13–14

B.1 SKILL INVENTORY AND COVERAGE

  • Across source activities, the 21 skill types include 13 found in multiple activities, with 37 exact subtask instructions shared across activities.Pick Up and Place together account for 73.8% of segments; Ignite and Wipe each come from a single activity. The Full benchmark assigns 20 anchors to each skill type.Source: §B.1 · p. 13
  • ManiUnit-Lite contains 10,588 segments across seven skill types, selected from eight high-success BEHAVIOR Challenge activities.Its benchmark subset retains 186 of the 1,260 Full test instances, covering 62 anchors under Original, Base Perturbation, and Joint Perturbation; these subsets reduce repeated-experiment cost.Source: §B.1 · p. 13

B.2 SKILL SEGMENT EXTRACTION AND MERGING

  • Merging rules combine supporting actions with a main operation, including actions separated by navigation when no other selected manipulation operation intervenes.For example, a push that prepares a game for grasping is merged with pickup; a handover may be merged with hanging while retaining intervening navigation.Source: §B.2 · p. 14

B.3 TEMPORAL BOUNDARY REFINEMENT

Temporal boundary refinement uses arm-posture departures from a fixed lowered-arm reference to identify manipulation intervals, with safeguards against trimming grasp and release motions.

B.3 TEMPORAL BOUNDARY REFINEMENT

  • The refinement monitors the first three joints of each arm and retains the span from the first qualifying departure run to the last.The reference is the mean of smoothed joint positions over 500 lowered-arm frames; runs must meet a threshold for a minimum duration, with defaults listed in Table 5.Source: §B.3 · pp. 14–15, Table 5 · p. 15
  • Gripper-event margins and a minimum retained-fraction rule limit excessive trimming; intervals without a qualifying arm run are retained for manual inspection.The procedure shortens 13,664 released segments, and selected videos are inspected to check that refined boundaries preserve the intended operation.Source: §B.3 · p. 15

B.4 QUALITY CONTROL AND DATA CONSISTENCY

Quality control checks segment traceability and the correspondence of sampled video, robot-state, and action sequences to source intervals.

B.4 QUALITY CONTROL AND DATA CONSISTENCY

  • Each exported segment records its source activity, episode, and frame boundaries before and after refinement, while clipped annotations retain source indices.Annotations are reindexed relative to the segment for traceability.Source: §B.4 · p. 15
  • The corpus check finds no mismatches between segment identifiers and instruction records.Sampled segments are checked for agreement among video frame counts, robot-state and action sequence lengths, and source-frame indices.Source: §B.4 · p. 15
  • The sampled checks confirm that inspected videos, states, and actions cover the same source-demonstration interval.This verifies temporal correspondence across the inspected modalities.Source: §B.4 · p. 15

C BENCHMARK CONSTRUCTION AND VALIDATION

The benchmark restores operation-start states from demonstrations, defines local completion conditions, and tests how starting-state and selected scene variations affect independent skill evaluation.

C.1 INITIAL STATE RESTORATION

  • Restoration replays scene history, recovers assisted-grasp constraints, and reduces residual motion before checking operation preconditions and that the local goal remains unmet.The default action-based generator uses 25 settling steps and 60 hold-action steps.Source: §C.1 · pp. 15–16

C.2 LOCAL GOAL DEFINITIONS AND OBJECT BINDING

  • Because full-activity goals do not specify intermediate completion, each subtask receives a local BDDL goal bound to its simulator objects and required states or relations.For chopping, the goal tracks the source object and resulting pieces; for sweeping, it targets particles removed by the demonstrated operation rather than the whole floor.Source: §C.2 · p. 16

C.3 ROBOT STATE PERTURBATIONS

  • Starting-state perturbations translate the base or change selected arm or torso joints while holding other configurations fixed; median joint change is 0.916 rad.The benchmark checks perturbed candidates for collisions, grasp disruption, and unintended changes before acceptance.Source: §C.3 · pp. 16–17

C.4 SKILL-SPECIFIC TEST EXTENSIONS

  • Targeted Pick Up tests vary position, color, or distractor presence while keeping the starting configuration, target identity, instruction, and local goal fixed.Position changes preserve arrangements of container contents and associated particles; the shared definitions also link test instances to training segments for selection and fine-tuning.Source: §C.4 · p. 17

C.5 TEST RECORDS AND VALIDATION

  • Each test record stores source episode and frame indices, instruction, local goal, and construction settings, and instances undergo automated checks and manual scene inspection.Instances failing either validation are corrected or replaced.Source: §C.5 · p. 17

D EXPERIMENTAL DETAILS AND ADDITIONAL RESULTS

This section reports training configurations, complete per-skill results, and supplementary-study details.

D.1 TRAINING CONFIGURATIONS

Policies are trained jointly across skill types in ManiUnit training sets, with configurations adapted to the evaluated models. Per-skill success rates are reported across three starting-state conditions.

D.1 TRAINING CONFIGURATIONS

  • Benchmark policies train for five epochs, with one policy jointly trained across skill types; Full evaluates StarVLA-PI and StarVLA-GR00T, while Lite also evaluates StarVLA-Cosmos2 and π0.5.Each prompt specifies the skill type and full subtask instruction.Source: §D.1 · p. 17
  • StarVLA-PI and StarVLA-GR00T use Qwen3.5-2B backbones and 30-action chunks, while Cosmos2 freezes its text encoder and VAE and trains its transformer with an action head.Cosmos2 receives the three camera views as one spatial mosaic; StarVLA joint targets are represented as offsets for selected arm and torso joints.Source: §D.1 · pp. 17–18

D.2 PER-SKILL SUCCESS RATES

  • Success rates are reported per skill for Original, Base Perturbation, and Joint Perturbation, averaging repeated trials by anchor and then equally weighting skill types.Overall success averages the three condition scores.Source: §D.2 · p. 18

D.3 PLANNER AND COMPLETE-TASK EVALUATION

The planner combines task and skill policies for complete-task evaluation, while the visual catalog organizes manipulation skills into interaction categories and pairs examples with their instructions.

D.3 PLANNER AND COMPLETE-TASK EVALUATION

  • The planner invokes the skill policy during manipulation when navigation and subtask predictions agree with the accepted instruction; otherwise, the task policy retains control.Robot-state histories support navigation classification, while scene features and video retrieval help identify the operation; confidence thresholds and a grace period govern switching.Source: §D.3 · pp. 18, 20
  • Complete-task evaluation compares the composed system with a task-policy baseline on Turning On Radio and Picking Up Trash, using success rate and final Q-score.Each activity has 10 test episodes repeated five times, and the two activities receive equal weight in overall results.Source: §D.3 · p. 20
  • Figure 5 shows successful rollouts in which the planner coordinates task-policy navigation and task control with ManiUnit-trained skill-policy manipulation.Colors indicate the active policy.Source: Figure 5 · p. 21

E VISUAL CATALOG OF MANIPULATION SKILLS

  • The visual catalog groups 21 skills as object handling, opening and closing, or activation and tool use; rows show five examples with their full subtask instructions.When all five examples share an instruction, it appears once below the images.Source: E VISUAL CATALOG OF MANIPULATION SKILLS · p. 22
Loading 2610.12089v1…