Source-linked AI summary
Behavior-Skill: A Fine-Grained Benchmark for Evaluating Vision-Language-Action Policies in Long-Horizon Tasks
Chunyun Ma, Lun Luo, Xingjian Luo, Xiexing Feng, Hang Zhang, Wei Liu, Feng Qiao, Yaonan Wang, Huimin Lu, Xieyuanli Chen
TL;DR
Reliable long-horizon mobile manipulation remains difficult, while conventional task evaluation can hide intermediate skill failures. Behavior-Skill addresses this with constituent-skill data and independent evaluation, finding that skill execution remains limited and failures concentrate in bottleneck skills.
Problem
Reliable long-horizon mobile manipulation remains difficult, and conventional task evaluation can hide intermediate skill failures.
Method
Behavior-Skill establishes constituent skills as units for learning and evaluation, combining a skill dataset, restored intermediate states, skill success goals, and independent capability-oriented metrics.
Results
Failures concentrate in a limited set of semantic skill categories, while Skill-setting TSCR reaches only 48.4% for π0.5 and 42.5% for GR00T N1.7.
Takeaways & Limitations
Behavior-Skill complements full-task evaluation by exposing intermediate capability profiles, showing that reliable skill execution remains a key challenge.
Takeaways & Limitations
Behavior-Skill does not cover task planning, automatic task decomposition, or sequential dependencies between constituent skills during execution.
Abstract
from arXiv · showhide
Reliable execution of long-horizon mobile manipulation tasks remains challenging because overall task success depends on the successful completion of multiple constituent skills. Existing benchmarks, however, still rely primarily on full-task rollouts and aggregate task-level metrics, making intermediate failures difficult to observe and analyze. We present Behavior-Skill, a benchmark that reformulates the learning and evaluation of long-horizon tasks around executable constituent skills. It contains 235,492 skill instances from 10,000 demonstrations across 50 household tasks and 34 semantic skill categories. Each instance pairs a skill instruction with an aligned observation-action segment, and is further associated with a restorable intermediate state and a skill success condition to enable independent evaluation under valid preconditions. We further introduce trajectory-level and skill-level metrics to characterize policy capability beyond aggregate task success. Extensive experiments across representative VLA policies including pi0.5 and GR00T on the complete 50-task benchmark show that failures are highly non-uniform across skills, with contact-rich manipulation skills forming persistent bottlenecks. These results demonstrate that Behavior-Skill complements full-task evaluation by exposing intermediate capability profiles for analyzing and improving long-horizon VLA policies. Behavior-Skill is publicly available at https://github.com/nubot-nudt/Behavior-Skill.
I. INTRODUCTION
Long-horizon mobile manipulation remains difficult because success requires reliable sequential execution of multiple constituent skills, while existing evaluation offers limited visibility into intermediate failures. Behavior-Skill addresses this gap with skill-centric data, independent evaluation, and capability-oriented metrics.
- Long-horizon mobile manipulation requires integrating perception, language understanding, planning, and manipulation over extended interaction horizons.
- Performance on challenging long-horizon benchmarks such as BEHAVIOR-1K remains considerably lower, indicating that reliable complex multi-stage execution is unsolved.
- A long-horizon task succeeds only when its constituent skills execute sequentially, because any skill failure interrupts task progress.
- Behavior-Skill establishes constituent skills as the basic unit for studying long-horizon VLA policies and provides a unified dataset and independent evaluation framework.
- Experiments across representative VLA backbones find that failures are concentrated in a few semantic categories, with contact-rich skills acting as bottlenecks.
- 235,492 skill instances span 34 semantic skill categories, while the framework measures skills under valid preconditions with capability-oriented metrics.
II. RELATED WORK
Related work advances VLA policies, fine-grained task representations, and challenging benchmarks, but constituent skills are not yet supported by a unified learning and independent evaluation framework.
- VLA models trained on large-scale robot demonstrations have improved generalization across tasks, environments, and robot embodiments.
- Fine-grained representations such as subgoals, executable programs, and process descriptions support planning, reasoning, and language-guided policy learning.
- Existing work primarily treats skills as supervision signals or planning abstractions rather than as units for unified learning and independent evaluation.
- Behavior-Skill fills this gap by systematically establishing constituent skills as a shared basis for learning and evaluation.
B. Evaluation of Long-Horizon Tasks
Conventional long-horizon benchmarks mainly evaluate complete task rollouts with aggregate metrics, limiting systematic comparison of constituent-skill capabilities. Behavior-Skill instead restores intermediate states and evaluates skills independently with complementary metrics.
- Representative long-horizon benchmarks predominantly evaluate policies through complete task rollouts using aggregate task-level metrics.
- Existing finer-grained analyses generally retain task-level protocols and do not independently execute constituent skills under valid preconditions.
- Behavior-Skill annotates constituent skills, restores valid intermediate states, and evaluates them independently of preceding trajectory outcomes.
- TSCR and STSR summarize execution from trajectory-level and semantic skill-type perspectives, respectively.
A. Skill Dataset
The Behavior-Skill dataset annotates each constituent skill with multimodal context describing its action, objects, spatial relations, execution context, and end-effector usage.
- Each skill annotation describes the action, involved objects, spatial relation, execution context, and end-effector usage.
- Original skill records provide structured fields including action labels, object identifiers, manipulated objects, skill types, and temporal intervals.
- Synchronized multi-view observations and robot actions augment the structured records with scene and manipulation-process information.
- Actions within each skill interval are aggregated into compact motion descriptions such as arm movement and gripper state.
- A multimodal model generates annotations, which are checked against reference annotations and manually reviewed when inconsistencies are flagged.
B. Independent Skill Evaluation
Behavior-Skill independently evaluates constituent skills from restored intermediate states under valid preconditions, using skill instructions and executable success conditions. This separates intermediate capability measurement from failures accumulated earlier in a task trajectory.
- Evaluation Unit: Each evaluation unit combines a simulator state, skill instruction, skill success condition, and maximum evaluation horizon.These components are denoted by S_i,j, L_i,j, G_i,j, and H_i,j, respectively.
- Intermediate State Restoration: Behavior-Skill evaluates each constituent skill from a restored intermediate state while preserving the original task context.The restored state initializes independent evaluation before skill execution.
- Skill Success Specification: Skill success conditions are manually defined as BDDL goals capturing only the current constituent skill’s completion criterion.The construction preserves the original object scope and logical representation.
- Skill Success Specification: Skill-specific predicate extensions represent intermediate behaviors such as navigation, grasping, and handover when terminal task states are insufficient.Most manipulation skills use existing BDDL predicates for placement, articulated-object states, spatial relations, or object activation.
- Evaluation Procedure: The evaluation procedure restores the simulator snapshot, injects the skill goal, and executes the policy with the corresponding skill instruction.This enables independent execution under satisfied preconditions.
C. Evaluation Metrics
Behavior-Skill summarizes binary skill outcomes at complementary trajectory and semantic-category levels. TSCR measures completion within demonstrations, while STSR aggregates success for each semantic skill category.
- Task Skill Completion Rate (TSCR): Task Skill Completion Rate (TSCR) measures how completely a policy executes the constituent skills required by each demonstration.It is computed from binary success outcomes for the skills in an execution trajectory.
- Task Skill Completion Rate (TSCR): The benchmark reports overall performance as the average TSCR across all evaluated demonstrations.N denotes the total number of evaluated demonstrations.
- Skill-Type Success Rate (STSR): Skill-Type Success Rate (STSR) measures execution success for each semantic skill category.It aggregates outcomes across skills sharing the same category rather than within one task trajectory.
- Complementary Metrics: Together, TSCR and STSR summarize policy performance from trajectory-level and semantic skill-type perspectives.STSR uses the semantic category label and an indicator function to aggregate relevant skill outcomes.
A. Experimental Setup
Experiments evaluate π0.5 and GR00T on the complete 50-task benchmark using matched Task and Skill policy variants. The variants share training and evaluation conditions except for their language conditioning.
- Evaluation Settings: The primary evaluation uses the complete 50-task Behavior-Skill benchmark with π0.5 and GR00T N1.7.Ten demonstrations per task are reserved for evaluation, yielding 500 evaluation demonstrations.
- Evaluation Settings: For each task, ten demonstrations form the evaluation set and the remaining demonstrations are used for training.All skills in the selected evaluation demonstrations are included in evaluation.
- Controlled Comparison: Task and Skill variants use the same demonstrations, architecture, observation and action representations, optimization objective, and training schedule.They differ only in language condition: task prompts versus skill prompts.
- Evaluation Protocol: Independent skill evaluation executes each skill from a restored intermediate state with satisfied preconditions and shared success conditions and horizons.Task denotes task-prompt conditioning, while Skill denotes skill-prompt conditioning.
B. Overall Skill Capability
On the complete 50-task benchmark, constituent-skill capability remains limited, with fewer than half of skills completed on average and substantial variation across semantic categories. The two policies share similar low-performing bottlenecks despite differing overall performance.
- 48.4% and 42.5% TSCRs for π0.5 and GR00T N1.7, respectively, show fewer than half of constituent skills completed under Skill training.These results indicate that skill execution remains a major limitation for long-horizon task execution.
- Skills such as Sweep Surface, Place Under, Hold, Spray, Release, and Move To achieve consistently high success rates.
- Open Lid, Close Door, Close Lid, Pick Up From, and Pour remain among the lowest-performing skill types.
- The two policies exhibit remarkably similar capability profiles, suggesting shared bottlenecks across architectures.
- Skill training raises TSCR from 42.4% to 48.4% for π0.5 and from 36.9% to 42.5% for GR00T N1.7.Even the best-performing setting remains below 50% average TSCR, while several interaction skills remain rarely completed.
C. Comparison with Conventional Full-Task Evaluation
Behavior-Skill exposes intermediate skill outcomes that conventional full-task evaluation leaves unobserved after failures. It also reveals substantial task- and context-dependent variation in skill completion and reliability.
- C. Comparison with Conventional Full-Task Evaluation: Intermediate failures prevent later constituent skills from being executed, leaving part of the task unevaluated under conventional rollouts.In Make Microwave Popcorn, only the first five of eight constituent skills are reached.
- C. Comparison with Conventional Full-Task Evaluation: 0% QScore coexists with 80.0% TSCR for Wash a Baseball Cap and 69.0% TSCR for Cook Hotdogs.Make Microwave Popcorn achieves 10.0% QScore with 75.0% TSCR, showing that task-level outcomes and skill completion can differ substantially.
- D. Task-Dependent Skill Capability: TSCR ranges from 36.3% to 65.3% under Task and from 43.8% to 77.2% under Skill across the 12-task subset.
- D. Task-Dependent Skill Capability: Open Door success ranges from 9.0% in Make Microwave Popcorn to 78.0% in Cook Hot Dogs and Wash a Baseball Cap.
- D. Task-Dependent Skill Capability: Place In success ranges from 15.2% in Putting Shoes on Rack to 88.0% in Cook Hot Dogs, with comparable task-dependent variation under Skill.Object geometry, target relation, and surrounding scene context strongly affect execution reliability.
V. CONCLUSION
Behavior-Skill makes constituent skills fundamental units for long-horizon learning and evaluation through skill data, intermediate-state restoration, independent evaluation, and capability-oriented metrics. It complements full-task evaluation but does not cover planning, automatic decomposition, or sequential dependencies.
- Behavior-Skill provides a skill dataset, intermediate-state restoration, independent skill evaluation, and skill success goals for complex long-horizon tasks.
- The benchmark enables detailed investigation of constituent-skill execution through capability-oriented metrics.
- Behavior-Skill does not cover task planning, automatic task decomposition, or sequential dependencies between constituent skills during execution.Future work will extend it toward planning-aware and sequential skill evaluation.