Source-linked AI summary
Openpi Comet: Competition Solution For 2025 BEHAVIOR Challenge
Junjie Bai, Yu-Wei Chao, Qizhi Chen, Jinwei Gu, Moo Jin Kim, Zhaoshuo Li, Xuan Li, Tsung-Yi Lin, Ming-Yu Liu, Nic Ma, Kaichun Mo, Delin Qu, Shangkun Sun, Hongchi Xia, Fangyin Wei, Xiaohui Zeng
TL;DR
Reliable long-horizon execution remains difficult for VLA systems because they must chain interdependent skills while limiting accumulated errors. This report adapts π0.5 through heterogeneous pre-training and post-training, then reports competition and validation performance on BEHAVIOR. The authors also identify low RFT sampling efficiency as a remaining limitation.
Problem
Long-horizon tasks require reliable transitions between interdependent skills, but subtask policies do not resolve skill chaining or error accumulation.
Source: §1 · p. 1Method
The report adapts π0.5 with heterogeneous robot-data pre-training and rejection-sampling fine-tuning using successful policy rollouts from perturbed initial poses.
Source: §2 · p. 2, §4.3 · p. 4Results
The solution completed 22 of 50 tasks with a competition Q-score of 0.2514 and later achieved a validation Q-score of 0.3453, substantially outperforming previous results.
Source: §1 · p. 1Takeaways & Limitations
Scaling pre-training across more numerous and diverse BEHAVIOR tasks generates and unlocks success on rare compositional activities, while RFT avoids online-RL infrastructure issues and yields a performance boost.
Source: §5 · p. 7Despite RFT's effectiveness, sampling efficiency remains low, and the reported Q-score is still far from perfect.
Source: §5 · p. 7
Abstract
from arXiv · showhide
The 2025 BEHAVIOR Challenge is designed to rigorously track progress toward solving long-horizon tasks by physical agents in simulated environments. BEHAVIOR-1K focuses on everyday household tasks that people most want robots to assist with and these tasks introduce long-horizon mobile manipulation challenges in realistic settings, bridging the gap between current research and real-world, human-centric applications. This report presents our solution to the 2025 BEHAVIOR Challenge in a very close 2nd place and substantially outperforms the rest of the submissions. Building on $\pi_{0.5}$, we focus on systematically building our solution by studying the effects of training techniques and data. Through careful ablation studies, we reveal the scaling benefits in both the pre-training and post-training phases, leading to a validation Q-score of 0.345, significantly surpassing previous state-of-the-art performance. We summarize our practical lessons and design recommendations that we hope will provide actionable insights for the broader embodied AI community when adapting powerful foundation models to complex embodied scenarios. Project page: https://github.com/mli0603/openpi-comet
1 Introduction
Long-horizon tasks challenge VLA systems because reliable execution requires chaining interdependent skills while controlling error accumulation, a problem not resolved by common subtask policies. The BEHAVIOR Challenge tests this capability across realistic household tasks, and the report presents a π0.5-based solution with competition and validation results.
1 Introduction
- Long-horizon tasks require reliable transitions between interdependent subtasks, yet subtask policies do not solve skill chaining or prevent error accumulation.Online adaptation and modular approaches to skill chaining are often incompatible with large-scale offline, end-to-end VLA training.Source: §1 · p. 1
- The BEHAVIOR Challenge evaluates 50 household tasks requiring multi-step reasoning, precise manipulation, and coordinated navigation, with 10,000 expert demonstrations and a standardized protocol.The benchmark emphasizes generalization, control robustness, and error tolerance.Source: §1 · p. 1
- 22 of 50 tasks completed with a competition Q-score of 0.2514; the refined post-challenge method reached a validation Q-score of 0.3453, substantially outperforming previous results.The challenge score is reported in relation to Table 1; the higher score is for validation after the challenge.Source: §1 · p. 1, Table 1 · p. 2
2 Architecture
The system uses π0.5 as its base policy and combines heterogeneous pre-training data with iterative rejection-sampling fine-tuning. The post-training procedure adds successful rollouts from perturbed initial poses to the training data.
2 Architecture
- π0.5 provides the end-to-end VLA backbone, pretrained on heterogeneous data across embodiments, environments, and tasks, including challenge demonstrations and additional planner and offline-RL trajectories.The visual and language encoders feed a shared representation that conditions a transformer-style action expert.Source: §2 · p. 2, Figure 1 · p. 2
- RFT perturbs initial poses, rolls out the pretrained policy, and retains successful episodes as additional training data.The procedure begins from the official human demonstrations and iteratively expands the data with successful episodes.Source: §2 · p. 2, Figure 1 · p. 2
3 Dataset
The dataset combines expert demonstrations with planner and offline-RL trajectories for a 50-task challenge, and its skill mix and task complexity vary substantially. Evaluation across all tasks was not completed before the submission deadline.
3 Dataset
- The challenge provides 10,000 teleoperated demonstrations across 50 tasks, while the solution adds approximately 3.6K planner and offline-RL trajectories.The official demonstrations exceed 1,200 hours; planner trajectories provide precise, low-noise manipulation and RL rollouts broaden behavioral variability.Source: §3 · pp. 2–3
- Evaluation across all tasks was not completed by the submission deadline.This limits the reported evaluation coverage to a subset of tasks.Source: §3 · p. 2
- Move-to and pick-up-from account for roughly 33.3% and 24.4% of dataset frames, while many tasks span several hundred frames and 5–10 skills, with some exceeding 12.Tasks 48 and 49 are described as harder cases with long horizons and rich mixtures of navigation and manipulation skills.Source: §3 · p. 3, Figure 2 · p. 3
- Tasks with average trajectories below 250 frames are treated as relatively simple for system bring-up, while longer tasks test extended credit assignment and frequent skill switching.The per-task statistics are summarized in Figure 2(b).Source: §3 · p. 3, Figure 2 · p. 3
4 Experiment
The experiments examine implementation, pre-training coverage, rejection-sampling post-training, and training and inference ablations. Broader pre-training coverage and post-training improve performance, while ablations identify design choices that affect control and perception.
4.1 Implementation details
- Experiments use π0.5 in JAX, with 50k steps for multi-task pre-training and reduced learning rates for SFT and RFT fine-tuning.Evaluation is parallelized across 10 test instances to mitigate the simulator’s slow rate.Source: §4.1 · p. 3
4.2 Pre-training
- 0.192 validation Q-score is reached after pre-training, while broader task coverage produces successful rollouts on more tasks than single-task pre-training.pt1 succeeds on only 2 tasks; pt50 includes all 50 challenge tasks, including rare, highly compositional activities.Source: §4.2 · p. 4, Figure 3 · p. 4
4.3 Post-training
- 0.345 validation Q-score across 50 tasks is achieved after post-training, with a 15% validation success rate.The challenge-time post-training score was 0.224; the later 0.345 result used two model checkpoints.Source: §4.3 · p. 4, Figure 4 · p. 5
4.4 Ablations
- Receding Horizon control improves performance, whereas Temporal Ensemble and Receding Temporal yield near-zero success rates.The Receding Horizon scheme executes the predicted action segment before re-planning after manipulation.Source: §4.4 · p. 5
- A receding horizon of 32 gives the best results; excessively long horizons compromise stable control despite the benefits of moderate horizons.The reported relationship between prediction length and downstream performance is non-monotonic.Source: §4.4 · p. 5
- Reconstructed point clouds outperform depth maps, but their improvement over RGB-only inputs is limited and comes with considerable computational and latency overhead.The reported benefits of explicit 3D geometry are not consistently large enough to justify the increased system complexity.Source: §4.4 · pp. 5, 7
- Increasing head and wrist image resolution from 224 by 224 pixels to higher resolution more than doubles the success rate.The authors link the gains to more precise visual cues in visually complex environments.Source: §4.4 · p. 7
- Relative-action parameterization, removing proprioceptive state inputs, and 15 Hz action subsampling worsen results, while manipulation-focused skill reweighting yields no measurable improvement.The authors attribute the first two declines to the value of absolute action anchoring and explicit system observability, and the subsampling decline to reduced temporal precision.Source: §4.4 · p. 7
5 Conclusion
The conclusion reports that broader BEHAVIOR pre-training task coverage unlocks success on rare compositional activities and that RFT offers a post-training alternative to online RL. It also notes that the Q-score remains imperfect and RFT sampling efficiency is low.
5 Conclusion
- Scaling pre-training over more numerous and diverse BEHAVIOR tasks significantly generates and unlocks success on rare, compositional activities.Source: §5 · p. 7
- RFT avoids online RL infrastructure issues while providing trackable targets and a significant performance boost.Source: §5 · p. 7
- The Q-score remains far from perfect and RFT sampling efficiency is too low; DAgger, on-policy distillation, and positive-and-negative-reward RL are proposed directions.Source: §5 · p. 7
- The authors expect structured long-horizon reasoning, richer post-training objectives, and better curriculum design to further close the synthetic-benchmark-to-real-world deployment gap.Source: §5 · p. 7
6 Authors
The authors section lists core contributors and additional contributors; an asterisk identifies first authors whose ordering is random.
6 Authors
- Core contributors are Delin Qu, Qizhi Chen, Shangkun Sun, Zhaoshuo Li, Yu-Wei Chao, Xiaohui Zeng, Xuan Li, Junjie Bai, Tsung-Yi Lin, and Ming-Yu Liu.Source: §6 · p. 7
- Additional contributors are Kaichun Mo, Jinwei Gu, Moo Jin Kim, Fangyin Wei, Hongchi Xia, and Nic Ma.Source: §6 · p. 7
- The asterisk marks first authors listed in random order.Source: §6 · p. 7