Source-linked AI summary
What Actually Matters? A Paired Evaluation of Inference Settings for $\pi_{0.5}$ on BEHAVIOR-1K
Ruben Hammele, Samira Huber, Mohammad Nazeri, Amirreza Payandeh, Alexandyr T. Card, Aaron Nguyen, Anuj Pokhrel, Daeun Song, Xuesu Xiao, Sören Pirk
TL;DR
How inference and data-collection strategies generalize from smaller-scale evaluations to diverse household tasks remains an open question. The authors systematically evaluate proposed inference and data-collection strategies with a fine-tuned π0.5 model across 4 instances of all 100 BEHAVIOR-1K tasks. Across all 100 BEHAVIOR-1K tasks, strategies may not scale uniformly across task complexity: shorter action chunks significantly reduced Q-score, and playback speed of 1.3 improved performance.
Problem
How inference and data-collection strategies generalize from smaller-scale evaluations to diverse household tasks remains an open question.
Source: §1 · p. 1Method
The authors systematically evaluate proposed inference and data-collection strategies with a fine-tuned π0.5 model across 4 instances of all 100 BEHAVIOR-1K tasks.
Source: §1 · p. 1Results
Across all 100 BEHAVIOR-1K tasks, strategies may not scale uniformly across task complexity: shorter action chunks significantly reduced Q-score, and playback speed of 1.3 improved performance.
Source: §4 · p. 2Takeaways & Limitations
Fewer denoising steps and disabling overlap inpainting maintained performance, while snapshot restarts reduced simulation cost but not success and required restoring stage belief.
Source: §4 · p. 2
Abstract
from arXiv · showhide
Vision-Language-Action (VLA) models offer a promising paradigm for general-purpose household robotics. However, inference and data-collection strategies that have been deployed in smaller-scale settings do not necessarily generalize to large-scale, diverse task suites. To address this gap, we conduct a systematic, paired evaluation of previously proposed inference and data-collection strategies across all 100 tasks. Our evaluation uses a fine-tuned $\pi_{0.5}$ model based on the 2025 BEHAVIOR Challenge winner, which achieves a Q-score of 0.39 on the public test set. These results show that proposed strategies do not transfer uniformly to new environments and tasks. Project page, videos, and detailed analysis: https://rubenhammele.github.io/behavior2026
1 Introduction
Whether previously proposed inference and data-collection strategies generalize across diverse household tasks remains unclear. The paper evaluates these strategies with a fine-tuned π0.5 model across all 100 BEHAVIOR-1K tasks.
1 Introduction
- The generalization of inference and data-collection strategies across diverse household tasks remains an open question.Prior evidence includes garment folding and a single real-world manufacturing task.Source: §1 · p. 1
- The study evaluates proposed inference and data-collection strategies with a fine-tuned π0.5 model across 4 instances of all 100 BEHAVIOR-1K tasks.The reported findings include reduced Q-score with shorter action chunks, benefits from 1.3× playback speed, and lower simulation-step consumption with snapshot restarts.Source: §1 · p. 1
2 Experimental Setup
The policy is fine-tuned on BEHAVIOR-1K and further trained using collected rollouts mixed with demonstrations. Evaluation covers public test instances under official constraints and uses paired score differences with task-clustered bootstrap confidence intervals.
2 Experimental Setup
- The policy starts from the pre-trained π0.5 checkpoint, is fully fine-tuned on BEHAVIOR-1K, then continues training on collected partial successes mixed with demonstrations.Training frames carry binary advantage labels for RECAP-style conditioning.Source: §2 · p. 1
- Evaluation covers all 100 BEHAVIOR-1K tasks on public test instances under official constraints.Each cell uses the mean of two independent baseline runs as its control.Source: §2 · p. 2
- Paired score differences (∆Q) are assessed with task-clustered bootstrap 95% confidence intervals, with effects classified as helpful or harmful when the interval excludes zero.Source: §2 · p. 2
3 Experiments & Results
Inference ablations show that action-chunk length and playback speed affect performance, while fewer denoising steps and disabling overlap inpainting show no detectable change. Data-collection ablations find lower simulation cost but no significant success-per-step improvement, and identify policy stage-belief restoration as important.
3.1 Inference Ablations
- Halving the executed action prefix from 26 to 13 substantially reduced Q-score and success rate while increasing episode length.The inherited 1.3× playback speed-up outperformed standard-speed execution; 2× leaned worse without reaching significance.Source: §3.1 · p. 2
- Reducing denoising from 20 steps to 10 or one, and disabling overlap inpainting, produced no detectable performance change in Q-score.Increasing advantage guidance to β = 2 was worse, while β = 4 matched β = 1.Source: §3.1 · p. 2
3.2 Data Collection
- Restarting before the first drop in predicted task progress earned more progress than restarting at the last progress point.The section frames restart strategies as ways to reduce collection costs and improve recovery from failed episodes.Source: §3.2 · p. 2
- Restarts reduced simulation cost but did not significantly improve successes per simulation step.Omitting the policy’s predicted stage during restoration reduced Q-score and task progress, supporting preservation of policy-side state.Source: §3.2 · p. 2
4 Conclusion
Across all 100 BEHAVIOR-1K tasks, the paper finds that inference and data-collection strategies do not scale uniformly with task complexity. Shorter action chunks reduced Q-score, while 1.3× playback improved performance; snapshot restarts reduced simulation cost but not success and required restoring stage belief.
4 Conclusion
- Across all 100 BEHAVIOR-1K tasks, inference and data-collection strategies did not necessarily scale uniformly across task complexity.Shorter action chunks reduced Q-score, 1.3× playback improved performance, and snapshot restarts reduced simulation cost but not success.Source: §4 · p. 2