PaperlayerView exact revision on arXiv

Source-linked AI summary

Task adaptation of Vision-Language-Action model: 1st Place Solution for the 2025 BEHAVIOR Challenge

Ilia Larchenko, Gleb Zarin, Akash Karnatak

arXiv:2512.06951v2cs.ROcs.AIcs.CVcs.LG

TL;DR

Long-horizon household tasks pose challenges including non-Markovian states and a clean training dataset without recovery demonstrations. The authors adapt Pi0.5 with architectural changes and practical inference optimizations, earning first place with a 26% q-score across 50 tasks. They caution that limited ablations leave individual component contributions unidentified.

  • Problem

    Long-horizon tasks include visually ambiguous non-Markovian states, while the clean demonstrations provide no recovery examples for out-of-distribution situations.

    Source: §4.2 · p. 5, §7.4.1 · p. 10

  • Method

    The approach combines Pi0.5 architectural changes, including correlated noise, mixed-layer attention, and stage tracking, with inference-time compression and correction rules.

    Source: §10 · p. 14

  • Results

    First place with 26% q-score across 50 tasks, with negligible difference between public and private evaluation.

    Source: §9.1 · p. 12, §10 · p. 14

  • Takeaways & Limitations

    Gripper correction rules for accidental closures contributed significantly to the final score, though such task-specific rules are not generalizable research contributions.

    Source: §10 · p. 14

    Resource constraints prevented ablations that could isolate which innovations matter or are redundant.

    Source: §10 · p. 14

Abstract

from arXiv · show

We present a vision-action policy that won 1st place in the 2025 BEHAVIOR Challenge - a large-scale benchmark featuring 50 diverse long-horizon household tasks in photo-realistic simulation, requiring bimanual manipulation, navigation, and context-aware decision making. Building on the Pi0.5 architecture, we introduce several innovations. Our primary contribution is correlated noise for flow matching, which improves training efficiency and enables correlation-aware inpainting for smooth action sequences. We also apply learnable mixed-layer attention and System 2 stage tracking for ambiguity resolution. Training employs multi-sample flow matching to reduce variance, while inference uses action compression and challenge-specific correction rules. Our approach achieves 26% q-score across all 50 tasks on both public and private leaderboards.

1 Introduction

The BEHAVIOR Challenge tests policies on diverse, long-horizon household tasks, where compounding errors, visually ambiguous states, and absent recovery demonstrations create distinct difficulties. The paper adapts Pi0.5 with correlated action noise, stage tracking, learnable attention, and practical corrections, while noting that limited ablations prevent attributing results to individual components.

1.1 The BEHAVIOR Challenge

  • The challenge evaluates one policy across 50 household activities in photo-realistic simulation, with long episodes, bimanual manipulation, and mobile navigation.Episodes average 6.6 minutes, and the longest tasks average 14 minutes; q-score combines success with partial credit for subtasks.Source: §1.1 · p. 1

1.2 Key Challenges

  • Long episodes compound prediction errors; visually similar states can require different actions, and successful-only demonstrations provide no recovery examples.The training data also contains multiple valid action sequences and different completion speeds for the same task.Source: §1.2 · p. 1

1.3 Our Approach

  • Correlated noise sampled from the empirical action covariance models temporal and cross-dimension structure, improving training efficiency and enabling principled inference-time inpainting.The paper sets β = 0.5 in the noise covariance mixture.Source: §1.3 · p. 2
  • System 2 predicts and votes on task stages to stabilize non-Markovian context, while failure-analysis correction rules target common errors such as accidental gripper closures.The stage estimate is fused with task embeddings and fed back to the model.Source: §1.3 · p. 2
  • It uses learnable mixed-layer attention over VLM layers, 15 flow-matching predictions per VLM pass during training to reduce gradient variance, and cubic-spline action compression for 1.3× faster execution at inference.It removes text processing and uses trainable task embeddings instead of the text prompt.Source: §1.3 · p. 2

1.4 A Note on Evidence

  • The report does not claim each component is necessary or optimal, and says limited resources prevented isolating their individual contributions.The authors describe many design choices as based on intuition or quick experiments rather than systematic evaluation.Source: §1.4 · p. 2

1.5 Code and Models

  • The authors make their training and inference code available and also share model weights, stating that these resources should allow anyone to reproduce their results.Source: §1.5 · p. 2

2 Related Work

  • For chunked execution, prior inpainting balances consistency and reactivity; this work uses learned action correlations to smooth transitions between prediction windows.Multi-task training also yielded recovery behaviors, such as grabbing fallen objects, absent from single-task demonstrations.Source: §2.3 · p. 3, §2.4 · p. 3

3 Method Overview

The method adapts Pi0.5 to predict action sequences from multi-camera RGB observations and robot state using flow matching. Its overview groups changes into model architecture, training procedures, and inference optimizations.

3.1 High-Level Architecture

  • Three camera views and robot state feed a VLM and a separate Action Expert, which predicts action sequences via flow matching.The camera views are head, left wrist, and right wrist.Source: §3.1 · p. 3, Figure 1 · p. 4

3.2 Our method description structure

  • Architectural changes include task embeddings, learnable mixed-layer attention, stage tracking, custom attention masks, and correlated noise.Task embeddings replace language processing; stage prediction supplies task-stage context.Source: §3.2 · p. 3
  • Training uses 15 flow predictions per VLM forward pass and delta actions with per-timestamp normalization, alongside stage-prediction and FAST auxiliary losses.The overall loss weights action, stage-prediction, and FAST components.Source: §3.2 · p. 3
  • Inference uses soft correlation-aware inpainting, cubic-spline compression of 26 predicted actions into 20 execution steps, stage voting, and task-specific correction rules.The stated action-compression speedup is 1.3×.Source: §3.2 · p. 4

3.3 Mathematical Notation

  • The notation distinguishes flow time from trajectory timestep and defines action sequences, predicted velocities, covariance, and observed versus free action dimensions.Flow time t runs from pure noise at t = 1 to the target at t = 0.Source: §3.3 · p. 4

4 Model Architecture

This section adapts Pi0.5 to BEHAVIOR-1K, replacing its language-based task specification with task-specific learned embeddings. The choice matches a setting with the same 50 tasks in training and evaluation, rather than a requirement to generalize to new natural-language tasks.

4 Model Architecture

  • The section describes architectural modifications to Pi0.5 for BEHAVIOR-1K tasks.The original model uses language embeddings for task specification; this adaptation uses task-specific embeddings instead.Source: §4 · p. 4, §4.1 · p. 4

4.1 Task Embeddings

  • This simplification fits the challenge setting because its 50 tasks occur in both training and evaluation, with no explicit requirement to generalize to new tasks described in language.Source: §4.1 · p. 4
  • The policy uses one trainable 2048-dimensional embedding for each of the 50 tasks instead of processing natural-language prompts.Source: §4.1 · p. 4

1. Only 50 distinct tasks (fixed set)

The policy tracks task stage to disambiguate visually similar states, and adds learnable VLM-layer combinations and custom attention masks to its action model. It also uses delta actions, per-timestamp normalization, and an action expert with auxiliary FAST training.

4.2 System 2: Stage Prediction and Fusion

  • System 2 predicts task stage from images and task embedding, filters predictions by voting, and supplies stage information to later model steps.This addresses non-Markovian states where nearly identical images can occur at the beginning and end of a task.Source: §4.2 · p. 5

4.2.1 Stage Prediction

  • A linear classifier predicts one of 5–15 task stages from VLM output; invalid task stages are masked, and training accuracy is approximately 99%.The stage estimate provides context for action prediction.Source: §4.2.1 · p. 5

4.2.2 Stage-Task Fusion

  • Stage-task fusion supplies five task-related tokens using sine-cosine encodings, task-specific learned embeddings, and gated combinations.Source: §4.2.2 · p. 5

4.3.1 Learnable Layer Combination

  • Each action-expert layer forms transformed keys and values as learnable linear combinations of all VLM-layer caches, initialized to preserve Pi0.5’s layer-to-layer attention.The transformation uses separate key and value coefficients and learnable biases.Source: §4.3.1 · p. 5

4.3.2 Properties

  • The layer-combination design can select or blend VLM layers with few parameters; learned weights show some tendency toward the last VLM layer, though this may be noise.Figure 3 maps action-expert layers to VLM layers and shows deviation from identity initialization.Source: §4.3.2 · pp. 5–6, Figure 3 · p. 6

4.4 Custom Attention Masks

  • Custom masks isolate reliable image and task inputs from noisier state and predicted-stage inputs; FAST tokens attend causally, while action-expert tokens attend bidirectionally within the chunk.Images and task embeddings are not allowed to attend to stage tokens, preventing target leakage in stage prediction.Source: §4.4 · p. 6, Figure 4 · p. 6

4.5.1 Delta Actions; 4.5.2 Per-Timestamp Normalization

  • The policy predicts joint-position deltas and normalizes action dimensions per timestamp because action distributions vary across a trajectory’s horizon.Velocities and gripper positions are excluded from per-timestamp normalization.Source: §4.5.1 · p. 7, §4.5.2 · p. 7

4.6 Action Expert Architecture; 4.7 FAST Auxiliary Training

  • The action expert is a Gemma 300M, 18-layer model, and auxiliary FAST training uses global quantile normalization for DCT compression.FAST has loss weight 0.05; the paper says per-timestamp normalization would break temporal smoothness for DCT compression.Source: §4.6 · p. 7, §4.7 · p. 7

5 Correlated Noise for Flow Matching

The paper replaces independent flow-matching noise with noise shaped by empirical action correlations. Shrinkage regularization and correlated interpolation are used to balance denoising difficulty across steps.

5.1 Motivation: Action Correlations in Robot Control

  • Training actions show temporal similarity and coordinated joint velocities, reflected in a strong block-diagonal structure in the 690 × 690 correlation matrix.The matrix covers 30 timesteps and 23 action dimensions.Source: Figure 5 · p. 7, §5.1 · p. 7

5. Correlated Noise for Flow Matching; 5.5 Impact on Training

  • Correlated noise matching action structure makes denoising steps more balanced, addressing the difficulty of early steps under standard independent Gaussian noise.The interpolation uses noise sampled from the regularized action covariance.Source: §5.1 · p. 7, §5.5 · p. 8

5.2 Correlation Matrix Estimation

  • The empirical correlation matrix is estimated from flattened normalized action sequences using their sample covariance.Flattening maps each H × D sequence to a vector of dimension HD.Source: §5.2 · p. 7

5.3 Beta Shrinkage for Robustness

  • Shrinkage regularization combines the empirical covariance with identity using β, with β = 0.5 used as the balanced choice.Source: §5.3 · p. 7

5.4 Correlated Noise Generation

  • Correlated noise is sampled using a Cholesky decomposition of the regularized covariance matrix.Source: §5.4 · p. 7

5.5 Impact on Training

  • The interpolation preserves correlated structure from pure noise through intermediate denoising times, making training difficulty more uniform across steps.At t = 1, the noise has the same correlation structure as real actions.Source: §5.5 · p. 8

6 Training

Training uses BEHAVIOR-1K demonstrations and multiple flow-matching samples per observation, sharing one VLM pass to reduce gradient variance. The final training process combines action, stage, and FAST losses, then fine-tunes task groups into task-specific checkpoints.

6.1 Training Data

  • Training uses 10,000 expert demonstrations across 50 tasks, totaling more than 1,200 hours; demonstrations are divided into 5–15 stages for stage prediction.Source: §6.1 · p. 8

6.2 Multi-Sample Flow Matching; 6.2.1 Motivation; 6.2.2 Algorithm

  • Multi-sample flow matching reduces training-signal variance by amortizing one expensive VLM forward pass across 15 action predictions with different sampled times and noise.Gradients flow back through all 15 samples.Source: §2 · p. 8, §6.2 · p. 8, §6.2.2 · p. 8

6.3 Loss Function; 6.3.1 Action Loss; 6.3.2 Stage Prediction Loss

6.4 Multi-Task vs. Task-Specific Training

  • Training proceeds from a multi-task phase over all 50 tasks to group-specific fine-tuning, and the final submission switches among four checkpoints by task ID.The task groups are based on validation results.Source: §6.4 · p. 8

6.5 Training Budget

  • The competition budget was approximately $13,000, including about $3,000 in personal spending and $10,000 sponsored by Nebius for main training runs.The main sponsored runs used 8×H200 GPUs.Source: §6.5 · p. 8

7 Inference Optimizations

Inference uses rolling soft inpainting to carry actions across prediction windows, then propagates corrections through learned action correlations. This is intended to preserve smooth transitions between windows.

7.1 Correlation-Aware Inpainting; 7.1.1 Soft Inpainting Strategy

  • Rolling inpainting predicts 30 actions, executes the first 26, and saves the last four to guide the next prediction’s first four actions.The next window starts close to the saved actions through soft inpainting.Source: §7.1 · p. 8, §7.1.1 · p. 8, Figure 6 · p. 9

7.1.2 Correlation-Aware Correction Propagation

  • Hard constraints on the first four actions can create a discontinuity at the boundary and out-of-distribution inputs during flow matching.Source: §7.1.2 · p. 9
  • The correction method uses the learned correlation structure to propagate changes from the four inpainted timesteps to free action dimensions at each denoising step.The correction matrix is precomputed to specify how free dimensions adjust to corrections on inpainted dimensions.Source: §7.1.2 · p. 9

3. Apply hard constraint on inpainted dimensions

This section describes inference-time constraints and adjustments for maintaining smooth actions, tracking task stages, and recovering from common failures. It also notes that task-specific rules were limited because they were not considered scalable or generalizable.

3. Apply hard constraint on inpainted dimensions

  • The correction matrix adjusts free dimensions in response to inpainted-dimension corrections to ensure smooth transitions.The matrix is precomputed once using a numerically stable solver.Source: §3 · p. 9

7.1.3 Soft Inpainting via Time Threshold

  • Inpainting correction is applied only when t > 0.3, preserving constraints early while allowing adaptation to changed observations late.The authors describe this as soft inpainting that permits deviation from the initial plan near the target.Source: §7.1.3 · p. 9

7.2 Action Compression via Cubic Spline Interpolation

  • Compression predicts 26 actions but executes 20 at 30Hz; cubic splines smooth interpolation, and compression is disabled for significant gripper changes.Base velocities are scaled by 1.3×, while arm and trunk joint velocities remain unchanged.Source: §7.2 · pp. 9–10

7.3 Stage Tracking with Voting Logic

  • Stage tracking uses a three-prediction window: two votes advance one stage, unanimous votes can trigger a skip or rollback, and transitions clear history.Unanimity makes rollback more conservative than forward transitions.Source: §7.3 · p. 10

7.4.1 General Gripper Correction

  • A gripper-opening rule treats a closure absent from that task’s training data at the current stage as a failed grasp; it approximately doubled success in selected tasks.Recovery behavior sometimes emerged after training on 50 tasks, but was rare and did not substantially improve total success rate.Source: §7.4.1 · p. 10

7.4.2 Task-Specific Rules

  • Only one task-specific rule was retained: roll back two stages in turning_on_radio if the final stage is reached without success; the authors consider this approach not scalable or generalizable.They had experimented with more fine-grained task-specific rules but did not pursue them further.Source: §7.4.2 · p. 10

8 Evaluation

The evaluation covers 50 household tasks with randomized instances and uses averaged partial success as its ranking metric. Evaluation infrastructure parallelizes the workload across machines to complete the full evaluation in under two days.

8.1.1 Challenge Protocol

  • The protocol evaluates 50 household activities over 10 randomized evaluation episodes per task, with task-specific time limits of twice average human completion time.Success is based on goal-condition satisfaction, including partial success.Source: §8.1.1 · p. 10

8.1.2 Evaluation Instances

  • Public scores use 10 published instances per task, while private scores use 10 additional held-out instances; both test new spatial configurations drawn from the training distribution.The evaluation does not test unseen object categories, language-goal understanding, or entirely new tasks.Source: §8.1.2 · pp. 10–11

8.2.1 Primary Metric: Averaged Partial Success

  • The ranking metric averages each episode’s fraction of satisfied goal conditions across 10 episodes per task, then averages task scores across 50 tasks as q_score.Binary success is reported for analysis but is not the ranking metric.Source: §8.2.1 · p. 11

8.2.2 Secondary Metrics: Efficiency

  • Distance navigated and hand displacement are tracked as efficiency metrics, but neither is used for ranking.They measure cumulative base movement and gripper movement, respectively.Source: §8.2.2 · p. 11

8.3 Evaluation Infrastructure

  • The team used 20 RTX 4090 machines in parallel and an on-demand cluster, bringing full evaluation time to under two days.A lightweight balancer distributed tasks and reran crashed episodes.Source: §8.3 · pp. 11–12

9 Results

The system achieved a 26% q_score with negligible public-private difference, while task-level analysis found dexterity and action-order problems among the observed failures. The authors also report emerging recovery behavior from cross-task learning and note that their ablations were limited in scope.

9.1 Competition Results

  • 26% q_score was achieved with negligible difference between public and private evaluation; partial successes contributed roughly half of the total score.The leaderboard table identifies the top five teams evaluated on held-out instances but supplies no comparison values in the provided passage.Source: §9.1 · p. 12, Table 1 · p. 13

9.2.1 General Observations

  • Longer tasks were not a fundamental obstacle: full completion became harder with more steps, while partial success remained achievable.Some tasks were nearly solved except under particularly tricky initial conditions; zero-success tasks often involved a high-precision step or sequence beyond current model limits.Source: §9.2.1 · p. 12

9.2.2 Failure Mode Analysis

  • In failure labels from 15 of 50 tasks, dexterity was dominant and accounted for around one-third of failures; progress/order errors were also common.The figure describes failure-reason distributions across labeled evaluation episodes.Source: §9.2.2 · p. 12, Figure 10 · p. 13
  • Other reported failures included repeated exploration, out-of-distribution confusion, reachability limits, and idealized demonstrations that left small deviations outside a narrow trajectory manifold.The authors also cite navigation confusion, robot falls, whole-body coordination, and head-camera visibility as failure modes or constraints.Source: §9.2.2 · pp. 12–13

9.2.3 Additional Observations

  • The multi-task model exhibited recovery behaviors, such as picking up fallen objects, that single-task models did not show; some tasks nevertheless performed worse under multi-task training.The authors hypothesize low relative dataset weight or confusion between visually similar tasks as possible reasons for the exceptions.Source: §9.2.3 · p. 13

9.2.4 Summary

  • The authors connect the challenge’s analysis to real-world concerns in dexterous manipulation, System-2-style guidance for imitation-learning policies, and broader pre-training data.They state that this makes work on the challenge relevant to real-world problems.Source: §9.2.4 · p. 13

9.3 Ablations

  • The authors lacked budget for full ablations; tested image-quality changes showed no meaningful success-rate change, including noticeable degradation from missing NGX support.The passage also reports no success-rate improvement from duration-based dataset weighting on one task.Source: §9.3 · pp. 13–14
  • Small inference-parameter variations showed no significant change, while extreme changes degraded scores; gripper opening increased q-score 2.2× on 13 tasks across 39 episodes.The inference tests included execution length, speedup, and voting-history length.Source: §9.3 · p. 14

10 Discussion and Conclusion

The Pi0.5 adaptation placed first with a 26% q-score across 50 household tasks, but training was far from converged and the authors identify unresolved questions about checkpoints and component contributions. They propose data, learning, perception, and control directions for future work.

10 Discussion and Conclusion

  • 26% q-score across 50 diverse household tasks earned the Pi0.5 adaptation first place, while long-horizon manipulation remained challenging.The authors report only 26% success despite extensive engineering.Source: §10 · p. 14
  • Nearly a month of training yielded only ∼2 dataset epochs; the authors identify the challenge timeline, rather than model capacity, as the limiting factor.They expect longer training to improve results.Source: §10 · p. 14
  • Four task-specific checkpoints were chosen mainly to meet the deadline, and whether one multitask model can achieve comparable results remains an open question.The authors say single-model versus multiple-checkpoint comparison is an interesting research question.Source: §10 · p. 14
  • Gripper correction rules contributed significantly to the score, but the authors characterize task-specific rules as competition-specific and not generalizable research contributions.They suggest more task-specific rules could further boost performance.Source: §10 · p. 14
  • Resource constraints prevented ablations, leaving the individual contributions and possible redundancy of the proposed components unidentified.The authors call for rigorous ablations.Source: §10 · p. 14
  • Attempts to train a smaller model from scratch were not promising; the authors identify pretrained Pi0.5 weights as critical to their approach.They describe internet-scale pretraining followed by robotics fine-tuning as the most practical path forward.Source: §10 · p. 14

10.1 Future Directions

  • Future work proposed includes corrective demonstrations from failure states, offline RL with advantage reweighting, and VLM-based System 2 checklists.The authors expect corrective data could improve robustness and describe the VLM approach as a possible effort/results trade-off.Source: §10.1 · p. 14
  • Other suggested directions are simulation RL with teacher–student distillation, fall-prevention constraints, and an additional rear-facing camera.The authors note privileged-information simulation RL is not directly applicable to real-world problems.Source: §10.1 · p. 14

Appendix

The appendix details stage-task fusion and model implementation, including the action expert and auxiliary FAST training. The authors caution that the fusion design and FAST's fine-tuning impact were not established through proper ablations.

A Stage-Task Fusion Details

  • Stage-task fusion combines task and stage representations, including normalized sine-cosine and learned stage encodings, into four 2048-dimensional outputs alongside the base task embedding.The resulting five task-related tokens are passed to the model; the authors say they did not conduct proper ablations and do not advocate this specific fusion approach.Source: A Stage-Task Fusion Details · p. 15

B Implementation Details

  • Implementation uses 23-dimensional continuous actions over a 30-timestep horizon, a frozen SigLIP-So400m/14 vision backbone, and single-GPU inference tested on an RTX 4090.The 23 dimensions cover base velocity, trunk, both arms, and grippers.Source: B Implementation Details · p. 16

C Action Expert Architecture

  • The action expert uses Gemma 300M with 4096-dimensional feedforward layers and eight attention heads; its input combines projected noisy actions with a time embedding.The time embedding is processed by a two-layer MLP for AdaRMS conditioning.Source: C Action Expert Architecture · p. 16

D FAST Auxiliary Training

  • Auxiliary FAST training uses autoregressive token prediction with a 0.05 loss weight, while the authors speculate it may aid simpler tasks but note its fine-tuning impact may be limited.The described setup uses a trained tokenizer for 22 action dimensions and global quantile normalization.Source: D FAST Auxiliary Training · p. 16

Related papers

Loading 2512.06951v2…