From Play to Policy: Conditional Behavior Generation from Uncurated Robot Data

1 INTRODUCTION

• Robotic behavior generation from uncurated data remains challenging, despite conditional generation successes in language and vision.Prior play-data methods may require online reward-based training, while offline goal-conditioned methods often require rewards that real-world data lack.Source: §1 · p. 1
• C-BeT combines BeT’s multi-modal action generation with future-state conditioning to learn conditional behaviors from offline data.It does not require online environment interactions during training or rewards or Q functions.Source: §1 · p. 2
• C-BeT achieves significantly higher performance than prior play-data methods on future-conditioned tasks.The paper evaluates it on three simulated benchmarks and a real Franka robot trained with human-collected play data.Source: §1 · p. 3
• C-BeT learns competent visual policies for real-world tasks from fully offline multi-modal play data.The reported real-robot demonstration uses image and proprioception without human labeling or data curation.Source: Figure 1 · p. 2, §1 · p. 3

2 BACKGROUND AND PRELIMINARIES

• Play-like datasets may contain non-expert, non-unimodal behavior and lack both fully random actions and reward labels.Unlike conventional demonstration datasets, they need not reflect an expert repeating one task in the same way.Source: §2 · p. 3
• BeT models action distributions with a transformer and k-means action bins, supporting multi-modal behavior but not targeted modes at execution.Its unconditional rollouts prevent choosing a particular behavior mode during policy execution.Source: §2 · p. 3
• Conditional behavior learning maps observations and an additional condition to actions, where the condition may be a state, latent, or image observation.The paper describes evaluation through reward, distance to the specified outcome, or discounted visitation probability when an outcome mapping is defined.Source: §2 · p. 3
• GCBC trains on observation-action-goal tuples, while future-state relabeling can supply goals for play data without goal labels.Hindsight relabeling augments the dataset by treating a later reached state as a goal for an earlier observation and action.Source: §2 · p. 3

3 APPROACH

• C-BeT targets controllable generation of multiple tasks and behavior modes from play data with minimal additional annotation or curation.A future goal can leave one reasonable course of action or permit multiple paths, depending on the target.Source: §3 · p. 4, Figure 2 · p. 4

3.1 CONDITIONAL BEHAVIOR TRANSFORMERS (C-BET)

• The policy models action distributions from current and future observations, replacing individual observations with sequences for partial observability.The future observation sequence is drawn from a later point in a play trajectory.Source: §3.1 · p. 4
• C-BeT uses BeT because the conditional action distribution may be multi-modal, concatenating current and future observation sequences as input.Concatenation allows the current and future observation sequences to have independently chosen lengths.Source: §3.1 · p. 4
• Training dynamically pairs observation-action sequences with a later sequence of future observations as the conditioning input.The training objective applies the BeT loss to predicted actions, using focal loss for action bins and MT-loss for their offsets.Source: §3.1 · p. 4
• At evaluation, C-BeT samples actions using current observations and a future condition, primarily future observations but also trajectory-mode latent vectors in experiments.Target observations or newly collected demonstrations can provide the evaluation condition.Source: §3.1 · pp. 4–5, Figure 3 · p. 5

4.1 BASELINES

• The reward-free offline comparisons include goal-conditioned BC and RL, play motor primitives, hierarchical imitation, conditional IBC, GTI, unconditional BeT, and unimodal C-BeT.For image-based comparisons, WGCSL and GoFAR require a designed proxy reward, and several baselines are upgraded with observation sequences and transformers where applicable.Source: §4.1 · pp. 5–6

4.2 SIMULATED ENVIRONMENTS AND DATASETS

• The three simulated benchmarks span pixel-based self-driving, multimodal block pushing, and a multi-task kitchen, chosen for properties relevant to real-world robotics.CARLA uses 200 demonstrations and two possible routes to a goal; BlockPush uses 1,000 demonstrations and future block positions as conditions; the kitchen dataset has 566 human demonstrations covering four of seven tasks.Source: §4.2 · p. 6, Figure 4 · p. 6

4.3 HOW WELL DOES C-BET LEARN BEHAVIORS FROM PLAY?

• C-BeT performs significantly better than baselines on all three tasks; among MLP baselines, WGCSL leads state-based tasks and GoFAR leads CARLA.Play-LMP performs poorly in these long-horizon tasks, which lie beyond its intended short motor-primitive regime.Source: §4.1 · p. 5, Table 2 · p. 6, §4.3 · p. 7

4.4 HOW IMPORTANT IS MULTI-MODAL ACTION MODELING?

• Explicit multimodal action modeling helps outcome-conditioned learning in environments where multiple ways to achieve an outcome may exist.The comparison with unimodal C-BeT suggests that a unimodal policy after conditioning is not adequate for every environment.Source: §4.4 · p. 7

4.5 HOW DOES C-BET COMPARE TO OTHER FORMS OF CONDITIONING?

• On all three simulated environments, future-observation conditioning performs comparably to adding manual one-hot labels in CARLA and BlockPush or binary labels in Kitchen.Table 3 compares C-BeT with no supervised labels against versions using labels acquired with human supervision.Source: §4.5 · p. 7, Table 3 · p. 7

5.1 ROBOTIC ENVIRONMENT AND DATASET

• The robot used a Franka Panda in a toy kitchen, with seven normalized joint-angle deltas and binary gripper control as its action space.Volunteers supplied 460 play sequences totaling 265 minutes, without instructions to perform specific tasks.Source: §5.1 · p. 7

5.2 CONDITIONAL BEHAVIOR GENERATION ON REAL ROBOT

• C-BeT completed all real-world single-task conditionals consistently except knob rotation, outperforming the baselines.GoFAR moved toward targets but did not successfully grasp or interact with them.Source: §5.2 · p. 8, Table 4 · p. 8
• Without a high-level controller, C-BeT stitched together multiple tasks from play demonstrations to complete long-horizon goals.The evaluation measured how many tasks from each original sequence were completed across three conditioning sequences.Source: §5.2 · p. 8, Table 5 · p. 8
• With unseen conditioning trajectories, C-BeT retained 67% of single-task performance, completing 16 of 50 tasks; two distractors also reduced performance to around 67%.With four or more distractors, the robot completed no tasks.Source: §5.2 · p. 8

5.3 ANALYSIS OF FAILURE MODES

• BYOL-trained visual representations failed to identify knob status better than random chance, contributing to poor knob-operation accuracy.The authors link this representation failure to unsuccessful conditioning on knob state.Source: §5.3 · p. 9
• Multi-modal policies prevented collapse to sub-optimal solutions on more complicated tasks and interconnected play, while simpler oven-door data showed no need for multi-modality.Source: §5.3 · p. 9

• Outcome-conditioned learning includes goal- and reward-conditioned approaches, with demonstration relabeling used as data augmentation [Andrychowicz et al., 2017; Ghosh et al., 2019; Goyal et al., 2022].The passage distinguishes goal or state conditioning from transformer-based reward conditioning.Source: §6 · p. 9
• Prior play-data methods include Lynch et al. (2019) and Gupta et al. (2019), while Gupta et al. (2022) requires human annotation and environment instrumentation for goal labels.Source: §6 · p. 9
• Behavior generative models include inverse-reward approaches, generative action decoders, and energy-based models, the latter not primarily focused on goal conditioning.The cited approaches are Russell (1998), Ng et al. (2000), Ho & Ermon (2016), Pertsch et al. (2020a), Singh et al. (2020), Liu et al. (2020), Florence et al. (2021), Kostrikov et al. (2021), and Nachum & Yang (2021).Source: §6 · p. 9
• C-BeT builds on the Behavior Transformer architecture, alongside earlier offline behavior transformers such as Chen et al. (2021) and Janner et al. (2021).Source: §6 · p. 9
• C-BeT’s goal is orthogonal to transformer work on summarizing visual history or extrapolating long-term context, though the approaches could be combined.Source: §6 · p. 10

7 DISCUSSION AND LIMITATIONS

• If supplied features fail to capture relevant scene objects, robot execution often fails to interact with them; simpler unimodal tasks yield only meager gains from C-BeT.The authors suggest better visual representation learning and larger play datasets as future directions.Source: §7 · p. 10
• C-BeT is presented as a method for conditional behavior generation learned from offline play data.Source: §7 · p. 10

A BEHAVIOR TRANSFORMERS

• The appendix introduces Behavior Transformers as the backbone architecture and describes its architecture and training objective.Source: A BEHAVIOR TRANSFORMERS · p. 14

• BeT maps each continuous action to a k-means bin and a small continuous offset from that bin.The model is trained on observation sequences to predict action sequences.Source: §A.1 · p. 14
• Given observations, BeT predicts a distribution over action bins and associated offsets, then samples a bin and adds its offset to the bin center.Source: §A.1 · p. 14

A.2 BET TRAINING OBJECTIVE

• BeT’s loss is calculated from an observation and its associated ground-truth action.The model prediction includes discrete and continuous parts.Source: §A.2 · p. 14
• The ground-truth action is represented by its discrete bin and its offset from that bin’s center.Source: §A.2 · p. 14
• The objective combines focal loss for the discrete prediction with a multi-task loss for the continuous prediction.The passage identifies focal loss as a special case of negative log likelihood and names the second component as multi-task loss.Source: §A.2 · p. 14

B.1 IMPLEMENTATION USED

• C-BeT is based on the official BeT repository, while GCBC, WGCSL, and GoFAR use the GoFAR authors’ official repository.Source: §B.1 · p. 15

B.2 HYPERPARAMETERS LIST:

• The appendix reports environment-dependent and shared BeT training hyperparameters, mostly using defaults from Shafiullah et al. (2022).Source: §B.2 · p. 15, Table 6 · p. 15, Table 7 · p. 15

C ROBOT ENVIRONMENT DEMONSTRATION TRAJECTORIES

• Sample demonstration trajectories are shown for the real kitchen, CARLA, multi-modal block pushing, and Franka Kitchen environments.Source: Figure 5 · p. 16, Figure 6 · p. 17, Figure 7 · p. 17