PaperlayerView exact revision on arXiv

Source-linked AI summary

What Enables In-Context Behavior Prompting for Manipulation?

Austin Patel, Ben Pekarek, Joel Enrique Castro Hernandez, Shuran Song

arXiv:2606.30457v2cs.RO

TL;DR

Behavior prompting uses a robot sensorimotor demonstration to condition a policy on a new task, but the factors that enable this capability remain poorly understood. The paper studies those factors with new benchmarks, BPP, and iPhUMI. It finds task diversity is a key driver, with prompting improving unseen-task adaptation when diversity is sufficient and weaker task conditioning in low-diversity laundry.

  • Problem

    Prior work showed that policies can use raw sensorimotor demonstrations to perform new tasks in context, but the conditions that enable behavior prompting remained unclear.

    Source: §1 · p. 2

  • Method

    The authors study behavior prompting across procedurally generated drawing and tabletop tasks, using BPP, an in-context visuomotor policy, and iPhUMI, a handheld interface for test-time demonstrations.

    Source: §1 · p. 2, §3.2 · p. 3

  • Results

    Task diversity, rather than demonstrations per task, is a key driver; with sufficient diversity, prompts reduce drawing error by 80.7% over goal-image conditioning and improve chained-manipulation success by up to 20.8% over language conditioning.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Attention analysis finds that the policy uses the prompt as dense sub-goals, while prompt ablations show that observations anchor lookup and actions support fine-grained action adaptation.

    Source: §1 · p. 2, §6 · p. 9

    The benchmarks vary object positions and board rotations but not scene or object identity, so they do not assess prompts that differ in those aspects from the target environment.

    Source: §5 · p. 9

Abstract

from arXiv · show

Behavior prompting is a paradigm in which a sensorimotor robot demonstration, called a behavior prompt, serves as an in-context prompt for performing new tasks at test time. While prior work has shown that this capability is possible, the conditions that enable it remain poorly understood. We present an empirical study of when, how, and why behavior prompting works. To support this study, we introduce DrawAnything and LIBERO-Gen, benchmarks with up to 2000 procedurally generated tasks that evaluate test-time adaptation to unseen drawing and tabletop manipulation tasks. We also present Behavior Prompting Policy (BPP), an in-context visuomotor architecture, and iPhUMI, a handheld interface to demonstrate behavior prompts at test time. Our main finding is that task diversity, rather than demonstrations per task, is a key driver of prompting capability. Given sufficient diversity, a behavior prompt improves adaptation to unseen tasks, reducing drawing error by 80.7% over goal-image conditioning and improving success on chained manipulation tasks by up to 20.8% over language conditioning. Given insufficient diversity in a real-world laundry experiment, behavior prompting has weaker task conditioning than a language baseline. An attention analysis shows how the prompt is used: the policy follows it step by step as a source of dense sub-goals. Prompt ablations show that dense sensorimotor detail matters: removing actions or downsampling the prompt hurts fine-grained action adaptation. We have open-sourced all components to enable reproducible research on behavior prompting without needing industrial-scale data collection or compute.

1 INTRODUCTION

Behavior prompting conditions a robot policy at test time on a sensorimotor demonstration, but the factors that enable it remain poorly understood. The paper studies those factors with new benchmarks and an in-context policy, finding that task diversity is central to prompting capability.

Background

  • Behavior prompting uses a robot sensorimotor demonstration at test time to specify both the desired task and an example of how to complete it, unlike language instructions that can leave the how underspecified.Source: §1 · pp. 1–2

Research gap

  • The study addresses the gap left by ICRT, which showed behavior prompting is possible but did not establish the conditions that enable it (Fu et al., 2025).Source: §1 · p. 2

Study design

  • DrawAnything and LIBERO-Gen support tests of unseen drawing and tabletop tasks, while laundry folding probes a low-diversity real-world setting.The study introduces BPP, an in-context visuomotor architecture, and iPhUMI, a handheld interface for demonstrating prompts at test time.Source: §1 · p. 2

Main findings

  • Task diversity, rather than demonstrations per task, is a key driver of prompting capability; with sufficient diversity, prompts improve unseen-task adaptation, while low-diversity laundry conditioning is weaker than language conditioning.The study reports an 80.7% reduction in drawing error over goal-image conditioning and up to a 20.8% improvement in chained-manipulation success over language conditioning.Source: §1 · p. 2

Contributions

  • The authors release the benchmarks, study components, and reference architecture to support reproducible research without industrial-scale data collection or compute.Source: §1 · p. 2

2 RELATED WORK

Prior work studies adaptation to new tasks through zero-shot policies, demonstration-based adaptation, and varied prompt representations. Existing behavior-prompting work establishes feasibility, while this paper focuses on isolating its enabling factors in controlled, reproducible settings.

Test-time Adaptation to New Tasks

  • Language-conditioned policies and VLA models adapt zero-shot to some unseen environments and objects, while few-shot approaches use demonstrations through weight updates, structured transfer, or fixed-weight conditioning.Source: §2 · pp. 2–3
  • In-context demonstrations vary in representation, including keypoints, graphs, multimodal prompts, reasoning traces, structured rationales, and human video, which crosses an embodiment gap.Source: §2 · p. 3

Behavior Prompting

  • Behavior prompting conditions a policy on raw robot-space demonstrations; ICRT validated the paradigm in one tabletop domain, but did not study task diversity, prompt modalities, or prompt attention (Fu et al., 2025).Source: §2 · p. 3

Benchmarks

  • Prior benchmarks often test training tasks, permit scene-only task identification, or lack per-task demonstrations; VIMA-Bench instead holds out prompt-specified rearrangements with fixed primitives and object-centric prompts (Jiang et al., 2023).Source: §2 · p. 3

3 METHOD

The method defines behavior prompts as robot-space demonstrations, then uses BPP to condition closed-loop action generation on a prompt and the current observation. It also introduces iPhUMI for collecting and supplying demonstrations at test time.

3.1 BEHAVIOR PROMPT DEFINITION

  • A behavior prompt is a single demonstration of a desired task in a different environment configuration, represented by observations, proprioception, and actions in the robot’s sensorimotor execution space.It can clarify spatial-temporal structure, preferences, or manipulation strategy that language leaves underspecified.Source: §3.1 · p. 3

3.2 BEHAVIOR PROMPTING POLICY (BPP)

  • BPP takes a behavior prompt and current observation as input and outputs closed-loop actions through a prompt encoder and action decoder.Source: §3.2 · p. 3
  • BPP groups each prompt chunk's observation and proprioception with the actions leading to the next chunk, then attention-pools them into an embedding sequence.Observations and proprioception are typically downsampled to 1 Hz, while actions retain the full behavior sequence.Source: §3.2 · p. 4, Figure 2 · p. 4
  • The prompt encoder cross-attends from the current observation to prompt chunks, and the action decoder uses that information with the observation to denoise actions through diffusion.Training samples one prompt demonstration and batches of observations and future action chunks from other demonstrations of the same task, without requiring explicit spatial or temporal correspondence.Source: §3.2 · p. 4

3.3 IPHUMI

  • iPhUMI adapts a handheld UMI gripper with an iPhone, using ARKit localization and wireless streaming to let a human provide a behavior prompt to BPP at test time.Source: §3.3 · p. 4
  • The benchmark suite covers unseen drawing reproduction across board poses, bowl selection and placement, and two-step tabletop interactions including chained tasks.Source: Figure 3 · p. 5

4 EVALUATION

Across procedurally generated drawing and tabletop benchmarks, the evaluation tests unseen-task adaptation. Results vary with task diversity, while attention and prompt-content analyses examine how behavior prompting works.

4 EVALUATION

  • DrawAnything covers unseen simulated and real drawings, while LIBERO-Gen tests unseen pick-place combinations and two-step sequences built from individually seen skills.Source: §4 · p. 5

4.1 WHEN DOES BEHAVIOR PROMPTING WORK?

  • BPP reduces unseen-drawing error by 80.7% over Goal-Image and gains up to 20.8% over Language on unseen chained tasks, while rivaling finetuned π0.5 without pretraining.Source: Figure 4 · p. 6, §4.1 · pp. 6–7
  • With only three laundry tasks, language conditioning succeeds on all tasks, while BPP sometimes executes the wrong fold competently, indicating weak task conditioning.Source: Figure 6 · p. 7, §4.1 · pp. 7–8
  • Task diversity matters more than demonstrations per task: unseen-task adaptation emerges only at high diversity, with roughly 1,000 drawing tasks versus a few hundred LIBERO-Gen tasks benefiting prompting.Source: §4.1 · p. 7, Figure 5 · p. 7

4.2 HOW DOES THE POLICY USE THE PROMPT?

  • Attention tracks the prompt section matching current drawing observations and LIBERO-Gen task milestones, supplying upcoming states and actions as dense, stepwise guidance.Source: Figure 7 · p. 8, §4.2 · p. 8

4.3 WHY DOES THE PROMPT HELP?

  • For DrawAnything-Sim, observations anchor prompt lookup and actions improve unseen-task adaptation, while proprioception adds no benefit because the cursor is visible in the image.Source: §4.3 · p. 9, Figure 8 · p. 9
  • Downsampling prompt observations below 1 Hz hurts unseen-drawing adaptation, while attention pooling improves performance and shortens the prompt sequence.Source: §4.3 · p. 9, Figure 8 · p. 9
  • Prompt perturbations cause moderate degradation: three inserted pauses raise drawing error by 24%, while slowing by 1.25× or adding trajectory jitter causes smaller increases.Source: §4.3 · p. 9

5 LIMITATIONS

The study’s conclusions are bounded by the benchmarks’ scope, the types of unseen tasks, and the settings in which prompting is compared with other task descriptors.

5 LIMITATIONS

  • The benchmarks vary object positions and board rotations, but not scene or object identity.The authors call for broader manipulation data to study prompts that differ more from the target environment.Source: §5 · p. 9
  • LIBERO-Gen tests recombinations of seen skills, not whether one prompt can teach entirely new tabletop action primitives.A benchmark isolating new primitives is needed to test that capability.Source: §5 · p. 9
  • The study identifies different task-diversity requirements in DrawAnything-Sim and LIBERO-Gen, but not which task properties determine them across domains.The authors identify that cross-domain question as future work.Source: §5 · p. 9
  • The attention analysis characterizes BPP; whether dense sub-goal attention lookup generalizes to other in-context architectures remains untested.Applying the analysis to other architectures is proposed as future work.Source: §5 · p. 9
  • Goal images or language may suffice in many cases, so a full behavior prompt is not always necessary for the desired test-time adaptation.The authors propose evaluating policies that can select among task descriptors at test time.Source: §5 · p. 9

6 CONCLUSION

The study finds that task diversity, prompt attention, and dense sensorimotor detail matter for behavior prompting. Its benchmarks, data, and code are released for reproducible study without industrial-scale resources.

6 CONCLUSION

  • Task diversity, rather than demonstrations per task, is a key driver of prompting capability; the policy uses prompts as dense sub-goals, with observations anchoring lookup and actions improving fine-grained adaptation.The authors report the diversity finding as reproducing claims in concurrent industrial reports, while the attention analysis and prompt ablations extend beyond those reports.Source: §6 · p. 9

REPRODUCIBILITY STATEMENT

  • All simulation results are directly reproducible with the open-sourced code and datasets, and the iPhUMI hardware and application software are also open-sourced.Training hyperparameters are specified in the code release.Source: REPRODUCIBILITY STATEMENT · p. 10

A.1 SCRIPTED DATA GENERATION

  • Real-drawing training combines 200 human-collected iPhUMI tasks with five demonstrations each and 800 scripted tasks with six demonstrations each.iPhUMI data supports human prompts at test time, while scripted data helps reach task diversity for adaptation.Source: §A.1 · p. 15
  • For scripted demonstrations, operators randomize canvas pose, estimate it from four corner markers, transform it into the robot frame, and execute a scripted trajectory.The procedure also randomizes the robot’s starting and retreat poses across tasks and demonstrations.Source: §A.1 · p. 15

A.2 EVALUATION PROCEDURE

The evaluation pairs behavior prompts with goal images from the same demonstration and controls initial conditions through resets. Qualitative comparisons show BPP producing coherent unseen drawings where Goal-Image often does not.

A.2 EVALUATION PROCEDURE

  • Each trial uses a matched pair: the full demonstration conditions BPP, and a goal image extracted from that same demonstration conditions Goal-Image.The canvas is placed at a random position and orientation for evaluation.Source: §A.2 · p. 15
  • The policies are run from the same initial state after a reset, then the procedure repeats at a new random canvas placement with a new prompt-image pair.Training-task pairs come from six demonstrations across four procedural tasks; unseen-task pairs come from three iPhUMI demonstrations for each of six tasks, with ten rollouts per task.Source: §A.2 · p. 15
  • Goal-Image can roughly match training-drawing structure but fails to replicate unseen drawings, while BPP reconstructs both from a single demonstration.Figure 10 shows green executed drawings and red goal-image overlays.Source: Figure 10 · p. 16

A.3 QUALITATIVE COMPARISON: GOAL IMAGE VS. BEHAVIOR PROMPTING

  • On real-drawing training tasks, Goal-Image often produces recognizable or partial shapes with spatial misalignment or local geometric errors that increase Chamfer distance.This failure mode is based on qualitative inspection of Figure 10a.Source: §A.3 · p. 16
  • BPP produces coherent unseen drawings aligned with demonstrated behavior, while Goal-Image often degenerates into near-random strokes.The figure shows successful BPP drawings as well as a failure case.Source: Figure 9 · p. 15, §A.3 · p. 16
  • Goal-Image’s similar quantitative error on training and unseen tasks can mask poor unseen-task fidelity, because strokes may cover the target region without reproducing its structure.The authors attribute moderate Chamfer error to this rough region coverage.Source: §A.3 · p. 16

B DRAWANYTHING-SIM TASKS

DrawAnything-Sim uses procedurally generated training drawings and held-out evaluation drawings with varied task complexity and board rotations; performance is measured by Chamfer distance.

B DRAWANYTHING-SIM TASKS

  • Performance is measured by the lowest Chamfer distance in pixels reached during each rollout, averaged over 32 environment seeds and 50 unseen tasks.Source: B DRAWANYTHING-SIM TASKS · p. 17
  • The training set contains 2,000 procedurally generated tasks with five demonstrations each, combining one to six randomly parameterized drawing parts.Parts include lines, Bézier curves, partial or full ovals, and pen-up movement; some connect to earlier parts, and trajectories include varied board rotations, speed, and noise.Source: Figure 11 · p. 17, B DRAWANYTHING-SIM TASKS · p. 17
  • Evaluation uses 50 hand-collected tasks unseen during training, with five demonstrations per task and varied board rotations, complexity, and duration.The dataset visualization shows the training and evaluation sets.Source: B DRAWANYTHING-SIM TASKS · p. 17, Figure 12 · p. 18

C RESULTS ON ORIGINAL LIBERO DATASET

On original LIBERO, behavior prompting performs similarly to a language-conditioned diffusion policy, while LIBERO-Gen expands evaluation to procedurally generated tasks. Its task definitions vary object locations and goals, and its demonstrations combine scripted policies with existing teleoperation data.

C RESULTS ON ORIGINAL LIBERO DATASET

  • On original LIBERO, behavior prompting achieves performance similar to the language-conditioned diffusion policy.The authors note that the original tasks are sufficiently described by language and the benchmark splits do not test adaptation to unseen tasks.Source: C RESULTS ON ORIGINAL LIBERO DATASET · p. 19, Table 1 · p. 19

D LIBERO-GEN IMPLEMENTATION DETAILS

  • LIBERO-Gen generates new task definitions and demonstrations, supporting its Combination and Chain benchmarks.The implementation builds on the original LIBERO task definitions.Source: D LIBERO-GEN IMPLEMENTATION DETAILS · p. 19

D.1 PROCEDURAL TASK GENERATION

  • LIBERO-Gen enumerates initial object locations and desired final states to create task variations, including sequentially chained action primitives.Users specify procedural operations, such as varying starting locations or goal states.Source: §D.1 · p. 19

D.2 PROCEDURAL DEMONSTRATION GENERATION

  • Demonstrations combine simulator-state-based scripted policies with original LIBERO data, varied poses and initial states, and predicate-based validation.For chained tasks, the desired primitive order is stored separately because BDDL does not encode it.Source: §D.2 · p. 20

D.3 LIBERO-GEN COMBINATION TEST TASKS

LIBERO-Gen Combination tests unseen pick-place pairings whose component locations appeared in training, while Chain tests unseen sequences of individually known primitives. The supplied baseline and prompt-sensitivity passages describe controlled model encodings and moderate effects from prompt perturbations.

D.3 LIBERO-GEN COMBINATION TEST TASKS

  • Unseen LIBERO-Gen Combination tasks pair pick and place locations that appeared individually in training but never jointly.The setup evaluates adaptation to unseen instructions, not unseen action primitives.Source: §D.3 · p. 20, Figure 13 · p. 20

D.4 LIBERO-GEN CHAIN TEST TASKS

  • LIBERO-Gen Chain evaluates unseen sequences of primitives that were seen individually during training, not entirely new primitives.All two-step chains start from the first-step state; red initializations represent second-step single actions, excluded in the no-second-step ablation.Source: Figure 14 · p. 21, §D.4 · p. 22

E BASELINE DETAILS

  • The Language, Goal-Image, and BPP models share a diffusion CNN U-Net action decoder, while their observation encoders differ.Goal images do not indicate the ordering of two chained operations.Source: E BASELINE DETAILS · p. 24

F PROMPT-QUALITY SENSITIVITY

  • Slowing, jittering, or inserting pauses into test prompts each moderately degrades BPP drawing performance relative to a clean prompt.The sensitivity evaluation uses 50 unseen DrawAnything-Sim tasks across three seeds.Source: F PROMPT-QUALITY SENSITIVITY · p. 25, Table 2 · p. 25

G BPP ARCHITECTURAL COMPARISON TO ICRT

The architectural comparison contrasts BPP’s prompt-focused modules and action diffusion with ICRT’s causal transformer and L1 loss. BPP supports unbounded rollouts, while the reported DrawAnything-Sim comparison attributes most of its gap to ICRT to the action head rather than history.

G BPP ARCHITECTURAL COMPARISON TO ICRT

  • BPP uses separate prompt-understanding and action-generation modules with cross-attention and action diffusion; ICRT uses causal self-attention and L1 action loss.ICRT is the prior behavior-prompting architecture used as a baseline.Source: G BPP ARCHITECTURAL COMPARISON TO ICRT · p. 25
  • BPP training samples one prompt and observation-action pairs across demonstrations, whereas ICRT trains on concatenated prompt and rollout sequences with teacher forcing.Both models use prompts and rollouts from a single task per batch.Source: G BPP ARCHITECTURAL COMPARISON TO ICRT · p. 25
  • ICRT takes 6.5 ms per inference call versus 49 ms for BPP, while BPP can roll out indefinitely and ICRT’s duration is tied to training context length.The authors make no efficiency claim for BPP; ICRT uses one L1 regression pass, while BPP uses 16 denoising steps.Source: G BPP ARCHITECTURAL COMPARISON TO ICRT · p. 25
  • 4.84 drawing error for BPP with an L1 head versus 5.07 for ICRT, compared with 3.43 for BPP on unseen DrawAnything-Sim drawings.Replacing BPP’s diffusion head with L1 regression accounts for most of the gap to ICRT, leaving little to attribute to history.Source: G BPP ARCHITECTURAL COMPARISON TO ICRT · p. 25
  • The disturbance protocol reverts the canvas after drawing progress while leaving each model’s context unchanged, so ICRT retains states mismatched to current canvas progress.The comparison uses a 10-step action horizon and reports drawing error without and with the mid-rollout revert.Source: G BPP ARCHITECTURAL COMPARISON TO ICRT · p. 25, Table 3 · p. 26

H LAUNDRY FOLDING DETAILS

The laundry experiment covers three folding tasks and evaluates with new prompts for tasks seen in training. The authors report poor behavior-prompting performance in this low-diversity regime and describe a possible failure involving spurious task cues.

H LAUNDRY FOLDING DETAILS

  • The experiment covers left-arm, right-arm, and two-arm bottom-up folding, with each task starting from an unfolded sweater.The dataset includes about 150 full demonstrations per task, about 1,000 error-correction demonstrations total, and five new evaluation prompts per task.Source: H LAUNDRY FOLDING DETAILS · p. 26
  • With low task diversity, BPP sometimes grasps the sweater bottom with the right arm but directs the left arm to fold the sleeve instead.The authors suggest prompt indirectness might make BPP sensitive to overfitting to spurious task cues.Source: H LAUNDRY FOLDING DETAILS · p. 26
  • Behavior prompting performs poorly in the low-diversity regime, while advantages in a high-diversity setting are presented as an expectation.The authors envision demonstrations spanning folding styles and garments clarifying temporal steps and spatial interaction points.Source: H LAUNDRY FOLDING DETAILS · p. 26
  • The authors suggest a single demonstration might be a more natural way for users to specify folding preferences than language.This is presented as a practical possibility, not an evaluated result.Source: H LAUNDRY FOLDING DETAILS · p. 26

I BPP TRAINING DETAILS

BPP training groups determine the granularity at which prompts can shape execution, while iPhUMI supports demonstration collection, robot deployment, and test-time behavior prompting. The section describes the training and interface components that support these workflows.

I BPP TRAINING DETAILS

  • Task groupings set the granularity at which a behavior prompt can influence execution, including adherence to a grasping strategy.Distinct strategies for the same pick-and-place task require separate groupings.Source: I BPP TRAINING DETAILS · p. 26
  • The training details include compute requirements, a transformer-decoder prompt encoder, and a CNN U-Net action-diffusion decoder shared by BPP, language, and goal-image models.The compute table includes intermediary and final rollout evaluations for simulation experiments.Source: Table 4 · p. 26, Table 5 · p. 26, Table 6 · p. 27

J IPHUMI

  • iPhUMI combines a handheld gripper with an iPhone application for recording demonstrations and using the phone as a robot camera during deployment.For drawing, its gripper accepts a spring-mounted marker for compliant drawing without a force-torque sensor.Source: Figure 15 · p. 28, J IPHUMI · p. 29

J.1 IPHUMI DATA COLLECTION

  • iPhUMI records camera, depth, gripper-pose, and gripper-width data, while ARKit tracks phone pose and supports a shared coordinate frame across multiple devices.Depth is not used in the paper’s models or experiments; users can also calibrate the gripper, label tasks, review demonstrations, and export data.Source: §J.1 · pp. 29–30

J.2 IPHUMI POLICY DEPLOYMENT

  • The deployment interface streams iPhone camera feeds and depth to the robot over USB or Ethernet.Main and ultrawide camera feeds are streamed at 960x720 and 60Hz; depth is streamed at 320x240 and 30Hz.Source: §J.2 · p. 30, Figure 19 · p. 30

J.3 IPHUMI FOR TEST-TIME BEHAVIOR PROMPTING

  • For test-time prompting, iPhUMI wirelessly sends a user’s single demonstration to a robot-connected desktop, which processes it into a prompt for BPP.This workflow lets users specify new tasks at test time.Source: §J.3 · p. 30
Loading 2606.30457v2…