PaperlayerView exact revision on arXiv

Source-linked AI summary

In-context Robot Learning Made Simple: A Democratized Recipe for Manipulation Tasks

Minxing Li, Minghao Han, Weizhi Zhao, Hanwen Wang, Xiangshuo Liu, Shuyao Shang, Jingxiang Zhou, Mingchao Sun, Hongyu Pan, Mu Xu, Yu Liu, Lue Fan, Zhaoxiang Zhang

arXiv:2609.38173v1cs.RO

TL;DR

Robot ICL remains under-defined because visual demonstrations convey multiple kinds of task information without specifying what the robot should learn. The paper defines that target and presents SimpleICL, a minimalist framework with a visual prompt encoder and low-cost data collection. Experiments in simulation and the real world report ICL effectiveness and reveal capabilities and limits across action, semantic, compositional, and affordance discrimination.

  • Problem

    Visual demonstrations convey action trajectories, object semantics, affordances, spatial relations, and goals, leaving unclear what robot ICL should learn from them.

    Source: §1 · p. 2

  • Method

    SimpleICL defines robot ICL’s learning target and combines a visual prompt encoder with a low-cost, reproducible data collection protocol.

    Source: §1 · p. 2, §3.3 · p. 5, §3.2 · p. 5

  • Results

    Experiments in simulation and the real world demonstrate ICL effectiveness and reveal action, semantic, compositional, and affordance properties, including difficulty spontaneously separating object identity from affordance.

    Source: §1 · p. 2, §4.3.1 · p. 8, §4.3.2 · p. 9

  • Takeaways & Limitations

    The authors present robot ICL as a promising direction for further investigation and propose an open, reproducible pipeline to support research on it.

    Source: §1 · p. 2, §5 · p. 10

    Under the current setting, the model cannot spontaneously disentangle object identity from affordance; different affordances must be explicitly represented in training data.

    Source: §4.3.2 · p. 9

Abstract

from arXiv · show

We study robotic in-context learning (ICL), an emerging paradigm that enables robots to infer and execute tasks from visual demonstrations. Despite its growing promise, the problem itself remains under-defined: a visual demonstration simultaneously conveys action trajectories, object semantics, manipulation affordances, spatial relations, and task goals, making it unclear what information the robot is actually expected to follow. In this work, we first provide a clear problem definition of robot ICL that explicitly defines its learning target and resolves this fundamental prompt ambiguity. Building on this definition, we develop a minimalist and reproducible ICL framework (SimpleICL) with a visual prompt encoder and a low-cost data collection protocol. Without massive pre-training or specialized data infrastructure, our framework achieves strong performance in both simulation and real-world environments. Extensive experiments further reveal several key properties of robot ICL, including action, semantic, composition, and affordance discrimination. We will fully open-source our data and training pipeline to facilitate systematic and reproducible research on robot ICL. The project page can be found at https://simpleicl.github.io/simpleicl.

1 INTRODUCTION

Visual demonstrations can show how a task is performed without clarifying which aspects the robot should learn. The paper defines robot ICL’s learning target and presents SimpleICL as a simple, reproducible framework for studying it.

1 INTRODUCTION

  • The paper relates visual-prompt ICL to human skill learning: people observe others and autonomously imitate procedures rather than being physically controlled.Visual prompts can carry spatial locations, object appearance, and fine-grained action trajectories.Source: §1 · p. 2
  • Visual prompts convey abundant information but can leave intent unclear, including whether to follow motion trajectories, task semantics, manipulation style, or final goals.The paper frames this prompt ambiguity as a central open problem in robot ICL.Source: §1 · p. 2
  • SimpleICL defines what robot ICL should learn to resolve prompt ambiguity, then adds a universal visual prompt encoder and standardized data-design and collection protocol.The protocol defines actions, objects, and spatial layouts and includes cost-free data augmentation.Source: §1 · p. 2
  • The paper presents a minimalist architecture and reports ICL effectiveness in simulation and real-world experiments, while identifying underlying properties and open challenges.The properties include action, semantic, composition, and affordance discrimination.Source: §1 · p. 2
  • The authors propose a low-cost, reproducible data infrastructure and say they will open-source the data and training pipeline.They position this recipe as facilitating broader adoption and advancement of robot ICL research.Source: §1 · p. 2

2 RELATED WORK

Related work covers policies built on vision-language and video-generation models, as well as approaches that learn from human manipulation videos. Recent robot ICL uses demonstration videos as prompts, but remains under-explored.

2 RELATED WORK

  • Existing VLA policies often rely on task-specific finetuning to generalize to novel tasks (Kim et al., 2024; Black et al., 2024; Team et al., 2025).VLAs align text instructions and visual observations with a pretrained vision-language model, while WAMs use future-video prediction as a prior for inferring actions.Source: §2 · p. 3
  • Human manipulation data typically provide video rather than multimodal state and action signals, motivating test-time memory optimization and demonstration retrieval approaches (Feng et al., 2026; Park et al., 2026).WAM-TTT optimizes a lightweight memory; ReCAP adds a new task demonstration to a retrieval pool.Source: §2 · p. 3
  • Robot ICL uses human demonstration videos with text instructions, either as full-video context (Zhou et al., 2026; Li et al., 2026) or temporally aligned frames (Chen et al., 2026).The paper positions definition, phenomena, data design, and model architecture as issues it addresses in this under-explored area.Source: §2 · p. 3

3 METHOD

SimpleICL defines the semantic targets and robustness boundaries of robot ICL, then builds data and a visual prompt encoder around them. Its collection protocol and encoder are designed to support task-relevant learning while suppressing irrelevant variation.

3.1 PROBLEM DEFINITION OF ROBOT ICL

  • Robot ICL targets action, compositional, object, and affordance semantics while remaining robust to spatial mismatches, human-action details, and scene variations.The authors treat these four semantic dimensions as sufficient for most common manipulation tasks, while noting that some deployments may require additional domain-specific factors.Source: §3.1 · pp. 3–4

3.2 DATA DESIGN AND COLLECTION PROTOCOL

  • Four data types construct demonstrations to distinguish action, object, composition, and affordance semantics, including grasping a mug’s body versus its handle.Object-discriminative pairs perturb target positions and add distractors, while assuming only one instance of each target object per scene.Source: §3.2 · p. 4, Figure 2 · p. 4
  • Demonstrator and environmental variation support invariance; collection groups pair scenes with executable tasks, using scene switches to produce up to 3,000 valid pairs per working day and break demo-execution coupling.Minor transitions perturb an existing scene; major transitions replace it entirely.Source: §3.2 · p. 5, Figure 3 · p. 5

3.3 UNIFIED VISUAL PROMPT ENCODING MODULE

  • A visual prompt encoder uses cross-attention to turn video embeddings into compact conditioned tokens that query task-relevant cues and suppress background noise and irrelevant human-action details.The fixed number of context tokens supports arbitrary video lengths, and the jointly differentiable module is optimized with the downstream policy.Source: §3.3 · p. 5

4 EXPERIMENTS

Experiments evaluate SimpleICL in simulation and on unseen real-world tasks, examining in-context learning, semantic discrimination, robustness, and attention. Results show benefits from video conditioning and future prediction, alongside different learning requirements for composition and affordance.

4.1 IMPLEMENTATION DETAILS.

  • The implementation uses a pretrained Wan2.2-5B backbone, a 0.6B-parameter query module, and LoRA training for the video DiT.The action horizon is 32; other components are trained with full parameters.Source: §4.1 · p. 6

4.2 SIMULATION EXPERIMENTS

  • Simulation uses 18,000 ICL-oriented demonstrations with multiple task-relevant objects and varied placements, alongside 27,500 native RoboTwin demonstrations.The ICL-oriented data pipeline is designed to introduce genuine visual prompt ambiguity while retaining scene variation.Source: §4.2 · p. 6
  • ICL-oriented training improves success under visual ambiguity, while video conditioning supports unseen-task learning and removing future prediction sharply reduces performance on both tests.Without video conditioning, performance drops sharply on novel OOD tasks; removing future prediction changes ambiguity success from 72.1% to 20.2% and OOD success from 65.9% to 17.9%.Source: §4.2 · pp. 6–7

4.3 REAL-WORLD EXPERIMENTS

  • Real-world evaluation uses a FRANKA robot with three RealSense cameras and tests eight unseen tasks across easy, medium, and hard difficulty levels.Medium scenes offer multiple executable tasks; hard tasks require selecting among multiple valid targets.Source: §4.3 · p. 7, §4.3.1 · p. 7

4.3.1 MAIN RESULTS

  • In qualitative real-world comparisons, SimpleICL locates the intended drawer and chopsticks destination where FAST-WAM selects distractors.Evaluation includes eight unseen tasks with objects absent from training.Source: §4.3.1 · pp. 7–8, Figure 5 · p. 8

4.3.2 ANALYSIS OF COMPOSITION AND AFFORDANCE

  • Composition discrimination can emerge from training data without explicit same-scene composition contrasts, whereas affordance discrimination requires demonstrations of varied affordances for the same object.The affordance analysis uses objects such as mugs, cups, tape measures, and tape with alternative grasp targets.Source: §4.3.2 · pp. 8–9

4.3.3 ROBUSTNESS ANALYSIS

  • SimpleICL remains robust to spatial, demonstrator, and scene disturbances, while removing cross-group pairing significantly reduces spatial robustness.Without cross-group pairing, the model can bind task intent to fixed scene layouts and demonstrated spatial configurations.Source: §4.3.3 · p. 9, Table 5 · p. 9

4.4 ATTENTION VISUALIZATION

  • Attention to conditioning videos is temporally structured, concentrating on hands initially and peaking at object contact and task completion.During intermediate execution, attention follows both hands and the object with moderate intensity.Source: §4.4 · p. 10, Figure 7 · p. 10

5 CONCLUSION

The conclusion presents robot ICL as a promising research direction, supported by a data paradigm, collection pipeline, and video-conditioning module validated in simulation and real-world settings.

5 CONCLUSION

  • The paper pairs simulation and real-world validation with a data paradigm, efficient collection pipeline, and video-conditioning module for robot ICL.Its experiments also report findings about the nature of robot ICL.Source: §5 · p. 10

A SIMULATION NOVEL (OOD) TASKS

The simulation OOD evaluation comprises seven novel tasks spanning object placement, touch, laptop closing, and block movement.

A SIMULATION NOVEL (OOD) TASKS

  • The seven OOD tasks are placing a toy car in a plastic box, placing a cup in a basket, touching a toy car, placing playing cards on a plate, closing a laptop, moving a block onto a pad, and placing a mouse in a bowl.Figure 8 depicts the tasks used for simulation evaluation.Source: A SIMULATION NOVEL (OOD) TASKS · p. 15, Figure 8 · p. 15

B REAL-WORLD SAMPLES

The real-world sample figures show eight manipulation tasks, from weighing and wiping to serving, organizing, and object placement.

B REAL-WORLD SAMPLES

Loading 2609.38173v1…