Source-linked AI summary

In-Context Learning for Robots: Methods and Applications

Haojian Huang, Zexi Li, Junhao Guo, Yehang Zhang, Wenxuan Peng, Bohan Zhou, Weilin Ruan, Leyi Wu, Chenxu Wang, Jianchong Su, Binghui Xie, Wosong Chen, Yingjie Xu, Tianhao Zhou, Suzeyu Chen, Pukun Zhao, Jiaqi He, Xinyi Li, Runze Li, Peiran Dong, Shaoxiang Dang, Jing Huang, Yingbing Chen, Yifan Chang, Tianyi Zhang, Shiyuan Deng, Haozhi Wang, Yangkai Wei, Wenqian Li, Han Yang, Kaiwen Zhou, Huaping Liu, James Cheng, Rui Shao, Donglin Wang, Yaochu Jin, Jianye Hao, Ying-Cong Chen, Yinchuan Li

arXiv:2609.36012v1cs.ROcs.LG

TL;DR

Robots may have reusable competence yet lack evidence about what a present task requires. This review organizes how contextual evidence reaches action across four interfaces and examines how training, correspondence, and memory support transfer. Its synthesis centers on preserving task-relevant distinctions through execution, while identifying compositional acquisition and physical recursive self-improvement as research directions.

  • Problem

    Robots can possess reusable competence yet lack evidence about which procedure, order, or interaction response the present task requires.

    Source: §1 · p. 3
  • Method

    The review organizes robot ICL into four interfaces for carrying contextual evidence into action and examines the roles of training, correspondence, and memory.

    Source: §1 · p. 3, §9 · p. 76
  • Results

    The synthesis identifies preserving task-relevant distinctions from evidence to execution as central, with evaluation separating teaching responsiveness, physical transfer, and retained-experience benefits.

    Source: §9 · p. 76
  • Takeaways & Limitations

    The research agenda calls for compositional task acquisition, revising affected constraints from feedback, discarding inapplicable experience, and improving how successors learn tasks.

    Source: §9 · p. 76

    The interfaces depend respectively on compatible representations, applicable interaction geometry, realizable predictions, and sufficient executor capabilities.

    Source: §9 · p. 76

Abstract

from arXiv · show

General-purpose robots must infer what a new task requires and translate that understanding into appropriate physical action. In-context learning (ICL) for robots supports this process by using demonstrations and interaction to direct existing competence with neural parameters held fixed during deployment. We organize this literature review around the interfaces connecting contextual evidence to execution, distinguishing four families: context-conditioned policies, geometric demonstration transfer, world-model-based control, and skill- and agent-based execution. Comparing these interfaces clarifies their transfer assumptions and the roles of training, correspondence, and memory in making context useful. Across manipulation and navigation, we examine how these mechanisms preserve taught requirements as objects, environments, and execution conditions change. This analysis links method design to evaluation practices that distinguish responsiveness to teaching, physical transfer, and benefits from retained experience. The resulting agenda connects compositional task acquisition and faithful transfer with physical recursive self-improvement, in which experience improves the ability to learn subsequent tasks.

1 Introduction

The review asks how contextual evidence resolves what reusable competence leaves undetermined and how that resolution survives execution. It organizes the interfaces carrying evidence into action and examines how training, correspondence, memory, and evaluation shape transfer.

1 Introduction

  • Teaching and interaction resolve task-specific information gaps while deployed neural parameters remain fixed, but the task relation must survive changes in physical realization.Examples include demonstrations specifying procedure, interaction revealing material response, and cross-object transfer after replacing interacting objects.Source: §1 · p. 3
  • The organizing question is how new evidence resolves what existing competence leaves undetermined and how the resulting information survives physical execution.Different intermediate representations carry this burden, while correspondence, training, and memory affect whether they preserve needed information.Source: §1 · p. 3
  • The review traces both intended effects and prescribed order or contact through execution, since a correct destination can still violate a required grasp.Source: §1 · p. 3
  • The taxonomy distinguishes context-conditioned policies, geometric demonstration transfer, world-model-based control, and skill- and agent-based execution.Correspondence and memory are shared mechanisms across the families.Source: §1 · p. 3
  • The review examines how training relationships and changes in objects, environments, and execution conditions constrain transfer, distinguishing motor competence from the ability to learn through teaching.Source: §1 · p. 3
  • Its evaluation synthesis separates context dependence, physical transfer, and retained-experience benefits, motivating compositional learning and improved teachability.Source: §1 · p. 3

2 Foundations: Why Do Robots Need Context?

This section explains why robots need context: the same visible scene can require different actions, and contextual evidence can direct competence they already possess. It traces how evidence is adapted, represented, and connected to control through interfaces such as demonstration transfer, prediction, and procedural reuse, including how teaching and physical adaptation raise the question of whether experience improves later learning.

2 Foundations: Why Do Robots Need Context?

  • A demonstration, interaction history, or contact evidence can resolve task-relevant ambiguity and direct actions the robot already knows how to execute.For example, packing order, completed placements, or a tighter fit can change what action is appropriate despite the same visible scene.Source: §2 · p. 5

2.1 From competence to contextual adaptation

  • The review distinguishes history-based adaptation from fixed-parameter learning from supplied teaching, and frames physical recursive self-improvement and collective knowledge evolution as broader objectives.Physical self-improvement concerns experience improving how subsequent tasks are learned; collective knowledge evolution extends knowledge exchange to groups of robots.Source: §2.1 · pp. 5–6

2.2 Contextual and parameter adaptation

  • Adaptation may change contextual inputs, retained artifacts, neural parameters, or the model-external execution harness, so evaluation must isolate which component changed.Matched interventions can remove evidence, clear recurrent state, suppress retrieval, or restore pre-adaptation weights to test the source of change.Source: §2.2 · pp. 6, 9

2.3 Four method families and shared design dimensions

  • Four execution interfaces connect context to action, while task specification, correspondence, physical response, and execution state describe the information context contributes.A demonstration can condition action inference, define a motion reference, constrain a predicted future, or specify a skill request; memory and modality are separate design choices.Source: §2.3 · pp. 9–11

2.4 Historical development

  • Historically, task-family adaptation learned reusable responses [1] [2] [4] [334], while sequence interfaces and broad pretraining expanded what teaching could specify and redirect [3] [5] [6] [17] [24].Broader priors make more competence available to teaching, but familiar continuations may conflict with new examples, so evaluation must test whether evidence redirects behavior.Source: §2.4 · pp. 12–13
  • Demonstration reuse ranges from nearest-neighbor action selection [149] and explicit alignment [34] [35] to video-initialized interaction learning such as WHIRL [150], each relying on different transfer assumptions.A one-example interface may support replay, transformed reference execution, or new policy learning.Source: §2.4 · p. 13
  • World models evolved from using recent transitions to evaluate action sequences [19, 20] to conditioning predictions on which future should be realized [21] [24].This combines physical prediction with task interpretation: the controller must preserve the procedure inferred from an example while predicting its realization in the current scene.Source: §2.4 · pp. 13–14
  • Of 412 references, 297 first appeared since 2024 and 197 in the partial 2026 window, showing a concentration in recent publications.Figure 9 covers the full reference set; the 2026 count is for the partial year ending 29 September 2026.Source: Figure 9 · p. 14, §2.4 · p. 15
  • The 260-reference method subset includes 123 context-conditioned policy, 24 geometric-transfer, 22 world-model-control, and 91 skill- and agent-based references.The subset also includes supporting mechanisms and parameter-adaptation comparisons.Source: Figure 9 · p. 14, §2.4 · p. 15
  • Task structure supports compositional reuse through demonstration-derived routines [26], language-model programs [27], or video-derived outlines for existing planners and executors [31].The intermediate must preserve binding details such as contact and ordering, which the executor cannot recover if omitted.Source: §2.4 · p. 14
  • For robots, the learning question is whether combining capabilities improves subsequent acquisition: LMPC studies responsiveness to later teaching [117], while reward and learning-program search change skill acquisition [89] [115].The review develops this question for individual and shared learning.Source: §2.4 · p. 14
  • Context resolves what observation and prior competence leave undetermined—task, relevant history, or physical response—and matters when it changes an executable decision.The historical convergence of demonstration following and physical adaptation joins inference of a new task with inference of an unfamiliar body’s response.Source: §2.4 · p. 15

3 Methods: How Does Context Change Action?

This section examines how action distributions, motion references, predicted consequences, and skill or tool specifications turn contextual evidence into action. It focuses on how these interfaces preserve task requirements, bind evidence to changing scenes, support feasible execution, and retain guidance or procedures for later tasks while neural weights remain fixed.

3 Methods: How Does Context Change Action?

  • The method taxonomy distinguishes evidence sources and forms, context operations such as retrieval, alignment, estimation, and fitting, and execution modes such as replay, tracking, planning, and tool invocation.The four families differ by the carrier that passes between components and its role in the full control path.Source: §3 · p. 15
  • Semantic abstraction can preserve object roles, prerequisites, and skill order, but execution must retain binding constraints such as plate-first order or a required handle grasp [31] [165].A generic pick-and-place request can erase these distinctions.Source: §3 · p. 16

3.1 Context-conditioned policies

  • Context-conditioned policies use retrieval to replay or combine recorded actions, or use interpreted evidence to condition a generator; replay requires compatible control coordinates and archive coverage.A learned generator can transform examples beyond local replay, provided its representation preserves the taught distinction.Source: §3.1 · pp. 16, 20
  • In REGENT’s unseen-MuJoCo-embodiment comparison, Retrieve-and-Play outperforms the frozen contextual transformer, while fine-tuning improves the learned agent [154].Retrieved actions remain useful where the learned interpreter transfers less effectively, separating archive coverage from learned transformation.Source: §3.1 · pp. 16, 17, 20
  • Execution-dependent attention, phase-aligned retrieval, and spatial matching help identify which demonstration evidence applies, while interaction history can reveal uncertain physical response [158] [18] [161] [39] [146].Physical-response inference requires transitions that expose how commands affect the current system; task demonstrations alone may not resolve that uncertainty.Source: §3.1 · pp. 18–19
  • KAT finds contextual inference competitive with few examples, while policy training becomes stronger with more; the crossover differs against an image-based diffusion policy.Both representation and adaptation mechanism affect the useful demonstration budget.Source: §3.1 · p. 20

3.2 Geometric demonstration transfer

  • Geometric transfer preserves a motion or contact reference for execution to track, with perception establishing correspondence and alignment or retargeting adapting it to the query scene.IMOP transfers a local target pose using invariant-region correspondences and a state router [156]; object-centric and hand-based transfer preserve different motion variables [35] [168].Source: §3.2 · pp. 20–21
  • Object-relative pose transfer places demonstrated end-effector motion in the query scene, assuming the reference stays fixed and the relative motion remains applicable.Shape changes, moving references, or embodiment differences require richer correspondence; choosing the reference object is part of task interpretation.Source: §3.2 · p. 21
  • Geometric transfer shifts the burden to correspondence, reference switching, and feasible tracking; shape variation, lost contact, and independent object motion challenge its applicability assumption.Task-relevant local geometry, object-part relations, and changing contact relations can replace whole-object matching as the basis for transfer.Source: §3.2 · pp. 22–23

3.3 World-model-based control

  • Demo-JEPA’s diffusion action head has higher mean success on familiar behavior grounding, while planning has higher mean success on unseen tasks; the familiar-task advantage does not persist in simulation.The relative value of search and learned decoding depends on task novelty and execution setting.Source: §3.3 · p. 24
  • World models either predict a desired future for action decoding or evaluate candidate actions by their predicted consequences; auxiliary future supervision instead shapes training without requiring deployment-time prediction.Prediction-conditioned control separates task interpretation from physical realization, so control quality depends on forecast accuracy and the action interface.Source: §3.3 · pp. 23, 26

3.4 Skill- and agent-based execution

  • Skill- and agent-based execution uses context to specify operations, arguments, or skill order for a separate executor, allowing code or argument revisions to change behavior without updating neural weights.Specifications can be generated as programs or tool requests, or composed from learned skills.Source: §3.4 · pp. 26–27
  • RAPID’s nonprehensile experiments support a demonstration-conditioned pathway that constructs operations and checks from demonstrations, while its separate LIBERO-Pro experiment constructs programs without demonstrations.Reconstruction and program search contribute to the time and interaction needed to acquire the task.Source: §3.4 · p. 27
  • Skill and program execution must test feasibility and use feedback to revise actions, procedures, or learned capabilities when the scene or unknown physical properties demand it.Skill-affordance estimates predict applicability before execution, while scene feedback and targeted measurements such as mass or stiffness estimates can change what happens next [185] [186] [187]. Recombining available operations differs from learning a missing predictive or motor capability [9], [166], [89], [184].Source: §3.4 · p. 28
  • Carriers and execution operations jointly define transfer assumptions: context may become a future, motion reference, skill, or program, each preserving different constraints for its executor.Predicted futures can guide planning or condition action decoding, while motion references can support tracking or replay.Source: §3.4 · p. 29
  • Choosing an interface relocates generalization to a learned interpreter, matched interaction, future model, or supplied executor; hybrids can combine these assumptions.Direct action inference exploits learned correspondence, geometric transfer exposes contact or relative motion, predictive control exposes consequences, and skill or program interfaces expose ordering and preconditions.Source: §3.4 · p. 29

3.5 Shared mechanisms: correspondence and memory

  • A transferred task must preserve its taught invariant—such as an object relation, contact constraint, process, or intended effect—even when substitute objects require changed motion [101].A new vessel or receiver may require coupled grasp, orientation, position, and height adjustments; different methods use demonstrations, functional keypoints, contact records, or supplied part equivalencies [346], [347], [172], [38].Source: §3.5 · p. 31
  • WAA transfers frozen LIBERO-derived skills to robosuite lifting, stacking, and restacking despite changes in objects, scenes, and viewpoints, grounding poses from the current scene [388].The interface must bind the requirement to current objects, retain its evidence until needed, and express an executable change [edac94].Source: §3.5 · pp. 29, 31
  • Memory must select which evidence matters, preserve it for the relevant decision, and update it when its validity ends; task requirements, physical responses, and completed steps need different rules.Recurrent state and external archives can complement one another, while event records preserve otherwise invisible actions and object tracking updates referents [abb759], [5d984f], [5d21].Source: §3.5 · pp. 32–33
  • Remembering evidence under a learned rule differs from inferring what an unfamiliar task requires memory to retain; µVLA improves held-out tasks with familiar memory needs but yields little gain when the required structure changes [199].Past observations establish what happened, whereas combining retrieved events with imagined states adds what could happen next [45], [209].Source: §3.5 · p. 33
  • Budgeted retrieval selects up to K records above a relevance threshold, permits an empty read, and then uses a fallback policy or information-acquisition step if no useful history is selected.Sequence-aware retrieval can score combinations when one event’s value depends on another.Source: §3.5 · p. 33
  • Contextual control requires decisive evidence to remain recoverable, commands to express the needed change, and execution to work from the state reached under prior instructions.A policy may complete two instructions from their usual starting states yet fail to switch between them midway; transition data and policy refinement can improve switching [108].Source: §3.5 · pp. 34–35

3.6 External knowledge reuse and revision

  • External knowledge reuse retains guidance, callable routines, or revised context and verification procedures, each changing how later tasks use evidence or execute.Guidance changes later inference, routines enlarge the available repertoire, and revised procedures change how evidence is used; these artifacts can coexist [53].Source: §3.6 · pp. 35, 38
  • Show-Harness [31] separates explicit command correspondence, which is easier to read, from arbitrary command meanings that must be inferred through probing.Its VLM selects semantic motion commands while a body-specific interpreter supplies calibrated execution; video can specify task order.Source: §3.6 · p. 36
  • Retained experience supports distinct benefits: WAA [388] makes visual guidance explicit, Teach and Grow [53] evaluates library additions, and Hi-VLA [219] favors prior-episode summaries.WAA's one-demonstration stove study uses an observed activation signal as the success criterion; Teach and Grow's agent-based induction and Experience Memory updates remain proposed; Hi-VLA finds greater benefits from prior-episode affordance summaries than longer within-episode history.Source: §3.6 · p. 37
  • RoboRSI reports 174/600 successes with code on versus 129/600 with code off over 120 tasks and five layouts each.This comparison tests the contribution of retained callable routines; the paper separately describes policy training that consolidates corrective trajectories into neural parameters.Source: §3.6 · p. 37
  • Consolidating experience into neural parameters can teach execution through successful trajectories or responsiveness to feedback through successful teaching exchanges [117].The training target determines which capability is consolidated; LMPC uses teaching exchanges to train responsiveness to later feedback.Source: §3.6 · p. 38

4 Data and Training: How Is Context Use Learned?

This section examines how data and training make earlier evidence useful for later robot decisions, from data sources and annotations to the relationships between teaching and execution. It considers how control intermediates and action representations preserve task distinctions, and how experience contributes when it supplies missing competence, task-defining alternatives, or evidence that changes a later attempt.

4.1 Input sources and context modalities

  • Teleoperation, UMI, egocentric video, and simulation differ in directly observed evidence and action correspondence; support, query, and supervision channels must be specified separately.Language may accompany any source as instructions, examples, corrections, or summaries; a collection channel need not be a policy input.Source: §4.1 · pp. 38, 40, Figure 13 · p. 39

4.2 Training units and context dependence

  • Support channels and action supervision are separate choices: Vid2Robot uses video without support commands [14], whereas ICRT supplies support actions [13], and both supervise query execution.Paired support–query supervision works when the support explains which query behavior is appropriate.Source: §4.2 · p. 42
  • Paired training teaches support-to-query dependence only when pairing preserves the intended task distinction and compatible action meaning, rather than shared labels alone [54] [16] [233].Task-grouped examples can preserve distinctions such as grasp or route preferences, while calibration helps ensure commands realize the intended intervention.Source: §4.2 · p. 42
  • Streams expose prior events without explicit support–query roles, while linked-attempt training uses outcomes or corrections to shape later decisions [6] [10].RoboTTT uses a failed rollout as context but applies action loss only to human corrections [10].Source: §4.2 · pp. 42–44
  • Mean Chamfer error was 9.5 pixels (500 tasks, 20 demonstrations each), 5.1 (1,000 tasks, ten each), and 3.4 (2,000 tasks, five each) with 10,000 drawing demonstrations [54].This controlled comparison indicates that, at a fixed demonstration budget, greater task diversity improved drawing performance.Source: §4.2 · p. 43, Figure 15 · p. 43
  • Unseen-task success rose from 1% to 66% with demonstration prompting and from 0% to 9% with language prompting across 1k–100k training hours.The reported cumulative per-step success measure includes human recovery interventions, mainly for the language baseline.Source: §4.2 · p. 43, Figure 15 · p. 43

4.3 Data coverage and action correspondence

  • Broader data increase behavior coverage, but contextual use depends on annotations linking intended procedures to executable actions across human, robot, or motion-bearing records [56–58, 60].Cross-body pairs need procedural metadata to identify shared requirements and kinematic metadata to describe differing realizations; a shared end effector can reduce correspondence uncertainty [66].Source: §4.3 · pp. 44–45
  • Human video broadens usable recording coverage, but robot supervision inherits reconstruction and retargeting error, making reliability-weighted pseudo-actions valuable and small positional errors consequential near contact [16] [245] [246].Reliable boundaries help preserve order, whereas contact-sensitive transfer requires relative motion and object response [250].Source: §4.3 · pp. 45–46
  • WAM-TTT learns video-to-robot-action correspondence through paired human–robot meta-training, then uses action-free human video for deployment-time fast-weight adaptation [11].Robot supervision establishes the connection during training, while the later teaching input can omit recorded robot commands.Source: §4.3 · p. 47

4.4 Acquisition cost and contextual capability

  • Zero of three video-only plug-reinsertion trials succeeded, compared with two of three using references augmented with aligned end-effector poses, gripper states, and commands [94].This is a combined state–action comparison; the passage does not isolate the contribution of each added signal.Source: §4.4 · p. 47

4.5 Scaling data construction

  • Retrieval, spatial supervision, and task-preserving synthesis add different teaching relationships, so constructed examples must remain tied to the governing procedure and validated for the intended task [80] [161] [176] [261] [262].Retrieved clips must match the intended procedure; spatial supervision can teach where a relation occurs or which interaction geometry is plausible, while simulation tests realization under modeled dynamics.Source: §4.5 · pp. 48–49
  • XR-2 folding success rose from 58% to 93% across three retraining rounds after separating policy actions from human takeovers [244].Corrective collection broadens recovery states represented in training, while contextual recovery also requires retaining the linked failure and feedback so the learner can infer which repair applies.Source: §4.5 · p. 50

4.6 Connecting teaching data to a pretrained policy

  • Teaching must enter through an accessible input, remain distinct in an execution intermediate, and be expressible in the action representation; pairing, context encoding, and action encoding must preserve the same taught distinction [274] [118] [119] [275] [282] [283].The four execution interfaces consume different intermediates—actions, geometric references, futures, or executable specifications—but each must preserve the distinction selected by the evidence.Source: §4.6 · pp. 50–51
  • Retrieval can select training data, construct a demonstration–execution pair, or supply deployment context; only the last directly changes a frozen policy’s input, and useful scaling adds distinctions or evidence that alter later behavior.Broad model exposure describes available prior competence, but does not by itself establish the causal contribution of individual training components to robot behavior.Source: §4.4 · p. 48, §4.6 · p. 51

5 Grounding: How Does Context Guide Recovery?

Recovery preserves the taught requirement while updating the robot’s understanding of the physical state and selecting a feasible continuation. Its value depends on matching evidence and correction to the failure, then retaining any repair with the conditions under which it applies.

5.1 Grounding the task in the current physical state

  • Grounding keeps the taught requirement binding while updating physical state, including whether a transferred segment’s entry conditions still hold.Current observations can establish prerequisites and required contacts; a slip may invalidate a segment even when geometric alignment persists.Source: §5.1 · pp. 51–52

5.2 From failure evidence to corrective execution

  • Failure assessment compares execution with the current taught procedure, while diagnosis distinguishes task ambiguity from physical uncertainty requiring sensing or interaction.Progress and physical acceptability can disagree, so verification must test both the context-defined requirement and acceptable execution.Source: §5.2 · p. 53
  • The useful diagnostic check is the one that distinguishes causes relevant to the next repair, such as a view for occlusion or contact feedback for grasp loss.Affordance Agent Harness verifies evidence and retries before committing an actionable region.Source: §5.1 · p. 51, §5.2 · p. 53
  • Correction must match the diagnosed error and remain executable from the reached state; parameter adjustments can help when the target is already correct.When a frozen policy cannot continue, CoRe chooses a rejoin state and realigns displaced objects before returning control.Source: §5.2 · p. 53
  • FARE compares causal-cache, recovered-prefix, and reset-cache continuations; its ablations find that always searching or committing an unverified prefix can underperform the base policy.A compatible historical prefix must be re-grounded in the current state, and continuation must preserve completed steps while restoring failed prerequisites.Source: §5.2 · p. 54

5.3 Applicability and reuse of corrections

  • Reusable corrections need their outcome and applicable physical conditions retained, since a changed tool or failure cause can invalidate a prior response estimate.Corrective training expands available repairs, whereas contextual recovery selects or revises one from fresh evidence.Source: §5.3 · p. 54
  • Program-based reuse preserves repairs in executable form, with later trials testing their benefit under applicable conditions.ENPIRE uses fixed neural models in its program-revision branch, while Zetta retains critics and recovery routines around a frozen policy.Source: §5.3 · p. 54

6 Applications: What Must Context Resolve?

Applications differ in what context must resolve: task relations, spatial knowledge, procedural state, or physical response. The review uses these demands to distinguish what teaching or feedback must supply, how it can transfer, and when retained experience remains valid.

6 Applications: What Must Context Resolve?

  • Across applications, context is useful when its intermediate preserves the information the task requires, which may concern relations, spatial knowledge, procedural state, or physical response.The supplied evidence and its validity boundaries depend on the application’s consequential uncertainty.Source: §6 · p. 55, §6.5 · p. 61

6.1 Task specification in manipulation

  • Demonstrations matter when they convey a choice left open by a task label or goal image, such as stroke order or a required path.In the three-task laundry study, language is more reliable than behavior demonstrations, with some demonstration-conditioned failures executing the wrong fold.Source: §6.1 · p. 55
  • The binding requirement determines method choice: geometric transfer can expose contact or motion constraints, while skill interfaces expose order and prerequisites.Training pairs based only on endpoint equivalence can erase a valued route or contact requirement.Source: §6.1 · p. 56
  • Manipulation can change task relation and physical response independently, as when a new arrangement changes the request but unfamiliar fabric changes how a taught fold is realized.A new fold on unfamiliar fabric couples both changes; teaching supplies the procedure while action–effect evidence supplies current material response.Source: §6.1 · p. 57

6.2 Navigation and spatial adaptation

  • Navigation context can retain a route, environment facts, an interpretation rule, or a conditional lesson, and each has different validity boundaries.A new goal can reuse a map but replace the route; a new building invalidates local locations, while changed terrain can invalidate a traversal estimate.Source: Table 13 · p. 57, §6.2 · p. 58

6.3 Long procedures and irreversible operations

  • Long procedures require tracking dependencies and distinguishing completed steps from failed prerequisites; irreversible operations also require evidence of completion before deciding whether to retry.A placement may permit local correction, whereas dispensing or fastening can require inspection before another attempt.Source: §6.3 · pp. 58–59

6.4 Adaptation to body and dynamics

  • Body and dynamics adaptation keeps the requested task fixed while interaction reveals how changed bodies, payloads, or objects respond.RopeFormer retains action–response history across trials on unseen ropes with fixed policy weights, while MetaPusher updates a dynamics model and revises its plan.Source: §6.4 · p. 60

6.5 Task changeover, teaching, and reuse

  • Task changeover is valuable when new teaching specifies what changes and repeated operations benefit from retained procedures, with acquisition and verification costs shaping the benefit.Precision and irreversibility determine the verification needed before accepting a new procedure.Source: §6.5 · pp. 60–61

7 Evaluation: When Does Context Help?

This section evaluates when context helps by distinguishing responsiveness to teaching, physical transfer, and improvement from retained experience, while accounting for the costs of acquiring, responding to, and reusing context. It emphasizes controlled interventions and matched comparisons to identify what evidence or capability explains an outcome, and treats retained knowledge and repeated attempts as part of deployment value.

7.1 Context use and transfer across tasks and environments

  • Context interventions must distinguish changed task meaning from appearance changes, and scene-based success can conceal weak teaching use; Show-Harness succeeds in one of 20 episodes without supplied conventions and infers 23.3% of mappings [31].Task interpretation and physical realization require separate observations; an equally informative textual order controls claims about video’s particular advantage.Source: §7.1 · pp. 62–63, Figure 22 · p. 62
  • Transfer tests should specify whether the unseen change concerns a relation, object, body, or observation interface, while separately checking task interpretation and physical completion [100] [101] [172] [347].Object substitutions can retain the taught requirement while changing manipulated or receiving objects; training exposure, correspondence assistance, updates, and retries affect interpretation.Source: §7.1 · pp. 63–64

7.2 Sources of improvement and comparable evidence

  • Reported comparisons link context gains to task alternatives and useful evidence: extra expert episodes often add little [102], while trajectory demonstrations and action conventions provide task-defining information [31] [54].ICRT varies task alternatives within a scene, and NOLO examines scene previews [13] [351].Source: §7.2 · p. 65, Table 16 · p. 66
  • Transfer depends on the execution interface: alignment and replay support MT3’s low-demonstration advantage, while correspondence, task familiarity, and validation conditions affect other methods [22] [36] [347] [390].The comparisons identify represented relations, correspondence assistance, motor competence, and validation beyond the demonstrated configuration as relevant conditions.Source: §7.2 · p. 65, Table 16 · p. 66
  • Retained-experience comparisons distinguish access, trace-based correction, selective history revision, and successor training; cumulative retry success measures a different quantity [88] [117] [195] [392] [402].TraceFlow’s ordering gains coexist with unresolved simulation deficits, and FARE’s always-search variant underperforms its base policy.Source: §7.2 · p. 65, Table 16 · p. 66

7.3 Physical realization and learning across attempts

  • Physical success and teaching adherence should be measured separately, with independent physical and memory resets distinguishing task progress, retained evidence, and extra retries across attempts.For a prescribed contact, such as grasping a handle, evaluation should measure both successful transport and preservation of handle contact; recovery tests should connect failure detection, revision, and outcome.Source: §7.3 · pp. 68–69
  • RoboRecover replays action prefixes across 2,000 RoboTwin and LIBERO scenarios to test continuation from reconstructed deviations [400]; a proposed extension compares retained and cleared prefixes.Initializing candidate policy history at the recovery point isolates continuation from the reconstructed off-nominal state; compare prefix conditions at that same state with the same continuation budget.Source: §7.3 · p. 69

7.4 Teaching effort, response delay, and reuse cost

  • Vary task-defining context independently of search budget to test learning beyond familiar-scene recognition, then use physical execution and held-out relation tests to assess transfer [315] [316].Context interventions establish dependence on task-defining evidence; physical execution and held-out relation tests establish how far requirements transfer.Source: §7.4 · pp. 69–70
  • Measure acquisition end to end as changeover time to repeatable quality, separating elapsed time, human labor, robot interaction, and resets [89] [317].Acquisition includes instruction preparation, demonstrations, exploration, and validation; ENPIRE requires physical trials and verification despite fixed neural weights [89], while SAIL illustrates preparation costs beyond inference [317].Source: §7.4 · pp. 69–70
  • SimpleMemVLA reduced feedback-to-action latency from 1.02 to 0.68 seconds with identical outputs; ReSteer makes feedback arrival explicit [45] [108].Inference time, command rate, and servo frequency capture different parts of the delay from informative feedback to the affected command.Source: §7.4 · p. 70
  • Common task outcomes can expose the contributions of training, supplied context, adaptive neural state, and retained knowledge, while acquisition and reuse have distinct resource costs.Across repeated batches, accepted output and intervention time quantify whether acquisition costs are repaid; changing a tool or requirement reveals the cost of obsolete knowledge.Source: §7.4 · p. 70
  • Teaching effort, feedback-to-action delay, and cost per accepted execution connect contextual learning to deployment value across repeated attempts.These measures capture different parts of the value of teaching and reusing context over repeated tasks.Source: §7.4 · p. 70

8 Future Directions: What Limits Transfer and Improvement?

The future agenda asks how contextual learning can infer unfamiliar rules, preserve physical constraints, revise experience when conditions change, and improve subsequent task learning. It ties these objectives to training, representation, experience, and behavioral tests.

8.1 From context dependence to a reusable learning rule

  • Proposed tests hold objects and motor skills fixed while varying familiar-task selection, known-relation composition, and inference of an unseen rule.For rule inference, examples must distinguish candidate mappings.Source: Table 18 · p. 70, §8.1 · p. 70

8.2 Instruction tuning for more complex contextual learning

  • Training should test whether composed orders and counts survive execution, targeted corrections preserve unaffected requirements, and clarification resolves missing information before action.Changing correction timing, scope, and precedence tests whether revisions are appropriately selective.Source: §8.2 · pp. 71–72

8.3 Preserving taught relations across physical change

  • Selective invariance retains binding requirements while permitting valid substitutions; impossible relations may need clarification, and out-of-repertoire motions require separate motor learning.In navigation, strategies may transfer across buildings while remembered locations remain layout-specific.Source: §8.3 · pp. 72–73

8.4 Retaining teaching while revising obsolete experience

  • Retained evidence must support future reinterpretation, not only current task meaning; obsolete corrections may interfere, so tests should check reuse and revision across changing layouts.A later correction may make a previously incidental contact or pause decisive.Source: §8.4 · pp. 73–74

8.5 Physical recursive self-improvement

  • Experience can be stored as retained history, programs, or weight updates, but the decisive test is whether successors acquire new tasks more effectively and gains persist across rounds.Table 19 compares update targets, operations, and reusable outputs.Source: §8.5 · p. 74, Table 19 · p. 74
  • LMPC links successful interactions across sessions to predicting teaching exchanges and searching for shorter routes; test-task success rises from 39.4% to 66.3%, while successful-session mean turns fall from 2.4 to 1.9 [117].The reported rollout variant searches teaching exchanges for a shorter route to success.Source: §8.5 · p. 74
  • Collective knowledge evolution exchanges a verified placement relation across robots while allowing grasps and balance control to change with embodiment.Testing both transfer directions and the teaching and practice costs measures whether exchange improves group learning.Source: §8.5 · p. 75
  • The agenda distinguishes example diversity, motion coverage, and model scale, while emphasizing composition, selective correction, information acquisition, and better learning from subsequent teaching.Its central later-cycle outcome is a successor that learns more effectively from new teaching.Source: §8.5 · p. 75

9 Conclusion

The conclusion frames robot ICL as preserving task-relevant information from contextual evidence through execution. It calls for transferable learning and evaluation of both later acquisition and the effort required.

9 Conclusion

  • Robot ICL addresses the gap between reusable competence and knowing how to apply it, using four interfaces: contextual action inference, geometric transfer, consequence prediction, and skill or program execution.Their benefits depend on compatible representations, applicable geometry, realizable predictions, and sufficient executor capabilities.Source: §9 · p. 76
  • Evaluation asks whether changed teaching redirects behavior, requirements survive physical change, and retained experience helps later attempts, with acquisition costs and motor competence shaping the scope of gains.Correspondence and memory connect evidence across situations and time; grounding establishes whether it remains usable.Source: §9 · p. 76
  • The next challenge is transferable inference, selective revision, and discarding experience whose conditions no longer hold; recursive improvement and collective exchange are judged by later learning and total effort.Collective knowledge evolution asks whether verified exchanges improve learning across embodiments.Source: §9 · p. 76

A Complete Reference Index

The reference index links Figure 9 to references grouped by year and method family, with accompanying bibliographic and classification details.

A Complete Reference Index

  • Table A1 links every reference to Figure 9 and groups entries by first public year.Full titles, dates, primary sources, and classification details accompany the reference data.Source: A Complete Reference Index · p. 96, Table A1 · p. 96
  • The index identifies context-conditioned policies, geometric transfer, world-model-based control, and skill or agent execution as method families.Other groups cover data, evaluation, foundations, and training references.Source: Table A1 · p. 96
  • Table A1 continues its complete reference assignments in a subsequent table segment.Source: Table A1 · p. 97

B Method and Resource Comparisons

The appendices compare correspondence, memory, revised artifacts, and data resources to connect retained information and supervision with contextual control.

B Method and Resource Comparisons

C Related Survey Perspectives

Table C1 compares related surveys by scope and organizing principle, situating ICL for robots through the evidence and interfaces that change execution.

Loading 2609.36012v1…