Source-linked AI summary
In-Context Iterative Policy Improvement for Dynamic Manipulation
Mark Van der Merwe, Devesh Jha
TL;DR
Dynamic manipulation can be difficult to address through textual or visual inputs alone, motivating an approach that iteratively adjusts policy parameters from prior interactions. The paper reports that in-context policy improvement outperforms alternatives across simulated tasks and a physical robot in the low-data regime.
Problem
Dynamic manipulation involves tasks that are not readily reflected in textual or visual inputs, while physical properties can be difficult to observe directly.
Source: §1 · p. 1Method
The method uses a pretrained LLM as an improvement operator that predicts parametric policy adjustments from prior interactions and in-context examples.
Source: §1 · p. 2, §4.1 · p. 5Results
Takeaways & Limitations
The results add to evidence for in-context learning’s utility in robotics and motivate further investigation with general and robotics-specific transformer models.
Source: §6 · p. 8Collecting the policy improvement dataset required by the method may be difficult for complex tasks.
Source: §6.1 · p. 9
Abstract
from arXiv · showhide
Attention-based architectures trained on internet-scale language data have demonstrated state of the art reasoning ability for various language-based tasks, such as logic problems and textual reasoning. Additionally, these Large Language Models (LLMs) have exhibited the ability to perform few-shot prediction via in-context learning, in which input-output examples provided in the prompt are generalized to new inputs. This ability furthermore extends beyond standard language tasks, enabling few-shot learning for general patterns. In this work, we consider the application of in-context learning with pre-trained language models for dynamic manipulation. Dynamic manipulation introduces several crucial challenges, including increased dimensionality, complex dynamics, and partial observability. To address this, we take an iterative approach, and formulate our in-context learning problem to predict adjustments to a parametric policy based on previous interactions. We show across several tasks in simulation and on a physical robot that utilizing in-context learning outperforms alternative methods in the low data regime. Video summary of this work and experiments can be found https://youtu.be/2inxpdrq74U?si=dAdDYsUEr25nZvRn.
1 Introduction
The introduction motivates in-context learning as a way to improve dynamic manipulation policies, where physical properties may be hard to observe and dynamics can make task-specific solutions costly. It proposes iterative policy updates from prior interactions and reports strong low-data performance across simulation and a physical robot.
1 Introduction
- The work explores whether in-context learning can generalize to dynamic manipulation, a task class not readily represented through textual or visual inputs alone.The motivation draws on evidence that in-context learning can extend to patterns beyond standard training domains.Source: §1 · p. 1
- Dynamic manipulation can expand workspace and improve efficiency, but hidden physical properties, higher dimensionality, and complex dynamics make general solutions difficult.Existing systems have largely been limited to single-task solutions that collect large datasets and train task-specific models.Source: §1 · pp. 1–2
- The method predicts adjustments to a parametric policy from previous interactions, using a small dataset of improvement labels from similar tasks as in-context examples.Previous interactions and policy parameters are tokenized as inputs to the improvement operator.Source: §1 · p. 2
- In-context policy improvement outperforms in-weights reasoning and alternative in-context policy approaches, and compares favorably with policy optimization methods in the low-data regime.The reported evaluation spans tasks in simulation and on a physical robot.Source: §1 · p. 2
2 Related Work
Related work distinguishes in-context learning from learning stored in model weights, reviews LLM-enabled robotics, and positions this work against prior in-context policy improvement and transformer adaptation.
2.1 Large Language Models and In-Context Learning
- In-context learning uses examples in the prompt without weight updates, and has been observed to generalize beyond the training domain, although its mechanisms remain unsettled.Proposed explanations include induction heads and exemplar-based generalization; findings on randomized labels complicate the picture.Source: §2.1 · pp. 2–3
2.2 LLM for Robotics
- Most LLM robotics methods exploit pretrained-model reasoning for planning or control, while focusing largely on quasi-static tasks rather than low-level dynamics.The authors motivate in-context learning because dynamic information is unlikely to be explicit in training data or text.Source: §2.2 · p. 3
2.3 In-Context Learning for Robotics
- Unlike sequence improvement [6], this work fits a policy improvement operator in context; unlike transformer training from scratch [28] [29], it uses a pretrained model.The passage also notes that behavior cloning [27] is difficult when task parameters are not available a priori.Source: §2.3 · p. 3
3 Problem Formulation
The paper formulates dynamic manipulation as choosing parameters for a rollout on a task drawn from a distribution, then details five tasks across simulated and physical setups.
3 Problem Formulation
- The objective is to identify the policy parameters that minimize task cost for a task drawn from a distribution of physical settings or goals.The environment returns a state trajectory from executing the policy, which the task cost evaluates.Source: §3 · p. 3
3.1 Tasks
- The evaluation uses five tasks and three robot setups: two simulated and one physical, with parametric motions as policies.Task setups are shown in Fig. 1.Source: §3.1 · p. 3
3.1.1 Slide Task (slide)
- In the simulated slide task, the robot strikes a puck toward a fixed goal while puck radius and surface friction vary unbeknownst to the policy.The state is the puck center’s planar location, and the cost is distance to the goal.Source: §3.1.1 · p. 4
3.1.2 Slide Goal-Conditioned Task (slide-gc)
- The slide goal-conditioned task varies the puck’s goal configuration while holding puck radius and friction fixed.It uses the slide task’s cost and a modified approach-angle parameter to ensure contact.Source: §3.1.2 · p. 4
3.1.3 Rope Swing Task (rope-swing)
- In the simulated rope-swing task, the robot swings a rope tip through a fixed location while rod and rope lengths vary.The state is the rope end’s location in the swing plane; the policy adjusts angular velocity and two joint motions.Source: §3.1.3 · p. 4
3.1.4 Rope Swing Goal-Conditioned Task (rope-swing-gc)
- The rope-swing goal-conditioned task fixes rod and rope lengths and instead varies the goal configuration.It retains the rope-swing policy and task-cost parameterization.Source: §3.1.4 · p. 4
3.1.5 Real Roll Goal-Conditioned Task (roll-gc-real)
- The real goal-conditioned roll task uses a MELFA-Assista robot to strike a cue ball toward a sampled goal tracked in a top-down camera view.Its state is the ball’s pixel location, and it uses the slide goal-conditioned task’s cost and policy parameterization.Source: §3.1.5 · p. 4
4 Method
A first attempt may fail when task parameters are not readily observable or dynamics are complex, motivating iterative correction from system interactions. The method uses an LLM prompt with improvement examples and current execution information to predict policy adjustments.
4.1 Policy Improvement Operator
- ICPI prompts a pretrained LLM with the query and improvement dataset to predict policy updates, fitting the operator in one forward pass without model design or training.The input is the current policy and state trajectory; the output is the policy change.Source: §4.1 · p. 5
4.2 Tokenization
- Policy parameters and updates become numeric-character tokens, while trajectories are encoded as relative goal error at a task-specific time.For slide and roll, the selected time is the final timestep; for rope-swing, it is the timestep closest to the goal.Source: §4.2 · pp. 5–6
4.3 Dataset Example Selection
- A k-D tree retrieves 20 examples using normalized policy and error vectors, and the examples are supplied in decreasing-distance order.The normalization makes policy and error terms comparable in scale across dimensions.Source: §4.3 · p. 6
4.4 Dataset Construction
- Each improvement label is the difference between a solution policy and a nearby execution; solutions may come from demonstrations, reinforcement learning, or brute-force search.The authors describe this as distilling progress from another, more laborious algorithm into a sequence-to-sequence task.Source: §4.4 · p. 6
- For simulated slide and rope-swing, the authors use brute-force search; for real roll, they generate labels through hindsight goals from a guiding rollout.For real roll, alternative policy parameters are sampled and their errors are computed relative to the guiding execution’s final state.Source: §4.4 · p. 6
5 Results
Across iterative policy-improvement comparisons, ICPI performs consistently across tasks and reaches better costs than random shooting and piecewise-linear modeling, including on a real robot. The experiments also compare data-efficient baselines and find that model and input representation choices affect performance.
5.1 Iterative Policy Improvement
- The evaluation focuses on data-efficient baselines because state-of-the-art dynamic-manipulation methods generally require large datasets.Source: §5.1 · p. 6
5.1.1 Baselines
- Baselines included random policy offsets, Bayesian optimization, nearest-neighbor and piecewise-linear operators, and LLM reasoning and sequence-improvement variants.Source: §5.1.1 · pp. 6–7
5.1.2 Policy Improvement Results
- ICPI consistently outperforms random shooting and converges to better results than piecewise-linear modeling, including on the real robot.The comparison includes task-cost convergence across three tasks and reports that ICPI was the only method to perform consistently well across all tasks.Source: Figure 3 · p. 7, §5.1.2 · pp. 7–8
5.2 Design Ablations
- Across ablations, gpt-4o seems to outperform gpt-3.5-turbo and gpt-4o-mini, while relative-error input beats separate state and goal inputs.Source: Table 2 · p. 8, §5.2 · p. 8
6 Discussion
The discussion presents ICPI as a sample-efficient approach to dynamic manipulation and reports that in-context learning outperforms alternative policy-iteration methods in simulation and on a real robot. It also identifies costs and scope limits that motivate further study.
6 Discussion
- In-context learning outperforms alternative policy-iteration methods in simulation and on a real robot, while LLMs struggle with in-weights reasoning on dynamic tasks.The authors hope this work motivates further investigation with robotics-specific and non-robotics transformer models.Source: §6 · p. 8
6.1 Limitations
- Large pretrained models add computational, financial, and environmental overhead, and opaque model details make the impact of model choice difficult to understand.Collecting the policy-improvement dataset may also be difficult for complex tasks; self-play and demonstrations could ease the burden.Source: §6.1 · p. 9
- Feature selection mattered for task success, and the study's relatively low-dimensional inputs and outputs may limit system dexterity.The authors leave broader study of task dimensionality and feature extraction for future work.Source: §6.1 · p. 9
A.1 Example Prompts
The appendix presents prompt and response examples for the simulated slide task, including the proposed method and LLM baselines. It shows how each approach frames inputs and requests actions or outputs.
A.1 Example Prompts
- The examples cover the simulated slide task and include the proposed method alongside other LLM baselines.Source: §A.1 · p. 13
A.1.1 In-Context Policy Improvement (Ours)
- The proposed method uses a task-agnostic header fixed across experiments, with further prompt construction details in Section 4.Source: §A.1.1 · p. 13
- Its prompt frames examples as INPUTS;OUTPUTS lines and requests an estimate of three space-separated output values.The appendix includes an example input and a three-value response.Source: §A.1.1 · p. 13
A.1.2 In-Context Sequence-Improvement [6]
- The sequence-improvement baseline follows Mirchandani et al. [6], prompting with a randomly sampled improved cost.Source: §A.1.2 · p. 13
- Its example prompt also uses INPUTS;OUTPUTS lines and requests three space-separated output values.A sample response is shown.Source: §A.1.2 · p. 13
A.1.3 In-Weights Reasoning
- The in-weights baseline supplies task, setting, action-space, and prior-interaction descriptions to ground the model's reasoning in the task.Source: §A.1.3 · pp. 13–14
- For the slide task, the prompt describes a one-action puck slide using bounded polar motion angle, distance, and duration, with actions clipped and interpolated into tool locations.Source: §A.1.3 · p. 14
- The illustrated in-weights response is a three-value action: 0.000 0.350 0.780.Source: §A.1.3 · p. 14