PaperlayerView exact revision on arXiv

Source-linked AI summary

LMAct: A Benchmark for In-Context Imitation Learning with Long Multimodal Demonstrations

Anian Ruoss, Fabio Pardo, Harris Chan, Bonnie Li, Volodymyr Mnih, Tim Genewein

arXiv:2412.01441v3cs.AIcs.LG

TL;DR

The paper examines whether expert demonstrations help language models act effectively in interactive tasks, and presents a multimodal benchmark for testing this in long contexts. Models often remain below expert performance, with demonstration count making little difference in most cases.

  • Problem

    The paper asks whether the gap between language models’ factual knowledge and effective decision-making reflects prompt mismatch or a deeper difficulty, and whether expert trajectories can help.

    Source: §1 · p. 1

  • Method

    The authors evaluate eight frontier LMs across six interactive tasks, varying demonstrations in a unified zero-, few-, and many-shot benchmark.

    Source: §1 · p. 2

  • Results

    Despite hundreds of demonstrations and million-token contexts, models often miss expert performance; performance is mostly demonstration-insensitive, though some cases show in-context learning.

    Source: §1 · p. 2, §5 · p. 9

  • Takeaways & Limitations

    The released benchmark, demonstrations, and evaluation code provide a unified resource for evaluating zero-, few-, and many-shot imitation learning.

    Source: §1 · p. 2

    Closed-source APIs and prompt sensitivity constrain reproducibility; the authors describe the reported results as a lower bound on model performance.

    Source: §4 · p. 8

Abstract

from arXiv · show

In this paper, we present a benchmark to pressure-test today's frontier models' multimodal decision-making capabilities in the very long-context regime (up to one million tokens) and investigate whether these models can learn from large numbers of expert demonstrations in their context. We evaluate the performance of Claude 3.5 Sonnet, Gemini 1.5 Flash, Gemini 1.5 Pro, Gemini 2.0 Flash Experimental, GPT-4o, o1-mini, o1-preview, and o1 as policies across a battery of simple interactive decision-making tasks: playing tic-tac-toe, chess, and Atari, navigating grid worlds, solving crosswords, and controlling a simulated cheetah. We study increasing amounts of expert demonstrations in the context $\unicode{x2013}$ from no demonstrations to 512 full episodes. Across our tasks, models rarely manage to fully reach expert performance, and often, presenting more demonstrations has little effect. Some models steadily improve with more demonstrations on a few tasks. We investigate the effect of encoding observations as text or images and the impact of chain-of-thought prompting. To help quantify the impact of other approaches and future innovations, we open source our benchmark that covers the zero-, few-, and many-shot regimes in a unified evaluation.

1. Introduction

The paper asks whether large numbers of expert demonstrations can help language models turn task knowledge into effective decisions. It introduces a multimodal benchmark spanning interactive tasks and evaluates models from zero-shot through many-shot settings.

1. Introduction

  • The paper addresses the gap between LMs’ factual task knowledge and effective decision-making, and asks whether expert trajectories can unlock that capability.Source: §1 · pp. 1–2
  • The benchmark evaluates eight frontier LMs on six interactive tasks, including games, grid navigation, crosswords, and cheetah control.Source: §1 · p. 2
  • Even with prompt optimization, frontier LMs fail to reach expert performance on Atari, chess, and DM Control; some models approach it on crosswords, grid world, and tic-tac-toe.All models beat random-action baselines except on Atari.Source: §1 · p. 2
  • Performance is mostly independent of demonstration count, although some models show strong in-context learning in some cases.The study varies demonstrations from zero to as many as 512 episodes, subject to model and task limits.Source: §1 · p. 2
  • In a replay control, models except o1-mini perform well when asked to repeat the single demonstrated episode.Source: §1 · p. 2
  • The authors release the benchmark, expert demonstrations, and evaluation code for unified zero-, few-, and many-shot evaluation.Source: §1 · p. 2

2. Methods

The methods evaluate frontier language models as policies across six interactive environments. Prompts combine expert demonstrations with a growing test trajectory, and evaluation uses unseen initial states.

2.1. Models

  • Eight closed-weights frontier models are evaluated with different temperature and output-token settings for reasoning models.Reasoning models use a maximum output length of 8192 tokens, versus 2048 for the others.Source: §2.1 · pp. 2–3

2.2. Environments

  • The six-task suite spans games, navigation, crosswords, and cheetah control, with varied state encodings and expert trajectories.Source: §2.2 · pp. 3–4

2.3. Prompt

  • Prompts pair fixed demonstration episodes with a growing evaluation trajectory; legal actions and chain-of-thought prompts are selected per model and task.Source: §2.3 · p. 4

2.4. Evaluation Protocol

  • Scores average 100 episodes with initial states differing from the demonstrations; context limits make demonstration counts depend on model, task, and representation.If a model generates an illegal action, the protocol substitutes a uniformly sampled legal action.Source: §2.4 · p. 4

3. Results

Across tasks, models often fall short of expert scores even in their best settings, with grid worlds, tic-tac-toe, and crosswords as exceptions. More demonstrations help some models on some tasks, but often have little effect or degrade performance.

3. Results

  • The evaluation varies demonstration counts and observation formats; ablations of chain-of-thought and legal-action prompts use one episode, and Claude 3.5 Sonnet is not swept across demonstration counts.Claude 3.5 Sonnet is also omitted from Atari because its image limit is reached by a single demonstration episode.Source: §3 · p. 4, §3.2 · p. 5

3.1. Best Scores Per Model/Task

  • Most models fail to match expert scores in their best settings; exceptions include grid worlds, tic-tac-toe for o1 models, and crosswords for o1-preview and o1.All models outperform the random-action baseline except on Atari – Phoenix.Source: Figure 2 · p. 5, §3.1 · p. 5

3.2. In-Context Imitation Learning

  • On Phoenix, one demonstration mildly helps almost all evaluable models, but none beats random; chess models rarely beat or draw Stockfish and do not benefit from more demonstrations.Gemini 1.5 Flash performs best on Phoenix; PGN yields the best chess results, but GPT-4o still loses most games.Source: Figure 3 · p. 5, §3.2 · pp. 5–6, Figure 4 · p. 6
  • Most models' crossword performance barely changes with demonstrations, while o1-preview and o1 nearly solve simple puzzles; on tic-tac-toe, o1 models with ASCII reach expert performance.Other models generally struggle to beat random in tic-tac-toe; o1-mini and o1-preview improve with more demonstrations, reaching expert performance at 256 episodes.Source: §3.2 · pp. 6–7, Figure 5 · p. 6, Figure 6 · p. 7
  • In grid worlds, Gemini improves with more demonstrations for coordinate and RGB observations, while most models deteriorate with more demonstrations on ASCII.o1 performs very well across most settings and is near-perfect with ASCII and coordinate observations.Source: §3.2 · p. 7, Figure 7 · p. 7
  • On cheetah control, Gemini 1.5 Pro and Gemini 2.0 Flash Experimental score highest at roughly half the expert score, while more than two demonstrations generally degrade performance.Models other than o1-mini learn a non-trivial policy from one or two episodes, but fail to leverage more episodes for further gains.Source: §3.2 · pp. 7–8, Figure 8 · p. 8

4. Discussion & Related Work

The paper finds that more demonstrations usually do not improve performance, although replaying an exact demonstrated episode works for all models except o1-mini. It also identifies demonstration-related performance declines and outlines evaluation limits and directions for harder tests.

4. Discussion & Related Work

  • LMAct complements zero-shot BALROG by evaluating long-context in-context imitation learning, though its current grid world has no obstacles.The paper presents many demonstrations as a potential way to improve on few-shot learning, drawing on prior non-interactive results and models’ ability to imitate patterns in context.Source: §4 · p. 8
  • Performance is largely independent of demonstration count in most cases, but models except o1-mini perform well when replaying the exact episode shown.The authors leave open whether models fail to infer desired behavior from demonstrations or to use dense long contexts effectively.Source: §4 · p. 8
  • Closed-source API evaluation limits control over data processing and makes quantitative reproducibility uncertain as models may be updated.The authors also cannot rule out better results from small prompt changes or tokenization issues in some observation formats, so they characterize current results as a lower bound.Source: §4 · p. 8
  • LMAct’s findings may inform future work on transferring these skills to real-world challenges, although direct transfer is beyond this paper’s scope.The environments are simpler than real-world scenarios such as robotics but test capabilities the authors describe as fundamental to agentic behavior.Source: §4 · p. 8
  • On Cheetah Run and ASCII grid navigation, performance deteriorates as expert demonstrations increase, with analyses pointing to increasingly suboptimal rather than illegal actions.The authors call this pattern “in-context interference”; its cause remains unresolved.Source: §4 · pp. 8–9
  • The benchmark primarily tests fully observable tasks, and the authors judge current frontier models to be at beginner level on these simpler tasks.They identify partial observability and in-context reinforcement learning as future research directions, while noting that hidden expert information can create a self-delusion problem for imitation.Source: §4 · p. 9

5. Conclusion

The evaluation finds that frontier models often fail to reach expert performance on interactive tasks, even with hundreds of demonstrations and contexts up to one million tokens. LMAct is released as a benchmark for measuring progress toward more capable agents.

5. Conclusion

  • Even with hundreds of demonstration episodes and contexts up to one million tokens, models often struggle to reach expert performance on the evaluated tasks.The authors conclude that models often fail to translate factual knowledge about task strategies into effective decision-making.Source: §5 · p. 9
  • LMAct evaluates multimodal in-context imitation learning on interactive decision-making tasks that are simple for humans but challenging for state-of-the-art language models.The paper describes its open-source benchmark as a yardstick for measuring progress toward more generally capable agents.Source: §5 · p. 9
  • The authors identify comparisons with fine-tuning, retrieval-based methods, and reward-conditioning as useful future work.They state that direct transfer from benchmark findings to real-world challenges is beyond this paper’s scope.Source: §5 · p. 9

Impact Statement

LMAct benchmarks current frontier models’ interactive decision-making and serves as a yardstick for future progress, while the paper notes the potential positive and negative effects of automating intellectual labor. It distinguishes its long-context multimodal imitation-learning focus from prior work and covers zero-, few-, and many-shot evaluation in one benchmark.

Impact Statement

  • The benchmark reports the current state of frontier LMs and provides a yardstick for measuring future progress, rather than proposing capability-improvement methods.The impact statement frames intellectual-labor automation as having potentially positive and negative societal and socioeconomic effects.Source: Impact Statement · p. 9

A. Related Work

  • Prior work describes in-context learning in LLMs and explores sequential predictors and embodied agents adapting to tasks through context.The cited examples include decision transformers, GATO, an adaptive agent, and SIMA.Source: A. Related Work · p. 15
  • Researchers have explored fine-tuning and in-context demonstrations for LLM- and VLM-based agents, with observation representation and eliciting decision-making remaining open questions.The cited work also considers whether observations should include rewards as well as state-action sequences.Source: A. Related Work · p. 15
  • Many-shot prompts improve pretrained-LM performance on non-interactive tasks, while LMAct examines hundreds of full episodes for interactive decision-making.The authors describe their results as an upper baseline for retrieval-based methods and note the high computational cost of querying long contexts.Source: A. Related Work · p. 15
  • Specialized-agent research develops agents with in-context learning, whereas LMAct primarily benchmarks existing general-purpose frontier LMs.How insights from specialized-agent research might enhance pretrained models remains open.Source: A. Related Work · pp. 15–16
  • LMAct uniquely emphasizes long-context imitation learning in multimodal interactive environments and unifies zero-, few-, and many-shot evaluation.Unlike BALROG, which evaluates zero-shot performance, LMAct covers all three regimes in a unified evaluation.Source: A. Related Work · p. 16

B. Experimental Details

The section introduces additional experimental setup details.

B.1. Models

The experimental details describe model-specific context and image constraints, state representations, demonstration construction for Atari and cheetah control, and the prompt components. These choices shape which models and environments can be evaluated and how demonstrations are presented.

B.1. Models

  • Claude 3.5 Sonnet’s 100-image limit prevents in-context Atari evaluation, and its low monthly token limit also restricts the evaluation.Its maximum context is 200k tokens; a single episode can reach 100 images.Source: §B.1 · p. 16
  • Gemini 1.5 models support 10-million-token contexts but are evaluated up to 1 million tokens because longer prompts are prohibitively costly; Gemini 2.0 Flash Experimental supports 1 million input tokens.Gemini images consume a fixed 258 tokens; Gemini 2.0 Flash Experimental supports 8k output tokens.Source: §B.1 · p. 16
  • GPT-4o has a 128k-token context and a 250-image limit, with image token use depending on automatic low- or high-resolution processing.Low resolution uses 85 tokens; high resolution adds 170 tokens per 512px × 512px tile.Source: §B.1 · p. 16
  • o1-mini and o1-preview are text-only and therefore are not evaluated on Atari; o1 supports a 200k-token context and up to 100k output tokens.Source: §B.1 · p. 16

B.2. Environments

  • The benchmark varies state representations by environment; chess, for example, uses ASCII, FEN, PGN, and RGB images.The formats are illustrated across the environments.Source: §B.2 · p. 17, Figure A3 · p. 17
  • Atari demonstrations come from the 256 highest-scoring of the last 2,048 Gato trajectories, ranked by reward in the first 400 frames.The selected demonstrations average 459.9 return over those frames; evaluation covers the first 100 steps, with manually varied initial no-ops.Source: §B.2 · pp. 17–18
  • Cheetah demonstrations are the 1,000 highest-scoring replayed trajectories from the last 10,000 Gato episodes, ranked by reward over the first 100 steps.The trajectories are replayed because MuJoCo physics changes make the original dataset rewards mismatch the current environment; selected demonstrations average 37.4 return.Source: §B.2 · p. 18

B.3. Prompts

  • The frozen and dynamic parts of the evaluation prompt are illustrated separately in Listings 1 and 2.Source: §B.3 · p. 18

C. Additional Results

The section presents additional results and ablations from the experimental evaluation.

C.1. Ablating the Maximum Sample Length

The sample-length ablation finds that most tested models perform best with short outputs, while o1-mini and o1-preview need much larger output budgets. The authors select 8192 tokens for the o1 models as a cost–performance compromise.

C.1. Ablating the Maximum Sample Length

  • The evaluation prompt appends the evolving trajectory to a frozen prompt containing expert demonstrations, with observation and action placeholders replaced by actual inputs.The illustrated frozen prompt contains eight ten-step demonstration episodes with RGB observations.Source: §C.1 · p. 19
  • 32 output tokens suffice for Claude 3.5 Sonnet, GPT-4o, Gemini 1.5 Flash, and Gemini 1.5 Pro; o1-mini and o1-preview require 4096–8192 tokens for optimal performance.Claude 3.5 Sonnet and GPT-4o benefit from 64 tokens on RGB observations; below 4096 tokens, the o1 models degrade sharply.Source: §C.1 · p. 20
  • 8192 tokens is the chosen maximum sample length for o1 models, trading off cost and performance after illegal actions rise as their output budgets fall.The authors note that the o1 models incur substantially higher computational cost than the other models.Source: §C.1 · p. 20

C.2. Replaying a Demonstration Episode

The replay experiment tests whether models can retrieve and copy actions from a demonstration episode, separate from learning a policy for a changing environment. Most models replay well, but o1-mini struggles in most settings.

C.2. Replaying a Demonstration Episode

  • The control experiment uses teacher forcing: evaluation states follow the demonstration rather than model actions, and performance is measured by action-match accuracy, not return.It evaluates 100 episodes with one demonstration each and omits chain-of-thought prompting and legal-action lists.Source: §C.2 · p. 20
  • Most models replay demonstration actions successfully, with somewhat lower accuracy deeper in episodes; o1-mini struggles in most settings.Gemini 1.5 Flash also fails on tic-tac-toe with RGB observations.Source: §C.2 · p. 20
  • Observation type is theoretically irrelevant in replay because the model could count evaluation observations and select the corresponding demonstration action.The grid-world sample figure shows ASCII, coordinate, and RGB observation formats.Source: §C.2 · p. 21, Figure A6 · p. 21

C.3. Illegal Actions

The appendix tracks illegal actions separately because invalid outputs are replaced with randomly sampled legal actions and evaluation continues. Their frequency varies by model, task, observation type, and output budget.

C.3. Illegal Actions

  • On grid-world navigation with coordinate observations, o1-preview produces almost 100% illegal actions with two or more demonstrations; Gemini 1.5 models reach about 60% illegal actions on tic-tac-toe RGB.Invalid actions generally trigger a random legal-action substitution and the episode continues, except in crosswords.Source: §C.3 · p. 21
  • For grid-world navigation with one demonstration, o1-mini and o1-preview show sharply more illegal actions when maximum output length falls below 8192 tokens.The figure compares average score and illegal-action percentage across output lengths; the o1 models need 4096–8192 tokens for best performance.Source: Figure A7 · p. 22
  • For Phoenix Atari, all tested models perform best without legal-action lists and without chain-of-thought prompting.Source: Table A1 · p. 23

C.4. Hyperparameter Ablations

The ablations select prompting settings separately for each model and task, while generally retaining past evaluation actions because including the full trajectory usually improves performance. Claude 3.5 Sonnet is an exception without a clear overall trend.

C.4. Hyperparameter Ablations

  • Legal-action lists and chain-of-thought have task- and model-dependent effects: chess models generally benefit from legal actions, while most cheetah models do not benefit from either setting.For crosswords, legal-action lists are infeasible because they would include thousands of possible words; chain-of-thought helps some models but not all.Source: §C.4 · p. 23, Table A3 · p. 27, Table A4 · p. 28

C.4.1. INCLUDING PAST ACTIONS

  • Including past evaluation actions generally improves performance across models, although Claude 3.5 Sonnet shows no clear overall trend.For Claude 3.5 Sonnet, past actions help with coordinate observations but hurt with RGB; they are mostly immaterial with ASCII.Source: §C.4.1 · p. 23, Table A11 · p. 37
  • Most models replay chess demonstrations well across observation types, but o1-mini degrades near episode end and most models struggle with the last PGN actions.Source: Figure A9 · p. 24
  • In tic-tac-toe RGB replay, Gemini 1.5 Flash fails completely, while Claude 3.5 Sonnet and Gemini 1.5 Pro struggle at the first step.Source: Figure A13 · p. 29
Loading 2412.01441v3…