PaperlayerView exact revision on arXiv

Source-linked AI summary

Large Language Models as General Pattern Machines

Suvir Mirchandani, Fei Xia, Pete Florence, Brian Ichter, Danny Driess, Montserrat Gonzalez Arenas, Kanishka Rao, Dorsa Sadigh, Andy Zeng

arXiv:2307.04721v2cs.AIcs.CLcs.RO

TL;DR

The paper asks whether pre-trained LLMs already contain general sequence-pattern abilities that can extend beyond language into robotics. It evaluates in-context transformation, completion, and improvement across symbolic benchmarks and robotic tasks, finding nontrivial zero-shot pattern manipulation and policy discovery, while identifying deployment limits from inference cost and context length.

  • Problem

    Whether pre-trained LLMs can perform general pattern manipulation beyond language and whether those abilities can support robotic control remain limited questions.

  • Method

    The paper evaluates pre-trained LLMs with in-context prompts for sequence transformation, function extrapolation, and reward-conditioned sequence improvement across symbolic and robotic tasks.

  • Results

    LLMs show nontrivial zero-shot pattern completion, including up to 85 ARC problems, partial token-mapping invariance, and discovery of a simple CartPole controller.

  • Takeaways & Limitations

    LLMs can serve as basic general pattern machines whose in-context sequence abilities may transfer from abstract patterns to robotic motion and closed-loop policy improvement.

  • Takeaways & Limitations

    High inference and monetary costs, longer representations for higher-dimensional or higher-precision state-action spaces, and finite context lengths constrain practical deployment on complex tasks.

Abstract

from arXiv · show

We observe that pre-trained large language models (LLMs) are capable of autoregressively completing complex token sequences -- from arbitrary ones procedurally generated by probabilistic context-free grammars (PCFG), to more rich spatial patterns found in the Abstraction and Reasoning Corpus (ARC), a general AI benchmark, prompted in the style of ASCII art. Surprisingly, pattern completion proficiency can be partially retained even when the sequences are expressed using tokens randomly sampled from the vocabulary. These results suggest that without any additional training, LLMs can serve as general sequence modelers, driven by in-context learning. In this work, we investigate how these zero-shot capabilities may be applied to problems in robotics -- from extrapolating sequences of numbers that represent states over time to complete simple motions, to least-to-most prompting of reward-conditioned trajectories that can discover and represent closed-loop policies (e.g., a stabilizing controller for CartPole). While difficult to deploy today for real systems due to latency, context size limitations, and compute costs, the approach of using LLMs to drive low-level control may provide an exciting glimpse into how the patterns among words could be transferred to actions.

1 Introduction

The paper investigates whether pre-trained LLMs can manipulate abstract, nonlinguistic patterns without additional training and apply these capabilities to robotics. It evaluates zero-shot pattern completion across ARC and sequence-based robotic tasks, including trajectory extrapolation and policy improvement.

  • Motivation: LLMs can represent and extrapolate abstract, nonlinguistic patterns, suggesting basic general pattern-machine capabilities.The paper frames this as an assessment of abilities already present in pre-trained models rather than capabilities obtained through fine-tuning.
  • ARC motivation: ASCII-art prompting enabled LLMs to solve up to 85 of 800 ARC problems without additional training or fine-tuning.This exceeded some recent systems based on manually engineered methods.
  • Token invariance: Random token mappings preserved some valid ARC solutions, indicating pattern completion can be partly independent of the specific tokens used.The paper connects this observation to broader symbolic pattern manipulation relevant to robotics and sequential decision-making.
  • Research scope: The study assesses zero-shot pattern manipulation as complementary to robot-data pre-training and robotics foundation models.The authors state that these capabilities are not sufficient to replace specialized algorithms but may inform priorities for training generalist robotic models.
  • Contributions: The evaluation covers sequence transformation, sequence completion, and sequence improvement, including robotic motion extrapolation and reward-conditioned policy learning.Examples include extending wiping motions, drawing patterns, navigating a grid, discovering a CartPole controller, and optimizing trajectories with human feedback.

2 Related Work

Related work establishes in-context learning as inference-time task adaptation and situates the paper against prior uses of LLMs for high-level robotics. Direct zero-shot control-level influence remains an open problem.

  • In-Context Learning: In-context learning uses few-shot input-output examples as task specifications for predicting subsequent instances.Scaling model size has been associated with improved, particularly out-of-distribution, in-context generalization.
  • Explanations of In-Context Learning: In-context learning can be explicitly encouraged through meta-training or emerge implicitly from token distributions and Transformer architectures.The cited literature distinguishes these explanations from recurrent-architecture settings where the same behavior need not emerge.
  • In-Context vs. In-Weights Learning: In-context learning occurs during inference without gradient updates, unlike in-weights learning based on information stored during model training.The distinction is relevant to separating prompt-driven adaptation from memorized completion behavior.
  • LLMs and Robotics: Prior robotics applications use LLMs for plans, code, and planning representations, while direct zero-shot trajectory-level control remains open.This paper explores whether sequence-pattern abilities can extend LLM influence toward control.

3 Language Models as General Pattern Machines

The paper models LLMs as autoregressive sequence completers whose prompts can contain multiple examples and task conditions. It organizes abstract pattern learning into transformation, completion, and improvement settings.

  • Autoregressive sequence modeling: Decoder-only Transformers autoregressively model numeric or arbitrary-token sequences by predicting each symbol from preceding symbols.The model factors sequence probability into conditional next-symbol probabilities.
  • Autoregressive sequence modeling: A prompt supplies initial sequence tokens and the model completes the remaining suffix.This provides the basic mechanism for applying in-context learning to abstract sequences.
  • Prompt structure: Prompt prefixes can pack multiple variable-length input-output examples and additional task conditioning into one context.This flexibility lets the model infer a task from examples rather than from natural-language instructions alone.
  • Pattern categories: Sequence transformation maps example input-output sequences to a new query input.The examples specify a sequence-to-sequence pattern to be applied to the query.
  • Pattern categories: Sequence completion extrapolates discrete samples from a single function, such as s_i=a·sin(bi).Unlike transformation prompts, these prompts contain samples from one function rather than multiple input-output pairs.
  • Pattern categories: Sequence improvement uses reward-labeled trajectories as context to infer better sequences, potentially through iterative offline or online procedures.The paper gives least-to-most prompting and in-context reinforcement learning as examples.

4 Sequence Transformation

The sequence-transformation experiments test whether LLMs can infer increasingly complex symbolic and spatial transformations, including under arbitrary token mappings. ARC and PCFG results show nontrivial zero-shot performance, with PCFG accuracy scaling with model size.

  • ARC benchmark: ARC evaluates transformations such as infilling, counting, and shape rotation from input-output grid examples.Grids are flattened into row-major token sequences, allowing variable-length outputs.
  • ARC benchmark: LLMs correctly solved up to 85 ARC problems and outperformed some program-synthesis systems using manually engineered DSLs.They did not surpass brute-force search over a handcrafted grid-operator API.
  • Token mapping invariance: Random vocabulary mappings yielded 52 solved ARC problems for text-davinci-003 and an average of 43.6 across five random alphabets.Token-mapping invariance also held to an extent for randomly sampled embeddings not associated with vocabulary tokens.
  • Token mapping invariance: Random-alphabet evaluation may provide a closer estimate of sequence-transformation ability because ARC examples could appear in training data.The paper presents a procedurally generated benchmark as additional evidence against explaining the results solely through memorization.
  • PCFG benchmark: The PCFG benchmark generates adjustable-difficulty transformations by composing lexical rules such as reverse, shift, swap, and repeat.Difficulty varies with sequence length and the number of composed rules, making the benchmark suitable for measuring incremental progress.
  • PCFG benchmark: PCFG completion accuracy improves with model scale and correlates with ARC performance.The benchmark is used for evaluation rather than training to compare general pattern capabilities across models and pre-training regimes.

5 Sequence Completion

LLMs can extrapolate numerical function patterns from context and apply this zero-shot sequence-completion ability to simple robotic motions. Larger models and more context improve prediction quality.

  • Function completion: LLMs extrapolate sinusoidal functions, including variants whose oscillation amplitude increases with x.The tested families include f(x)=a·sin(bx), ax·sin(bx), and a 2x·sin(bx).
  • Function completion: Greater context and larger-scale LLMs yield higher-quality predictions across the tested function families.Figure 3 reports lower error rates for larger models and improved accuracy with more context.
  • Robotic motion completion: The sequence-completion capability is domainagnostic because prompts provide no specialized explanation of the function or linguistic grounding for metric tokens.This enables encoding demonstrated positions as sequences and predicting future positions for simple open-loop motion extrapolation.
  • Robotic motion completion: In Table Sweeping, the model continues kinesthetic demonstrations represented as discretized seven-dimensional end-effector poses.Demonstrations are recorded at approximately 3 Hz and each value is binned to an integer between 0 and 100.
  • Robotic motion completion: In Whiteboard Drawing, the model extrapolates scripted loop patterns from partial demonstrations encoded as discretized end-effector positions.The task uses parametric loop equations and provides part of the pattern in context.

6 Sequence Improvement

Sequence improvement uses prior trajectories, optionally labeled by reward, to generate higher-performing sequences. Experiments range from structured trajectory extrapolation to online discovery of a CartPole controller and human-guided optimization.

  • Method: LLMs generate new sequences conditioned on previous sequences, enabling iterative improvement of trajectories or the policies they represent.The formulation supports offline trajectory optimization and online in-context reinforcement learning.
  • Method: Reward conditioning prefixes each trajectory with its total reward, prompting the model to generate sequences with higher rewards than those seen in context.New rollouts can supply actual reward labels for subsequent iterations.
  • Trajectory extrapolation: Text-davinci-003 generates a Marker in Cup trajectory achieving reward >90 from trajectories stopping at 20%, 40%, 60%, and 80% of the way to the cup.The authors report that sorted trajectory order and reward annotations help expose crucial meta-patterns in context.
  • Online trajectory improvement: d3 can consistently find a solution to Grid within 50 episodes, whereas other agents receive rewards ranging from 6-78.The comparison concerns online sampling of higher-reward trajectories in a 9 × 9 Grid environment.
  • Online trajectory improvement: Across various LLMs, CartPole return improves over 100 episodes, and d3 discovers oscillatory behaviors that keep the pole upright.The first 100 episodes are generated by random exploration.
  • Human-guided optimization: LLMs can adjust robot trajectories online from sparse binary rewards supplied by a human.The setup applies feedback every 2 seconds to end-effector movements in particular directions.

7 Discussion

The paper presents LLM pattern reasoning as a possible basis for robotic control, including grasp detection, forward dynamics, sparse-reward adjustment, and ARC-style generalization. It also emphasizes that current deployment is constrained by cost, context, tokenization, unpredictability, and limited physical grounding.

  • Discussion: LLMs can react in context to sparse reward signals online to encourage an end effector toward a desired goal.This extends the paper’s sequence-improvement experiments to online robotic adjustment.
  • Discussion: The proposed robotics opportunities span complex-pattern priors for control and online optimization of closed-loop policies through sequence improvement.The paper also identifies mixed-autonomy applications such as real-time pattern extrapolation for assistive teleoperation.
  • Limitations: High inference and monetary costs, context-length limits, and tokenization requirements constrain trajectory control, while language-only control remains unpredictable and weakly grounded physically.The authors state that these limitations make the approach unsuitable outside constrained lab settings at present.
  • Discussion: ARC evaluation includes correctly and incorrectly solved problems, covering operations such as infilling, counting, translating, and rotating shapes.The examples are drawn from 800 ARC problems and use a few training examples followed by a test example.
  • Discussion: The paper illustrates an in-context grasp detector and a forward dynamics model using multiple examples.The grasp detector outputs target coordinates in a downsampled image, while the dynamics model predicts spatial rearrangement.

A.3 Token Invariance for New Token Embeddings

The experiment tests whether token-invariance extends to newly sampled embedding vectors unseen during training. Performance remains similar under 1σ sampling but degrades at 2σ, indicating pattern recognition with some out-of-distribution continuous inputs.

  • A.3 Token Invariance for New Token Embeddings: The experiment replaces most native token embeddings with newly sampled vectors that the model did not encounter during training.The new embedding matrix is formed from Gaussian samples based on the original embedding statistics, while separator embeddings remain native.
  • A.3 Token Invariance for New Token Embeddings: At 1σ sampling, the model solves the single-token pattern task with performance similar to native embeddings.The task predicts the final token in patterns such as (1, 1, 2) ↦→ 2 or (1, 2, 2) ↦→ 1.
  • A.3 Token Invariance for New Token Embeddings: At 2σ sampling, performance degrades relative to native embeddings and 1σ sampling.Results use K = 100, three random seeds, 30 instances, and context lengths of 5, 10, or 20 examples.
  • A.3 Token Invariance for New Token Embeddings: The results show pattern recognition even when prompts use out-of-distribution continuous input embeddings.Numbers used to index the sampled embeddings do not enter the LLM itself.

A.4 PCFG Benchmark: Additional Details and Ablations

The PCFG benchmark procedurally generates adjustable-difficulty sequence transformations from primitive operations. Its difficulty varies with sequence length and rule complexity, and the benchmark includes examples and solve-rate tables across these factors.

  • A.4 PCFG Benchmark: Additional Details and Ablations: The benchmark procedurally generates sequence transformations with adjustable sequence length and rule complexity.It is based on the probabilistic context-free grammar from prior work and measures abstract sequence transformation capabilities.
  • A.4 PCFG Benchmark: Additional Details and Ablations: The benchmark’s primitive operations apply to one or two token sequences and can be composed into transformations.Tables illustrate unary and binary operators, including examples with w=2 over k=3 tokens and w=8 over k=10 tokens.
  • A.4 PCFG Benchmark: Additional Details and Ablations: Solve rates are evaluated across the number of tokens k and number of rules w for different models.The benchmark includes an example context containing two input-output examples followed by a query input.

A.5 PCFG Benchmark: Program Synthesis

The paper contextualizes PCFG program synthesis with DreamCoder and compares structure-learning approaches for sinusoid extrapolation. DreamCoder can solve many tasks when supplied sufficient primitives, while structure-learning completions vary by function and context.

  • A.5 PCFG Benchmark: Program Synthesis: DreamCoder is evaluated with either PCFG-specific primitives or general list primitives, both sufficient to define the benchmark transformations.The PCFG set includes operations such as copy, reverse, shift, swap, repeat, and slicing; the list set uses general list-processing operators.
  • A.5 PCFG Benchmark: Program Synthesis: With oracle access to PCFG operations, DreamCoder performs well and sometimes matches or exceeds LLM performance, especially at lower rule complexity.List primitives also solve many low-complexity tasks but show a sizeable dropoff as complexity increases.
  • A.5 PCFG Benchmark: Program Synthesis: The structure-learning baseline fits sinusoid training data with low error, but extrapolation quality differs across functions.Sine waves are generally extrapolated well, whereas sinusoid variants produce high-variance samples.
  • A.5 PCFG Benchmark: Program Synthesis: More context generally improves prediction accuracy for the structure-learning baseline and LLMs.The structure-learning method uses a Gaussian Process with a covariance kernel inferred by MCMC over a PCFG of covariance functions.
  • A.5 PCFG Benchmark: Program Synthesis: Larger language models perform better on Table Sweeping trajectory completion, with text-davinci-003 performing best and showing the lowest variance.The comparison uses average DTW distance over 30 trials with 66% of each trajectory provided as context.

B.3 Whiteboard Drawing: Qualitative Results

Qualitative completions generally preserve the overall shape of demonstrated whiteboard loops, but fine motion patterns remain difficult to predict precisely. The broader robotics setting represents trajectories as sequences of states and actions, with context construction constrained by token limits and action encoding choices.

  • B.3 Whiteboard Drawing: Qualitative Results: Text-davinci-003 completions generally match the overall shape of the two demonstrated loops across narrow, medium, and wide styles.Figure 14 shows three trials for each loop style.
  • B.3 Whiteboard Drawing: Qualitative Results: Fine motion patterns remain challenging to predict precisely in the loop completions.The qualitative trials illustrate a gap between matching global shape and reproducing detailed motion.
  • B.3 Whiteboard Drawing: Qualitative Results: Trajectory-improvement prompts encode Cartesian states, executable position-control trajectories, and returns based on distance to a goal.The state dimensions are normalized between 0 and 200, and trajectories are conditioned on histories with increasing returns.
  • B.3 Whiteboard Drawing: Qualitative Results: Context length limits how many trajectories can be included because representation cost scales with observation and action dimensions, horizon, and trajectory count.CartPole experiments use a 1024-token context limit for the listed models.
  • B.3 Whiteboard Drawing: Qualitative Results: Action-token choices can affect online trajectory improvement, motivating 1-indexed integer actions instead of including zero.Initial experiments observed a tendency to sample “0” when zero was included in the action space.

C.4 Clicker Training: Additional Details

The clicker-training setup represents visual observations and end-effector movements as normalized numeric tuples. Context examples use reward-conditioned observation-action transitions, balanced across clicking and non-clicking outcomes.

  • Visual observations contain end-effector and approximate object positions, with x, y, and z values normalized between 0 and 300.
  • Actions encode end-effector movements normalized between 0 and 100, where 50,50,50 denotes no movement.
  • Listing 5 orders reward, observation, and action tuples by reward, balancing reward-0 and reward-1 transitions.A click produces reward 1, and the context includes an equal number of reward 0 and reward 1 transitions.
Loading 2307.04721v2…