PaperlayerView exact revision on arXiv

Source-linked AI summary

Foundation Models for Decision Making: Problems, Methods, and Opportunities

Sherry Yang, Ofir Nachum, Yilun Du, Jason Wei, Pieter Abbeel, Dale Schuurmans

arXiv:2303.04129v1cs.AIcs.LG

TL;DR

Foundation models for decision making address how pretrained models can interact with external agents and environments while performing long-term reasoning. The report scopes this intersection, reviews adaptation methods spanning generative modeling, planning, control, and reinforcement learning, and identifies persistent challenges involving data, task structures, and model capabilities.

  • Problem

    Foundation models face challenges when learning from external feedback, adapting across modalities, and performing long-term reasoning in interactive decision-making settings.

  • Method

    The report provides conceptual and technical background, surveys methods for adapting foundation models to decision making, and reviews open problems and promising approaches.

  • Results

    The reviewed approaches include generalist agents, conditional generative models, interactive dialogue optimization, text-based environments, and long-horizon generative planning.

  • Takeaways & Limitations

    Jointly considering foundation models and interactive decision making may combine broad world knowledge with feedback and long-term planning.

  • Takeaways & Limitations

    The field faces significant challenges from gaps in data modalities, ambiguities in environment and task structures, and missing components in current paradigms.

Abstract

from arXiv · show

Foundation models pretrained on diverse data at scale have demonstrated extraordinary capabilities in a wide range of vision and language tasks. When such models are deployed in real world environments, they inevitably interface with other entities and agents. For example, language models are often used to interact with human beings through dialogue, and visual perception models are used to autonomously navigate neighborhood streets. In response to these developments, new paradigms are emerging for training foundation models to interact with other agents and perform long-term reasoning. These paradigms leverage the existence of ever-larger datasets curated for multimodal, multitask, and generalist interaction. Research at the intersection of foundation models and decision making holds tremendous promise for creating powerful new systems that can interact effectively across a diverse range of applications such as dialogue, autonomous driving, healthcare, education, and robotics. In this manuscript, we examine the scope of foundation models for decision making, and provide conceptual tools and technical background for understanding the problem space and exploring new research directions. We review recent approaches that ground foundation models in practical decision making applications through a variety of methods such as prompting, conditional generative modeling, planning, optimal control, and reinforcement learning, and discuss common challenges and open problems in the field.

1 INTRODUCTION

Foundation models and sequential decision making are increasingly converging as models face interactive, long-horizon tasks. This report scopes the intersection, reviews approaches, and identifies challenges and opportunities.

  • Motivation: Foundation models are being applied to tasks involving long-term reasoning, control, search, planning, and interaction with external entities.Examples include dialogue with humans and robotic perception-control systems acting in physical environments.
  • Background: Traditional sequential decision-making methods span reinforcement learning, imitation learning, planning, search, and optimal control, but often learn from task-specific data with limited prior knowledge.
  • Background: Sequential decision-making methods can achieve strong task performance but generally struggle with generalization and sample efficiency when learning from scratch.One cited example required 7 GPU days of interactive gameplay to solve a single Atari game.
  • Scope and contribution: The report argues that combining the fields can provide foundation models with interactive feedback and planning while giving decision-making systems broader knowledge for faster learning and better generalization.
  • Scope and contribution: The manuscript organizes its review around how foundation models can characterize components of decision-making systems, including behavior and environment generation.It also presents background, technical tools, challenges, open problems, and promising approaches.

2 PRELIMINARIES

The preliminaries formalize sequential decision making through interactive environments, trajectories, rewards, imitation learning, reinforcement learning, and model-based control. They also contrast these foundations with foundation-model adaptation and motivate unified interfaces for diverse tasks.

  • MDP formalism: Sequential decision making is formalized with Markov decision processes containing states, actions, rewards, transitions, initial-state distributions, and discount factors.A policy samples actions, and the environment returns successor states and scalar rewards.
  • MDP formalism: An episode is a trajectory of state-action-reward tuples, and its return is the total discounted sum of rewards.
  • Datasets: Interactive trajectories containing actions and rewards differ from static vision or language datasets, although foundation-model techniques can also be applied to trajectory datasets.
  • Imitation learning: Imitation learning trains policies from fixed demonstrations, with behavioral cloning treating state-to-action prediction as supervised classification or regression.
  • Reinforcement learning: Reinforcement learning optimizes expected returns through trial-and-error interaction, while policy-gradient and value-based methods trade off sample efficiency, variance, stability, and bias.
  • Planning and control: Model-based decision making uses known or learned environment dynamics to support planning, search, and optimal control.The figure frames these interactions as repeated loops between generated actions and external feedback.

3 FOUNDATION MODELS AS CONDITIONAL GENERATIVE MODELS

Foundation models can model interactive trajectories from decision-making datasets, serving either as conditional generators of behavior or as models of environment dynamics. The section uses this framing to connect generative modeling with decision making.

  • The first concrete use case is probabilistic modeling of trajectory distributions from interactive datasets.Depending on the modeled trajectory component, the model represents actions or environment dynamics.
  • The section reviews generative models and examines how they can represent behaviors and environment models for decision making.

3.1 Generative Model Preliminaries

Generative models represent data distributions through different factorizations, including latent variables, autoregressive conditionals, diffusion chains, and energy functions. These choices determine how distributions are learned, approximated, and sampled.

  • Latent variable models: Latent variable models factorize a data distribution into a latent-variable distribution and a conditional distribution.When marginalization is intractable, VAEs optimize an evidence lower bound.
  • Latent variable models: VQ-VAE discretizes continuous latent representations with a codebook to learn more compact discrete representations.
  • Autoregressive models: Autoregressive sequence models factorize a joint sequence distribution into conditional factors, each modeled from preceding elements.Transformer architectures can parameterize these conditional factors.
  • Diffusion models: Diffusion models represent data generation as reverse sampling through a Markov chain of Gaussian transitions from noise.The forward process adds noise, while the reverse process approximates and removes it.
  • Energy-based models: Energy-based models represent distributions with unnormalized learned energy functions and typically sample using procedures such as Langevin dynamics.

3.2 Generative Models of Behavior

Generative models can represent behavioral priors, policies, exploration, and self-improvement from interactive trajectories, enabling behavior composition and scaling across tasks.

  • Behavioral Priors: Behavioral priors model reusable skills from diverse trajectories, allowing behaviors such as picking up, moving, and placing objects to compose unseen tasks.Pretraining generally uses maximum likelihood estimation of actions conditioned on trajectory-level information.
  • Behavioral Priors: VAE-based behavioral priors encode trajectories or future goals and decode action sequences, while history-dependent autoregressive policies model actions from interaction histories.The posterior can represent diverse priors across related tasks, whereas the test-time prior is constrained to past information.
  • Conditional Behavior Generation: Goal- or return-conditioning and diffusion guidance steer generated behaviors toward desired outcomes, including high-return trajectories.Reward can serve as classifier-free guidance when sampling from diffusion models.
  • Generalist Agents: Generalist sequence models scale behavior modeling across simulated tasks, more than 40 Atari games, over 700 robot tasks, and over 600 multimodal tasks.These systems reportedly perform significantly better than single-task specialists, with benefits from scaling parameters and training tasks.
  • Scaling Data and Training: Behavior modeling can scale through Internet-scale text and video data, but these sources typically provide fewer action annotations than interactive decision-making datasets.Large-scale online training in games and randomized environments has not yet leveraged foundation models, leaving integration with massive online settings as an open direction.
  • Exploration and Self-Improvement: Behavior models can also learn exploration and self-improvement from datasets containing exploratory or improving behavior, while corrective re-prompting incorporates environmental error feedback into plans.These approaches treat long-horizon planning or learning processes as in-context learning problems.

3.3 Generative Models of the World

Generative world models learn transition dynamics, rewards, and policies from offline trajectories, supporting model-based rollouts and long-horizon planning through autoregressive or diffusion generation.

  • World Models: World models learn transition dynamics T and reward functions R from offline trajectories, and conditional generation can produce model-based rollouts for policy improvement.Because trajectories also contain behavior-policy actions, π, T, and R can be modeled jointly.
  • One-Step Prediction: Autoregressive factorization of trajectory likelihood separates environment dynamics and policy components, so maximum likelihood learning decomposes into T, R, and π.This factorization provides a unified generative procedure for the offline dataset.
  • One-Step Prediction: States, actions, and rewards occupy different modalities, motivating discretization for GPT-style modeling or latent representations and dynamics models for image-based domains.Trajectory Transformer discretizes continuous-control dimensions, while image-based settings more commonly learn latent spaces and dynamics.
  • Latent World Models: Latent sequential VAEs encode histories into latent states, decode observations, and condition policies and rewards on those latent states.The latent dynamics model can be parameterized with an RNN or transformer-based state-space model.
  • Long-Term Futures: Planning with trajectory-level generative models supports conditioning on returns, goals, skills, dynamics constraints, and text, including high-dimensional video futures.Sampling complete trajectories is presented as avoiding compounding error from rolling out single-step dynamics and may support multiple properties such as return and safety.

4 FOUNDATION MODELS AS REPRESENTATION LEARNERS

Foundation models support decision making either by transferring pretrained representations from broad data or by learning task-specific representations from interactive trajectories.

  • Representation Learning Objectives: Sequential decision-making representation objectives include model-based representations, temporal contrastive learning, masked autoencoding, and offline RL pretraining.These objectives are illustrated on trajectories containing states, actions, and rewards.
  • Plug-and-Play Foundation Models: Off-the-shelf image, vision-language, and language models can initialize perception, enrich observations, provide task feedback, and initialize policies across decision-making settings.Plug-and-play transfer is most natural for tasks involving real-world images or text, and less applicable to idiosyncratic state-action spaces.
  • Plug-and-Play Foundation Models: Text commands and goal images can serve as task specifiers, augmenting robot states to support more robust, general, and multitask policies.Examples include commands to close a cabinet door or goal images showing the desired closed-door state.
  • Task-Specific Representations: Model-based objectives cluster states with similar rewards and transition probabilities, while forward or inverse dynamics provide alternative representation-learning signals.Representations can also derive from value functions, curiosity metrics, or bisimulation properties.
  • Masked Autoencoders: Masked autoencoders randomly mask trajectory tokens, encode the masked sequence with a transformer, and reconstruct the missing inputs while incorporating dynamics information.They can learn from broader datasets with missing actions and rewards.
  • Post-Representation Learning: Representation learners extract states, actions, and dynamics but require additional finetuning or model-based policy optimization for strong decision-making performance.It remains unclear how sequential objectives effectively incorporate broad multitask data when target-task dynamics differ.

5 LARGE LANGUAGE MODELS AS AGENTS AND ENVIRONMENTS

Large language models can function as agents or environments in sequential decision-making settings, using feedback from humans, tools, or the real world. The report examines dialogue, tool use, and iterative prompting through this lens, while identifying limitations that motivate interactive approaches.

  • Foundation models can serve as agents or environments, enabling learning from feedback produced by humans, tools, or the real world.
  • Interacting with Humans: Dialogue agents can be optimized by fine-tuning pretrained language models on dialogue data and then applying reinforcement learning with human feedback.RLHF trains a preference model from human rankings of model outputs and uses it to optimize the language-model policy.
  • Interacting with Humans: Human feedback alone leaves dialogue agents vulnerable to outdated information, hallucinated facts, and difficulties with complex reasoning and mathematical calculations.The report suggests that tools and external entities may provide feedback more scalably and reliably for these properties.
  • Interacting with Tools: Tool-using language-model agents must learn when to call tools, which tools to use, and how to use them, creating additional supervision and prompting challenges.Supervised approaches require API-call annotations, while prompting often relies on tool-specific hand-crafted prompts and rule-based parsing.
  • Interacting with Tools: Interactive environments such as web browsers offer a way to scale beyond supervised fine-tuning and prompting by sampling experience from executed queries and optimizing them with reinforcement learning.
  • Language Models as Environments: Iterative prompting can be formulated as an MDP in which a prompt provider acts on prompting history, receives language-model responses, and obtains rewards.High-level actions include decomposition, ranking, denoising, paraphrasing, summarization, pruning, and search, and can be recursively composed.
  • Language Models as Environments: Prompting rewards can mark whether a language model output reaches a goal answer, connecting iterative prompting to sequential decision making.

6 OPEN PROBLEMS, CHALLENGES, AND OPPORTUNITIES

The report identifies dataset, representation, context, grounding, behavior-extraction, and online-learning challenges for applying foundation models to decision making, while outlining methods to address them.

  • Dataset challenges: Decision-making datasets often differ from broad vision-language datasets in modality and structure, lacking explicit action and reward labels.Videos may show observations without labeling cause-effect relationships or indicating which outcomes are preferable.
  • Dataset challenges: Bridging these datasets can involve logging human interactions, hindsight relabeling, or adding descriptions, instructions, and task information.The report proposes scaling interactive data, annotating existing videos, and enriching decision-making datasets with instructional content.
  • Environment and task structure: Universal tokenization can encode states, actions, and rewards across environments, but may fail to preserve pretrained models’ rich knowledge and generalization abilities.Converting environments into text offers another interface, although it is unnatural for some visual applications and remains behind task-specific systems in simple games.
  • Foundation-model improvements: Effective decision making requires long histories, whereas transformer context windows are bounded; external memory and retrieval are proposed directions.Prompted intermediate computations may extend context but are sensitive to prompt selection and output parsing.
  • Foundation-model improvements: Foundation models must be grounded in physical geometry and physics, but simulator-based methods assume access to simulators unavailable in the physical world.The report highlights more accurate grounding of predictions in the physical world as an open research direction.
  • Improving decision making: Desirable behavior can be elicited through instruction or goal-conditioned finetuning, prompting, and iterative reasoning or improvement without explicit reinforcement-learning objectives.Direct reinforcement-learning finetuning needs large online samples, while scalable software environments could provide broader online access.

7 DISCUSSION AND PERSPECTIVES

The report frames foundation models for decision making as a route from basic capabilities toward more sophisticated intelligence through interaction with the external world. It organizes this opportunity around models of behavior, the world, and knowledge, while emphasizing unresolved data, task-structure, and paradigm challenges.

  • Perspectives: Foundation models for decision making aim to extend basic abilities such as seeing, hearing, speaking, reading, and writing toward world-class expertise.The report relates this progression to learning through practice, trial and error, and interaction with the external world.
  • Perspectives: Foundation models can characterize decision-making components including generative models of behavior and the world, world-knowledge representations, and language-based agents or environments.These components correspond to the report’s organization of research directions.
  • Perspectives: Major challenges include gaps between data modalities, ambiguity in environment and task structures, and missing components in current foundation-model and decision-making paradigms.The report presents these issues as continuing obstacles despite initial successes.
Loading 2303.04129v1…