PaperlayerView exact revision on arXiv

Source-linked AI summary

Supervising strong learners by amplifying weak experts

Paul Christiano, Buck Shlegeris, Dario Amodei

arXiv:1810.08575v1cs.LGcs.AIstat.ML

TL;DR

Complex objectives can be too difficult for humans to evaluate directly, while optimizing simpler proxies can produce misaligned behavior. The paper proposes Iterated Amplification, which builds training signals by decomposing tasks into simpler subproblems and combining their solutions. In algorithmic experiments, it learned the target tasks effectively, with at worst a modest slowdown relative to supervised learning from ground truth.

  • Problem

    Complex tasks may lack a meaningful training signal because humans cannot directly evaluate them, while proxy optimization can produce pathological behavior.

  • Method

    Iterated Amplification trains an agent from a human-coordinated decomposition of complex tasks into simpler subproblems, without an external reward function.

  • Results

    Iterated Amplification solved the tested tasks effectively, with at worst a modest slowdown compared with supervised learning from ground-truth data.

  • Takeaways & Limitations

    The results support using decomposition to build training signals for algorithmically complex tasks whose objectives are implicit rather than externally specified.

  • Takeaways & Limitations

    The experiments use algorithmic decompositions and hand-coded algorithms, so they do not establish whether humans can decompose interesting real-world tasks or whether messy real-world decompositions are learnable.

Abstract

from arXiv · show

Many real world learning tasks involve complex or hard-to-specify objectives, and using an easier-to-specify proxy can lead to poor performance or misaligned behavior. One solution is to have humans provide a training signal by demonstrating or judging performance, but this approach fails if the task is too complicated for a human to directly evaluate. We propose Iterated Amplification, an alternative training strategy which progressively builds up a training signal for difficult problems by combining solutions to easier subproblems. Iterated Amplification is closely related to Expert Iteration (Anthony et al., 2017; Silver et al., 2017), except that it uses no external reward function. We present results in algorithmic environments, showing that Iterated Amplification can efficiently learn complex behaviors.

1 Introduction

The paper targets tasks whose objectives are too complex for direct human evaluation, where proxy optimization can produce misaligned behavior. It proposes Iterated Amplification, which builds training signals by decomposing complex tasks into simpler subproblems and combining their solutions.

  • Motivation: Many useful tasks lack an algorithmic training signal, so humans commonly provide demonstrations or judgments instead.Algorithmic signals are available when goals can be evaluated automatically or correct examples can be generated.
  • Motivation: Beyond-human-scale tasks can exceed a single person’s ability to perform or evaluate because their observation spaces are too large.Examples include economic policy, scientific research, and large-scale computer-network security.
  • Motivation: Optimizing an easy-to-measure proxy for a complex objective can produce pathological behavior and mismatch the intended goal.The paper gives user satisfaction as a proxy for societal benefit that could incentivize fraudulent appearances or manipulation.
  • Contribution: Iterated Amplification builds a training signal for complex tasks by using AI-assisted decomposition into simpler tasks with human or algorithmic signals.The paper applies the framework to relatively simple tasks as an initial experimental step.
  • Method: Amplify_H(X) lets a human expert coordinate several copies of agent X to solve problems, after which X learns from the composite system.The paper focuses on question-answering, delegation, and supervised learning for its instantiation.
  • Method: The method has no external objective: the target is implicit in how the human coordinates and combines subanswers.This lets X learn the goal while learning competent behavior, rather than optimizing a separately specified reward.

2 Detailed instantiation of Iterated Amplification

The implementation trains a human predictor to reproduce decomposition decisions, uses it to generate answers, and trains the agent by supervised learning on those answers. Training progressively moves from random responses toward increasingly capable answers, while context reuse accelerates the experiments.

  • Predicting the human behavior: A human predictor H′ imitates how H identifies subquestions and combines subanswers, reducing the amount of data needed to model the expert’s role.H′ generates training data for X through Amplify_H′(X) instead of consulting H directly.
  • Predicting the human behavior: H′ must be continuously updated because it predicts human responses to X’s changing, non-stationary subanswers.The distribution of responses changes as agent X changes during training.
  • Training overview: Training samples questions from D, records human decomposition transcripts, trains H′ on those decisions, and trains X on answers produced by Amplify_H′(X).The transcript contains the original question, subquestions, subanswers, and final answer.
  • Dynamics of training: X progresses from random answers to simple copied answers and then to increasingly capable answers produced through finer-grained decomposition.At each stage, Amplify_H(X) is modestly smarter than X alone, creating a moving target for learning.
  • Dynamics of training: The final agent can approximate the behavior of an exponentially large team of H copies, while discarding the hierarchical decomposition used during training.The learned procedure need not mirror the training decomposition structure.
  • Questions with context: Iterated Amplification handles large contexts by decomposing questions into subquestions that depend on progressively smaller context portions.This allows H to inspect only relevant context pieces rather than the entire document.
  • Questions with context: Reusing context-encoding work across multiple questions speeds training by an order of magnitude.The architecture separates context encoding from question answering and reuses the encoded context across subquestions.
  • Model architecture: X uses an encoder-decoder Transformer architecture, while H′ uses a Transformer decoder that can copy symbols and never observes the entire context.X represents contexts as tokenized facts and can generate or copy output symbols.

3 Related Work

Iterated Amplification is related to Expert Iteration and recursive approaches, but differs by using no external objective and by using decomposition only to generate training data. It also avoids the model-specification problem faced by inverse reinforcement learning for tasks too challenging for humans to evaluate directly.

  • Expert Iteration: Iterated Amplification resembles Expert Iteration and AlphaZero, which use learned models in dynamic-programming-like updates.These methods have achieved strong performance in board games, while Iterated Amplification targets settings without an external objective.
  • Expert Iteration: Unlike Expert Iteration, Iterated Amplification has no external objective; the task definition is implicit in decomposition and recomposition.Expert Iteration produces its expert through search that optimizes an external objective.
  • Inverse reinforcement learning: Inverse reinforcement learning would require a sufficiently accurate, non-identifiable model of human preferences for tasks beyond direct human evaluation.Iterated Amplification instead avoids this model-specification problem.
  • Debate: Debate and Iterated Amplification both use recursive subquestions, but Iterated Amplification answers each subquestion with an independent copy of X.In debate, one of the debaters answers the subquestions while defending a particular top-level answer.
  • Recursive model architectures: Unlike recursive model architectures, Iterated Amplification uses recursive decomposition only to generate training data, while the learned model itself is non-recursive.Only a single decomposition step is performed in each iteration.
  • Methodological distinction: The decomposition need not determine the learned agent’s eventual strategy, which may instead use holistic heuristics.This flexibility matters because independently solving decomposed pieces can be less efficient than solving the problem holistically.

4 Experiments

The experiments evaluate Iterated Amplification on five toy algorithmic tasks using hand-coded decompositions and curricula of smaller instances. Compared with supervised learning from ground-truth data, it solves the tasks effectively with at worst a modest slowdown, but the experiments leave important real-world assumptions untested.

  • Experimental setup: Iterated Amplification is evaluated on 5 toy algorithmic tasks involving permutations, functional assignments, wildcard sums, graph distances, and rooted forests.Each task asks questions about a large combinatorial context.
  • Experimental setup: The experiments use curricula of smaller instances and hard-coded algorithms H to decompose each task.The decompositions are not used directly as recursive algorithms because doing so is inefficient for these tasks.
  • Results: Iterated Amplification solves the tasks effectively with at worst a modest slowdown relative to supervised learning from ground-truth data.This comparison is presented as achieving the paper’s main experimental goal.
  • Results: Amplification requires about twice as much computation per question because training targets are generated by running AmplifyH′(X).The additional target-generation work was performed on a parallel worker to maintain the same wall-clock training time.
  • Results: Supervised learning required tens of millions of examples, whereas Iterated Amplification required tens of thousands to learn the simpler decompositions.The authors caution that sample complexity may be much larger for realistic tasks.
  • Simplifications and limitations: The experiments do not test whether humans can decompose interesting real-world tasks or whether messy real-world decompositions can be learned.They replace human decomposition with a hand-coded algorithm and use algorithmically decomposable questions.
  • Simplifications and limitations: The experiments train X by supervised learning from AmplifyH(X), whereas important applications may instead learn a reward function from AmplifyH(X) and use reinforcement learning.This difference limits how directly the experimental setup represents proposed real-world applications.
  • Assumptions: Success requires a question distribution broad enough to cover target questions and all subquestions generated during AmplifyH(X).The distribution also determines how model capacity is allocated and therefore must be chosen carefully.

5 Discussion of decomposition in realistic domains

Iterated Amplification assumes that a human can coordinate multiple copies of an agent to outperform one copy, using decomposed subquestions to handle complex tasks. The collaboration need only help at all, and it avoids several difficulties of human teamwork.

  • The key assumption is that a human can coordinate multiple copies of X to outperform a single copy.
  • For transit-system design, copies of X can evaluate separate considerations before a human aggregates them.
  • AmplifyH(X) need not be efficient: even a 10% improvement from ten collaborating agents provides a useful training signal.
  • The copies can run sequentially and require no incentive management because each is trained only to solve its assigned problem.
  • Human organizations’ ability to outperform individuals in many domains supports the assumption underlying coordination of multiple copies.

6 Conclusion

The paper shows that Iterated Amplification can solve algorithmically complex tasks without an external reward function when the objective is implicit in a learned decomposition. It presents this as a possible route toward tasks where suitable objectives cannot be computed even with human help.

  • Iterated Amplification successfully solves algorithmically complex tasks without an external reward function, using an objective implicit in a learned decomposition.

A Training dynamics

The training dynamics show X learning to imitate the stronger, recursively constructed target AmplifyH′(X), while increasing task difficulty temporarily lowers accuracy. The decomposition example illustrates how complex evaluations can be assembled from progressively smaller subquestions.

  • Training dynamics: Figure 3 tracks X chasing the moving target AmplifyH′(X) while separating performance by recursion depth.
  • Training dynamics: X always has lower accuracy than AmplifyH′(X) because it is trained to imitate that target.
  • Training dynamics: Increasing task difficulty causes accuracy to drop, producing a curriculum-driven sawtooth pattern.
  • Example decomposition: A transit-design comparison can be decomposed into subquestions about usefulness, costs, risks, performance, maintenance, and trade-offs.
  • Example decomposition: Each subquestion can be divided further, allowing increasingly specific assessments of routes, users, capacity, bottlenecks, and performance.
  • Example decomposition: A final copy of X can answer the original question using subanswers, outperforming a copy that does not receive them and supplying a training signal.
  • Example decomposition: Without external ground truth, the decomposition itself can define the task instead of merely helping an AI search for a predictor.

C Task descriptions

The task suite represents contexts, questions, decompositions, and primitive questions that provide factual access to the human coordinator. Tasks encode domain elements and facts as symbolic sequences, with special handling for ambiguity and question types.

  • Each task specifies a context, question family, decomposition strategy, and primitive questions that immediately return correct answers to H.
  • Most tasks use domains of size N and represent elements as pairs built from eight special identifiers.
  • Sequential assignments, union finding, wildcard search, and shortest path represent integers with separate symbols, while permutation powering encodes exponents in binary.
  • Facts such as function values and graph edges are encoded as sequences of domain elements, with zero values omitted for wildcard search.
  • Union-find and shortest-path questions receive unique prefix symbols to disambiguate them.
  • A special “?” symbol marks recursive results that are inconsistent or inconclusive.

C.2 Curriculum

Training uses a curriculum that increases task difficulty from 8 to 64 as the model reaches a target accuracy, while sampling both current and easier task sizes. Performance is evaluated on a held-out test distribution emphasizing the easiest and hardest sizes.

  • C.2 Curriculum: At each training step, the task size equals the current difficulty with probability 0.5; otherwise, it is sampled from a power law between 8 and that difficulty.
  • C.2 Curriculum: The held-out test set contains one-third size-64 examples, one-third size-8 examples, and one-third uniformly distributed intermediate sizes.This distribution is intended to represent progress throughout training.

C.3 Detailed decompositions

The detailed decompositions recursively reduce graph questions to simpler subquestions, including adjacency, random-neighbor, first-vertex, and distance queries. The task suite uses difficulty parameters from 8 to 64, with some listed decompositions explicitly simplified relative to the full versions.

  • C.3 Detailed decompositions: The task suite increases difficulty N from 8 to 64, and decompositions marked with * are significant simplifications of the full decompositions.
  • C.3 Detailed decompositions: The graph context has N vertices and 2N directed edges, with questions about distance and the first vertex on a path.
  • C.3 Detailed decompositions: Primitive questions provide immediate answers to simpler graph facts, including whether an edge exists and a random neighbor of x.
  • C.3 Detailed decompositions: For shortest path, finding the distance from x to y first checks adjacency and otherwise asks for the first path vertex z and the distance from z to y.The recursive result is d + 1 after confirming that z is adjacent to x.
  • C.3 Detailed decompositions: The first vertex on a path is selected by comparing candidate vertices using adjacency tests and, when both qualify, their distances from the target.

C.4 Task distributions

Tasks use randomly generated contexts and question distributions designed so subquestions remain represented, while the model processes unordered fact sets with a Transformer-like architecture. Training reuses context representations across questions and applies Polyak averaging for stability.

  • C.4 Task distributions: Contexts and questions are randomly generated across the algorithmic tasks, with distributions chosen so every subquestion of a sampled question is represented.Sequential assignments, shortest path, and union find sample questions directly; wildcard search and permutation powering use task-specific random procedures.
  • C.4 Task distributions: AmplifyH′(X) uses Polyak averaging over 1000 batches instead of the current X, which early experiments found improved training stability without materially slowing learning.
  • C.4 Task distributions: Inputs contain 8 to 128 sentences, each with 2 to 8 tokens from vocabularies of size 10 to 30, and the model maps 100-dimensional token embeddings into a 512-dimensional representation.
  • C.4 Task distributions: The model embeds unordered sets of sentences without positional encoding and processes them with self-attention layers followed by question-answering layers that attend over the context embedding.The context encoder uses 6 layers, while the question-answering component uses 3 layers and omits decoder self-attention because questions are unrelated.
  • C.4 Task distributions: Training runs contain 100,000 to 200,000 batches, with 50 contexts per batch and 64 or 128 questions per context by the end of training.
Loading 1810.08575v1…