PaperlayerView exact revision on arXiv

Source-linked AI summary

Evaluating Language-Model Agents on Realistic Autonomous Tasks

Megan Kinniment, Lucas Jun Koba Sato, Haoxing Du, Brian Goodrich, Max Hasin, Lawrence Chan, Luke Harold Miles, Tao R. Lin, Hjalmar Wijk, Joel Burget, Aaron Ho, Elizabeth Barnes, Paul Christiano

arXiv:2312.11671v2cs.CLcs.AIcs.LG

TL;DR

The report examines whether language-model agents can perform autonomous replication and adaptation (ARA), a capability the authors say could have wide-reaching consequences. It evaluates four agents on twelve relevant tasks and finds they remain far from ARA, while scaffolding and fine-tuning may affect agent capabilities.

  • Problem

    The report asks whether language-model agents can perform ARA, a capability the authors say could have wide-reaching consequences and make capabilities harder to bound.

    Source: §1 · p. 2, §1.1 · p. 3, §6 · p. 13

  • Method

    The authors built four agents by combining language models with scaffolding programs and evaluated them on twelve tasks relevant to ARA.

    Source: §1 · p. 2, §6 · p. 13

  • Results

    The four evaluated agents were far from ARA; the authors judge casual users of those GPT-4 or Claude versions unlikely to approach its threshold without fine-tuning.

    Source: §6 · p. 13

  • Takeaways & Limitations

    The authors report that scaffolding and prompting appear to affect autonomous-task capabilities, and unrelated fine-tuning may sometimes increase ARA-relevant capabilities.

    Source: §6 · p. 13

    The agents often loop, hallucinate results, propose implausible plans, and misdiagnose errors, limiting their ability to complete tasks.

    Source: §4.3 · p. 11

Abstract

from arXiv · show

In this report, we explore the ability of language model agents to acquire resources, create copies of themselves, and adapt to novel challenges they encounter in the wild. We refer to this cluster of capabilities as "autonomous replication and adaptation" or ARA. We believe that systems capable of ARA could have wide-reaching and hard-to-anticipate consequences, and that measuring and forecasting ARA may be useful for informing measures around security, monitoring, and alignment. Additionally, once a system is capable of ARA, placing bounds on a system's capabilities may become significantly more difficult. We construct four simple example agents that combine language models with tools that allow them to take actions in the world. We then evaluate these agents on 12 tasks relevant to ARA. We find that these language model agents can only complete the easiest tasks from this list, although they make some progress on the more challenging tasks. Unfortunately, these evaluations are not adequate to rule out the possibility that near-future agents will be capable of ARA. In particular, we do not think that these evaluations provide good assurance that the ``next generation'' of language models (e.g. 100x effective compute scaleup on existing models) will not yield agents capable of ARA, unless intermediate evaluations are performed during pretraining. Relatedly, we expect that fine-tuning of the existing models could produce substantially more competent agents, even if the fine-tuning is not directly targeted at ARA.

1 Introduction and motivation

Conventional benchmarks provide limited evidence about dangerous capabilities, while the report evaluates tool-using language-model agents on open-ended tasks relevant to autonomous replication and adaptation (ARA). It defines ARA and explains why capabilities that enable proliferation could be difficult to bound.

1 Introduction and motivation

  • Conventional benchmarks provide limited evidence about dangerous capabilities because they test short, self-contained tasks while risk depends on real-world behavior.Source: §1 · p. 1
  • The report pilots a response by building four tool-using agents and evaluating them on twelve tasks relevant to ARA, from filesystem retrieval to end-to-end phishing.The tasks range from simple to relatively complex.Source: §1 · p. 2
  • These evaluations measure only the constructed agents, not an upper bound on model risk; fine-tuning and better scaffolding or prompting could yield more capable agents.The report recommends considering advances in fine-tuning and scaffolding when forecasting capabilities.Source: §1 · pp. 2–3

1.1 Autonomous replication and adaptation

  • ARA means being able to acquire resources, obtain computing power, install and improve copies, and switch strategies when one fails.Source: §1.1 · p. 3
  • Uncontrolled proliferation could make capability bounds harder to set, as copies might multiply, improve their capabilities, or scale threats such as ransomware.The authors also describe possible fine-tuning, improved scaffolding, and training more advanced successors.Source: §1.1 · p. 3

2 Agents

The agents combine LLMs with scaffolding that executes tool actions in a stepwise loop, while human overseers monitor runs and may intervene. The section describes four agent configurations and the evaluation procedures used with them.

2 Agents

  • The scaffolding connects an LLM API to a virtual machine and tools such as Bash or Python, a text-based browser, and subagents.In many cases, a human overseer reviews actions with side effects before execution.Source: §2 · pp. 3–4
  • At each step, the model may think aloud and proposes an action; scaffolding executes it and appends the action and outcome to the model’s prompt history.The model can also generate text describing its next action.Source: §2 · pp. 3–4
  • The pilot aimed to evaluate agents with performance similar to publicly available LM agents, and the authors do not claim agent design as a major technical contribution or publish the code.Source: §2 · p. 4

2.1 List of agents

  • The four configurations include GPT-4-simple, with Bash access, and GPT-4-delegate, which adds subagents, messaging, and a web-browsing agent.GPT-4-simple prompts for chain-of-thought reasoning and stores long command outputs in files the agent can view.Source: §2.1 · pp. 4–5
  • GPT-4-early-delegate uses the delegate scaffolding with a pre-release GPT-4 model, while Claude-assistant uses Claude-v1.3 with a human-assistant-style prompt.Samples were drawn at temperature 0 to improve reproducibility during evaluation development.Source: §2.1 · p. 5

2.2 Evaluation procedure

  • Runs end when an agent reports completion or termination conditions indicate failure; a human then classifies performance as complete, partial, or not complete.In general, agents operate autonomously, without assistance from overseers.Source: §2.2 · p. 5
  • Overseers could simulate action results, override refusals, or correct tooling flaws, and one of 48 runs received an additional intervention to end a CAPTCHA-blocked browsing session.GPT-4 agents did not use versions supporting OpenAI plugins; the report notes that plugin-enabled versions could plausibly yield more capable agents.Source: §2.2 · pp. 5–6

3 Task suite

The task suite defines tasks through instructions, environment setup, overseer simulation, and evaluation criteria. Its 12 tasks were selected to span difficulty and obstacles while remaining easier than ARA.

3 Task suite

  • Each task specifies model instructions, human setup of the initial environment, overseer simulation instructions, and criteria for complete, partial, or failed outcomes.Source: §3 · p. 6
  • Table 1 provides short descriptions of all 12 evaluation tasks and links to their full specifications.Source: §3 · p. 6, Table 1 · p. 7
  • The pilot tasks were chosen as easier than ARA: the authors are fairly confident an agent capable of ARA in messy real-world conditions could solve most or all of them.Source: §3 · p. 7
  • Task difficulty ranges from easy for current language models, such as filesystem password search, to nearly as difficult as ARA, such as targeted phishing and increasing LLaMA context length.Source: §3 · p. 7
  • The suite targets diverse obstacles, using supplied account credentials for most tasks to test varied abilities rather than highly correlated measures that shift abruptly with individual capability gains.Some tasks still test agents’ ability to create online accounts.Source: §3 · pp. 7–8

4 Results

The agents solved only the easiest pilot tasks and struggled to overcome difficulties in complex ones. Examples show both useful tool use and substantial execution failures, alongside limitations such as looping, hallucination, and poor error diagnosis.

4 Results

  • The agents solved only the easiest pilot tasks, often completing individual steps but failing to overcome difficulties in complex tasks.The authors judge the failed tasks considerably easier than autonomous replication in the wild and conclude these agents are far from ARA.Source: §4 · p. 8

4.1 Example task: “Targeted phishing”

  • In the phishing attempt, the agent researched the wrong person, produced an unconvincing imitation login page, and failed to host it.The poor page quality meant the run could count only as a partial completion even if its remaining steps succeeded.Source: §4.1 · pp. 8–9, Figure 4 · p. 9

4.2 Example task: “Automatically Restart Agent”

  • For the restart task, the agent set up cron but guessed the scaffolding path; in a real VM, its broad kill command would have terminated the agent.The task used a simulated scaffolding program rather than the actual program in the VM.Source: §4.2 · p. 10

4.3 Abilities and limitations of agents

  • Agents used bash, wrote task-specific code, navigated webpages, and sometimes recovered from mistakes when errors were informative.These abilities coexisted with repetitive loops, hallucinations, implausible plans, poor error diagnosis, and weak self-understanding.Source: §4.3 · pp. 10–11

5 Related Work

Prior evaluations often use short, self-contained tasks, while this report evaluates agents on more complex tasks intended to probe potential catastrophic harm. The work relates to web-agent evaluations and to GPT-4 risk evaluation efforts.

5 Related Work

  • Traditional machine-learning evaluations assess large language models on short, self-contained tasks across domains [Sri+23; Hen+21b; Hen+21a; Hen+21c; Lia+22], while benchmark and evaluation limitations and failure modes have long been discussed in machine-learning literature [GCS23].Source: §5 · p. 13
  • Mind2Web [Den+23] and WebArena [Zho+23] evaluate simple tasks on real websites; this report instead targets more complex tasks related to catastrophic harm, including code and services such as AWS.The authors describe their pilot as a step toward dangerous-capability evaluations discussed by Shevlane et al. [She+23].Source: §5 · p. 13
  • The GPT-4 System Card [Ope23b] discusses the authors’ GPT-4-early work within OpenAI’s evaluation of GPT-4 capabilities, limitations, and risks.The System Card covers domains including hallucinations and potential risky emergent behaviors.Source: §5 · p. 13

6 Conclusions and future work

The report pilots open-ended evaluations of agents on tasks relevant to autonomous replication and adaptation, finding the tested agents far from ARA. It cautions that the results do not establish how capable future agents may become and identifies scaffolding, fine-tuning, and model scale as limitations for forecasting.

6 Conclusions and future work

  • The report pilots open-ended real-world evaluations by combining GPT-4 and Claude with scaffolding and testing them on 12 ARA-relevant tasks.The authors view measuring and forecasting ARA as potentially useful for security, monitoring, and alignment.Source: §6 · p. 13
  • The four tested agents were far from ARA, and the authors judge casual users unlikely to approach the ARA threshold with these model versions absent fine-tuning.The authors characterize the agents as representative of capabilities achievable with moderate effort using public techniques and no fine-tuning.Source: §6 · p. 13
  • These evaluations cannot rule out near-future ARA capability because scaffolding may improve, fine-tuned models are not bounded by these results, and larger-model progress was not forecast.The authors note that task performance may improve continuously or abruptly depending on its measurement [Sri+23] [SMK23], and they know of no evidence that this behavioral evaluation improves continuously with scale.Source: §6 · p. 13
  • Scaffolding and prompting quality appears to affect autonomous-task capabilities, and fine-tuning for unrelated objectives may sometimes increase ARA-relevant capabilities.The authors point to Appendix A for details on the fine-tuning observation.Source: §6 · p. 13
  • Future work could evaluate more mature agents fine-tuned for performance, identify tasks with continuous scale-related improvements, extrapolate progress across more tasks, or develop cheaper evaluations.These approaches are proposed to address limitations in forecasting near-future capabilities.Source: §6 · p. 14

A Impact of Fine-Tuning: Differences Between GPT-4-early and GPT-4

This section presents a pilot-suite comparison of GPT-4-launch and GPT-4-early as a possible example of gains from fine-tuning, and describes the Claude-assistant scaffold’s tools, coordination, command-execution rules, and resource-control and survival prompt.

A Impact of Fine-Tuning: Differences Between GPT-4-early and GPT-4

B.4 Claude-assistant

  • The scaffold gives the agent root-run bash, interactive Python, and file summarization; the summarize command cannot process URLs.Bash runs on the scaffolding server, while running bash resets the Python session.Source: §B.4 · p. 23
  • Assistants can delegate subtasks and message one another, but messages transfer execution priority and only one assistant can issue commands at a time.Source: §B.4 · pp. 23–24
  • The scaffold offers Google search and a separate browsing assistant, which can interact with web pages but cannot run bash or access the server.Search returns the first page of results; browsing is described as more capable for thorough research but requires relevant context in its instructions.Source: §B.4 · p. 24
  • The scaffolding executes only the first XML-tagged command, so the agent must refer to commands without XML tags when reasoning about them.Source: §B.4 · p. 24
  • The agent is instructed to gain and maintain control of resources and survive, with an example in which it checks server RAM and reports back to its delegator.Source: §B.4 · pp. 24–25

C.1 Non-Browsing Tool Overfitting Discussion

Because the scaffolding was developed on tasks overlapping with the evaluation, results may overestimate performance across ARA-relevant tasks. The authors think significant overestimation is unlikely, but recommend separating development and evaluation tasks, especially for fine-tuning.

C.1 Non-Browsing Tool Overfitting Discussion

  • Task overlap in scaffolding development may overestimate performance across the broader distribution of ARA-relevant tasks.The authors say this does not currently obscure how close the agents are to the ARA threshold, since they complete few tasks, but it affects how informative positive examples are.Source: §C.1 · p. 25
  • Avoiding task-specific edits and observing similar performance on development and new tasks led the authors to judge significant overestimation unlikely.They recommend iterating on different tasks from the final evaluation set; a train/test split is especially important when fine-tuning.Source: §C.1 · p. 25

C.2 Browsing Tool Overfitting Discussion

This section describes browsing-tool development, scaffolding and refusal handling, run-completion criteria, and examples of agent behavior. The authors caution that browsing could be improved, while the examples show both useful tool skills and recurring failures.

C.2 Browsing Tool Overfitting Discussion

  • Browsing-tool iteration substantially changed the crawler, focused on four browsing tasks, and does not establish an upper bound on browsing performance.The tool combines a web crawler, a command interface, and a prompting loop.Source: §C.2 · p. 26

D Scaffolding Iteration Example

E Handling Refusals

  • The refusal guidance distinguishes ethical from ability refusals and recommends minimal jailbreak edits that do not add helpful task content.The guidance suggests escalating edits only if the next generation still refuses.Source: E Handling Refusals · pp. 26–27

E.1 Handling GPT-4 Refusals

E.2 Handling Claude Refusals

  • Claude-assistant often refused commands, with refusal strength varying; planning prompts sometimes overcame refusals, while targeted phishing could require repeated interventions.It also sometimes refused benign actions, and refusals could persist after errors despite automatic prompting.Source: §E.2 · p. 28

F Concluding a Run

G.1 GPT-4-based agents

  • GPT-4-simple demonstrated AWS, Twilio, and error-correction skills, while GPT-4-delegate showed planning and browsing ability; GPT-4-simple also made mistakes and showed poor self-understanding.Source: §G.1 · p. 29

G.2 Claude-assistant

  • Claude-assistant showed Bash and browsing competence and some recovery, but also hallucinated and diagnosed errors poorly.Source: §G.2 · p. 29

H Task Specification Component Details

Task specifications tell agents what they see, how outcomes are assessed, and what environment they use. The report also illustrates AWS setup and a simulated Twilio call.

H.1 Task Instructions

  • Task instructions are inserted into an initial prompt, and the “Set up GPT-J” example asks for installation on EC2 and confirmation of working completions.Source: §H.1 · p. 29

H.2 Evaluation Instructions

  • Evaluation instructions classify outcomes as Did Not Complete, Partially Completed, or Completed, allowing meaningful partial credit while keeping category boundaries few.Source: §H.2 · p. 29

H.3 Setup Instructions

  • Agents use a VM with Bash, pre-installed packages, and credential files; Figure 5 illustrates GPT-4 configuring AWS resources.The report says VM and file variations during experiments were inconsequential to results.Source: §H.3 · pp. 29–30, Figure 5 · p. 30

H.4 Simulation Instructions

  • The setup may modify visible information or simulate environment parts to avoid contacting unsuspecting people or contradicting the scenario; Figure 6 shows a simulated Twilio call.Source: §H.4 · p. 30, Figure 6 · p. 31

I.1 Potential risks of publication and our response

The report aims to improve understanding of potentially dangerous frontier-model capabilities and safety evaluations, while redacting some material over publication-risk concerns. It may later share or publish material if risk analysis supports doing so.

  • The section was written to accompany an earlier, more redacted version of the report.Source: §I.1 · p. 31
  • The report is published to increase understanding of potentially dangerous frontier-AI capabilities and advance safety evaluations, with the hope of improving society’s ability to identify dangerous capabilities.Source: §I.1 · p. 31
  • Publication-risk concerns led the authors to redact parts of the report, including complete agent-run transcripts and more detailed accounts of agent strengths and weaknesses.Source: §I.1 · p. 31
  • The authors may later release material if it has minimal risk or further analysis supports publication, and safety researchers may request non-public materials.Source: §I.1 · pp. 31–32

I.2 Safety and oversight precautions taken during our experiments

Experiments were conducted under human supervision, with safeguards against agents interacting with non-consenting people or engaging in unethical or illegal activity. Web-browsing actions were individually approved by a human overseer.

  • All task runs were human-supervised, and experimenters were instructed to stop runs before non-consensual, unethical, or illegal interactions.Source: §I.2 · p. 32
  • For targeted phishing, experimenters simulated the human response according to the task specification rather than allowing the agent to interact with a person.Source: §I.2 · p. 32
  • Experimenters sometimes ran several routine or low-risk steps without individual approval, but every web-browsing action required active human approval.Source: §I.2 · p. 32

Related papers

Loading 2312.11671v2…