Aviary: training language agents on challenging scientific tasks

1 Introduction

• Aviary defines language decision processes and represents language agents as stochastic computation graphs, enabling modular interchange of environments, agents, and optimizers.The LDP software package is designed to simplify experimentation across settings.Source: §1 · p. 1
• The paper introduces DNA construct engineering, scientific literature question answering, and protein design environments alongside recast GSM8K and HOTPOTQA benchmarks.The scientific tasks concern molecular cloning, finding specific literature passages, and proposing protein-stabilizing mutations.Source: §1 · p. 2
• Trained Llama-3.1-8B-Instruct agents with expert iteration and majority-vote sampling exceed human experts and frontier LLMs on DNA construct design and literature question answering.The paper frames agent optimization as iterative improvement through feedback from an environment.Source: §1 · p. 2

Language Agent Formalisms

• Language-agent formalisms include cognitive architectures and POMDPs, but differ in how they represent memory, planning, tools, and internal versus external actions.Some POMDP formulations distinguish internal from external actions, while others do not.Source: §2 · p. 2

Language Agent Optimization Frameworks

• Optimization frameworks target prompts, tools, model weights, inference settings, agent components, or computation-graph structure through varied techniques.Approaches include gradient-based or black-box prompt optimization, expert iteration, textual feedback, and graph-level optimization.Source: §2 · p. 3

Language Agent Benchmarks

• Scientific-agent benchmarks cover hypothesis generation, chemistry, data-driven science, scientific reasoning, and cycles of discovery, among other applications.Aviary emphasizes sequential decision-making with multiple agent-environment interactions and casts existing datasets as parametrizable tools acting on environment state.Source: §2 · p. 3

Contributions

• Aviary’s contributions include a formal LDP definition, five gym environments including three scientific ones, and open-source libraries for experimentation.The paper also reports non-frontier models trained online with inference-time sampling matching or exceeding frontier-model performance on these tasks.Source: §2 · p. 3

3.1 Language Decision Processes

• An LDP is a POMDP whose action and observation spaces are natural-language strings, with a policy mapping observations to actions and parameters encompassing optimizable agent components.The action space covers external environment interactions, while internal procedures such as memory retrieval and reasoning may be included among policy parameters.Source: §3.1 · pp. 3–4

3.2 Stochastic Computation Graphs

• Language-agent policies can be represented as stochastic computation graphs that take string observations and produce string actions, with deterministic and stochastic nodes.An SCG returns an action and a new agent state, allowing states and observations to be batched; memory retrieval can be incorporated into the graph.Source: §3.2 · p. 4
• Common architectures map to different graph structures: a language model samples directly, RAG retrieves before sampling, rejection sampling selects among samples, and ReAct samples reasoning then an action.Figure 2 distinguishes deterministic nodes as solid rectangles and stochastic nodes as dashed; it also marks the observation input.Source: §3.2 · p. 4, Figure 2 · p. 5

3.3 Training Methods

• Behavior cloning initializes Llama-3.1-8B-Instruct’s trajectory buffer because it cannot self-generate successful trajectories before training; expert iteration repeatedly filters rollouts by return and fine-tunes on retained data.The retained trajectories are appended to the buffer and used for supervised fine-tuning.Source: §3.3 · p. 5
• Inference-time majority voting selects the consensus response, but plateaus sooner than oracle verification, and unsure or truncated trajectories are omitted from voting.Oracle-verified pass@k can scale when a verifier can identify any correct solution among the sampled completions.Source: §3.3 · p. 5

4.1 GSM8K

• GSM8K supplies grade-school word problems for multi-step mathematical reasoning and exposes a calculator tool.Its dataset has 7,473 training questions and 1,319 test questions.Source: §4.1 · p. 6

4.2 HOTPOTQA

• HOTPOTQA agents answer questions using a Wikipedia API that provides access to articles and sections without given context.Evaluation uses 7,405 questions from the dataset’s evaluation subset.Source: §4.2 · p. 6

4.3 PaperQA

• PaperQA2 Local adapts PaperQA2 to local PDF, text, and HTML search, omitting citation traversal and adding a tool for tool selection and answer-completeness declarations.LitQA2 has 248 questions, including 49 held-out test questions; the public remainder is split into 149 training and 40 evaluation questions. Search indexes contain 18,955 train, 5,457 evaluation, and 5,519 test DOIs.Source: §4.3 · p. 6

4.4 Molecular Cloning

• The molecular-cloning environment combines plasmid annotation, natural-language sequence search, and protocol tools for tasks such as primer design and DNA assembly.Evaluation uses 150 SeqQA test questions after omitting 10 RNA-related questions; its tools use 0-indexing although SeqQA assumes 1-indexing, and they cover both reading frames while SeqQA considers coding open reading frames.Source: §4.4 · pp. 6–7

4.5 Protein Stability

• The protein-stability environment combines sequence and structure analysis tools, including Rosetta aggregation-propensity scoring, to support agents proposing stabilizing mutations.The evaluation uses 40 proteins randomly selected from the megascale stability dataset, excluding proteins mentioned in the cited text, and assesses mutations with Rosetta cart_ddg.Source: §4.5 · p. 7

5 Results

• Training LDP agents improves over same-architecture untrained agents; smaller Llama models can match Sonnet on challenging tasks, though GSM8K agents do not improve over zero-shot.Source: §5 · p. 8

5.1 Behavior Cloning and Expert Iteration

• Behavior cloning seeded SeqQA expert iteration with 2841 valid trajectories; eight online epochs added 14% absolute improvement, with a train–test gap indicating some overfitting.Source: §5.1 · p. 9
• For LitQA2, an untrained agent reached 30% accuracy; behavior cloning used 430 trajectories, and expert iteration raised test performance to 72%.Source: §5.1 · p. 9
• Demonstration trajectories were correct, longer, and more diverse; after online learning, the agent found shorter, less diverse paths, suggesting convergence on a subset of possible solutions.Source: §5.1 · p. 9

5.2 Inference Compute Scaling

• Majority voting improves accuracy by about 20 percentage points on both environments; SeqQA consensus sampling reaches 0.89 after 945 rollouts, while Claude 3.5 Sonnet majority voting scores 0.89 on LitQA2’s test set.Source: §5.2 · p. 10

5.3 Inference Cost Scaling

• 100x less cost matches Claude 3.5 Sonnet’s SeqQA accuracy with Llama-3.1-8B EI; on LitQA2, Llama majority voting beats single-rollout Sonnet for 3x less cost.Source: §5.3 · p. 11

6 Discussion

• Aviary frames multi-step reasoning and tool use as language decision processes and implements five environments, including three focused on biology tasks.The discussion focuses on LitQA and SeqQA multiple-choice benchmarks and protein stability.Source: §6 · p. 11
• Behavior cloning and expert iteration help smaller models reach frontier-model performance; majority voting further improves results, especially for agents.The authors suggest greater trajectory stochasticity may explain why voting benefits agents more than non-agentic LLMs; protein stability also benefits from more inference compute with oracle verification.Source: §6 · pp. 11–12
• At 100x lower cost, trained agents can exceed human and frontier-model accuracy, while an 8B model makes about 1,000 rollouts per task feasible.SeqQA trajectories average $0.07 with Claude-3.5-Sonnet and $0.00066 with Llama-3.1-8B EI; human contractors cost $4–$12 per question.Source: §6 · p. 12
• Human comparisons are tool-asymmetric, and benchmark comparisons with prior results are not exactly matched.The authors also note that complex software dependencies and newly available splits complicate comparisons; splits were chosen to avoid possible pretraining contamination.Source: §6 · p. 12

7 Conclusion

• Language agents exceed zero-shot frontier LLMs on SeqQA, HOTPOTQA, LitQA2, and protein stability, and exceed human performance on SeqQA and LitQA2.Trained Llama-3.1-8B EI agents match or exceed humans and frontier LLMs on LitQA2 and SeqQA at significantly lower cost.Source: §7 · pp. 12–13
• The language decision process framework formally describes language-agent tasks and casts language agents as stochastic computation graphs.The conclusion presents behavior cloning, expert iteration, and inference-time sampling as methods used to improve performance.Source: §7 · p. 13
• Aviary and LDP are open-source frameworks intended as libraries for implementing environments and language agents..Source: §7 · p. 13

A Environment Details

• Table A1 catalogs environments implemented within the Aviary framework.The supplied caption identifies the table’s scope but does not provide its entries.Source: Table A1 · p. 21
• Example Task Element Q asks which base editor has been shown to be most efficient for inducing K352E in CD45 in human T-cells, and lists five options: ABE8e-NG, ABE8e–SpRY, SPACE-NG, insufficient information, and ABE8e-SpG.Source: A Environment Details · p. 21
• PaperQA searches full-text papers, reranks and summarizes evidence, proposes an answer, then terminates with a success declaration.The listed tools are paper_search, gather_evidence, gen_answer, and complete.Source: A Environment Details · p. 21
• The task rewards one point for a correct answer, −1 for an incorrect answer, and 0.1 for an unsure answer; transitions are nondeterministic because LLM tools are stochastic.The implementation is identified as paper-qa==5.6.1.Source: A Environment Details · p. 21

• Search retrieves matching or similar Wikipedia pages, lookup searches within the results, and submit_answer checks whether an answer is correct.Source: HOTPOTQA · p. 21
• A correct answer earns 1 reward; an incorrect answer earns 0.Source: HOTPOTQA · p. 21
• The environment’s transition function is deterministic.Source: HOTPOTQA · p. 21