PaperlayerView exact revision on arXiv

Source-linked AI summary

AgentBrew: Offline Tool-Use Agent Learning from Raw Real-World Trajectories

Zhiyi Lyu, Yewen Li, Longtao Zheng, Shengtian Yang, Lang Feng, Lei Feng, Peng Jiang, Kun Gai, Qingpeng Cai, Bo An

arXiv:2609.05837v1cs.AI

TL;DR

Real-world tool-use agents are difficult to train because applications lack predefined tasks, verifiers, faithful simulators, and ample interaction budgets. AgentBrew learns from one batch of unfiltered trajectories by inferring aligned instructions and assigning per-action PMI credits, improving Qwen3-32B across three MCP applications while surpassing larger-model and filtering baselines.

  • Problem

    Real-world tool-use environments lack predefined tasks, ground-truth verification, faithful simulators, and feasible large-scale interaction, making environment-specific agent training difficult.

  • Method

    AgentBrew retrospectively infers each trajectory’s accomplished instruction and decomposes its information about that instruction into additive per-action PMI credits for offline policy training.

  • Results

    AgentBrew consistently outperforms training-based baselines across three real-world applications, including +5.9 Acc / +10.3 Score over Rejection Sampling and +7.9 Acc / +10.6 Score over Vanilla SFT.

  • Takeaways & Limitations

    Fine-grained offline learning can extract useful supervision from noisy raw trajectories that filtering-based approaches would discard, enabling a 32B model to surpass Qwen3-235B across three applications.

  • Takeaways & Limitations

    The evaluation covers only three MCP applications, and PMI may not capture all causal effects in highly entangled action sequences.

Abstract

from arXiv · show

LLM-based agents are increasingly deployed in real-world applications through tool-use APIs, yet training them for specific environments remains fundamentally difficult: real-world applications provide no pre-defined tasks or verifiers, no faithful simulators, and limited budget for large-scale environment interaction. In this paper, we propose \textbf{AgentBrew}, an offline training framework that learns effective tool-use policies from a single batch of raw interaction trajectories, without task verifiers or iterative on-policy rollouts. The agent first explores the target environment to collect a raw trajectory corpus without quality filtering. To extract training signal from this noisy corpus, \emph{retrospective task inference} reconstructs an aligned instruction for each trajectory based on its actual outcome, and \emph{PMI-Based credit assignment} decomposes the trajectory's total information about the inferred instruction into additive per-action credits via pointwise mutual information (PMI). These credits weight the policy training objective, amplifying informative actions while suppressing ineffective ones. On three real-world MCP applications (GitHub, Notion, PostgreSQL), AgentBrew improves Qwen3-32B by +8.7 Acc / +9.7 Score on average, surpassing Qwen3-235B (+2.3 / +4.4) and outperforming rejection sampling (+5.9 / +10.3). These results demonstrate that fine-grained offline learning can recover useful supervision from raw trajectories that filtering-based approaches would discard. The code is available at https://github.com/alphatogo/AgentBrew

1 Introduction

AgentBrew addresses environment-specific tool-use training when tasks, verifiers, simulators, and interaction budgets are limited by learning from one unfiltered trajectory batch. It realigns each trajectory to its actual outcome and assigns action-level credits, yielding improvements across three real-world MCP applications.

  • Motivation: Environment-specific tool-use training is difficult because applications expose distinct, stateful tool landscapes and general-purpose models achieve low completion rates.The motivating environments include Notion and GitHub, whose APIs require precise parameter composition and live-state awareness.
  • Motivation: Existing approaches rely on verified task suites and filtering, or on simulators and repeated rollouts, requiring prerequisites unavailable in many real-world applications.These prerequisites include predefined tasks, reliable rewards, sufficient interaction budgets, or faithful simulators.
  • AgentBrew: Trajectory-level filtering can discard entire failed trajectories, whereas AgentBrew preserves potentially useful actions by assigning credits under an inferred task.This allows the entire raw corpus to contribute to training rather than retaining only trajectories judged successful.
  • AgentBrew: AgentBrew reconstructs an aligned instruction for each raw trajectory and uses PMI-based per-action credits to weight policy training.The framework first explores the environment and collects trajectories without quality filtering, then distills supervision from them.
  • Results: +8.7 Acc / +9.7 Score on average over Qwen3-32B was achieved across GitHub, Notion, and PostgreSQL without external reward signals or iterative interaction.AgentBrew also surpassed Qwen3-235B and outperformed rejection sampling.

2 Problem Formulation

The problem is to improve a tool-use policy from a single unlabeled trajectory corpus in a real-world environment where tasks, verifiers, rewards, and further interaction are unavailable. The agent must extract useful training signal despite trajectories lacking ground-truth success verification.

  • Setting: The environment exposes callable tools with natural-language descriptions and typed parameters, and the agent produces tool-call actions from observed interaction states.A trajectory records states, actions, and environment observations over discrete steps.
  • Constraints: Training must proceed without predefined tasks or verifiers, large-scale additional interaction, or external reward signals.The available data is a single trajectory corpus collected in the target environment.
  • Constraints: Trajectory termination does not verify success because it occurs after a completion action or a maximum step limit.The objective is to extract a training signal from unlabeled trajectories that improves the policy on user tasks in the environment.

3 Methodology

AgentBrew is a three-stage offline framework that grounds exploration in the live environment, realigns raw trajectories to their achieved outcomes, and uses PMI-derived credits to weight policy training. It retains naturally terminated trajectories without quality filtering while emphasizing actions that provide information about inferred tasks.

  • Grounded Exploration: AgentBrew collects grounded task trajectories from the live environment and retains naturally terminated interactions without quality filtering.Tasks are grounded in existing entities and configurations; trajectories exceeding the turn limit are discarded as degenerate.
  • Experience Distillation: Retrospective task inference replaces each original instruction with one consistent with the trajectory’s observable outcome.The inferred instruction describes only genuinely achieved, trajectory-supported results and introduces no unsupported entities or outcomes.
  • PMI-Based Credit Assignment: PMI-based credit assignment decomposes trajectory-level information about the inferred instruction into additive per-action contributions.The chain rule assigns each action’s marginal information contribution, so informative actions receive positive credit while redundant or misleading actions receive near-zero or negative raw credit.
  • PMI-Based Credit Assignment: Each action credit measures the reduction in inferred-instruction NLL after conditioning on that action and preceding observations.The total information is ΔL = L∅−LTn, while the per-action credit is wt = Lt−1−Lt.
  • PMI-Weighted Policy Training: Negative credits are clamped to zero and normalized within each trajectory before weighting the policy’s negative log-likelihood objective.The weighted loss applies to full actions, including chain-of-thought tokens, amplifying high-credit behavior and suppressing low-contribution actions.

4 Experiments

AgentBrew is evaluated on three real-world MCP applications using raw trajectories and compares favorably with model-scaling, training, credit-assignment, task-inference, noisy-data, and transfer baselines. The experiments show that retrospective alignment and per-action PMI credit recover useful supervision without external rewards or additional environment interaction.

  • Experimental setup: AgentBrew is evaluated on GitHub, Notion, and PostgreSQL against frontier models, open-source models, and Qwen3-32B training baselines.The setup includes Vanilla SFT, Rejection Sampling, and TOUCAN comparisons.
  • Main results: AgentBrew achieves average 18.2 Acc and 58.7 Score, surpassing its Qwen3-235B variant by +2.3 Acc and +4.4 Score.The results target data efficiency through environment-specific offline adaptation rather than model scaling alone.
  • Main results: AgentBrew improves over Vanilla SFT by +7.9 Acc / +10.6 Score and over Rejection Sampling by +5.9 Acc / +10.3 Score across the three environments.On Notion, Rejection Sampling obtains Acc 0 while AgentBrew reaches 14.2 Acc on the same corpus.
  • Effect of PMI-Based Credit Assignment: PMI-based credit assigns low mean and total credit to misaligned trajectories, suppressing their gradient contribution without an explicit quality filter.The comparison fixes retrospective task-inferred pairs and varies only the per-action weighting strategy.
  • Effect of Retrospective Task Inference: 14.2 Acc and 70.1 Score result from inferred instructions on Notion, compared with 7.1 / 65.6 using original instructions under PMI-based credit.Retrospective task inference replaces original instructions with instructions reflecting each trajectory’s actual outcome.
  • Robustness to Noisy Trajectories: Adding lower-quality trajectories monotonically increases Notion performance, with Acc rising from 10.7 to 14.2 and Score from 61.7 to 70.1.The reported mechanism extracts informative actions from imperfect trajectories while suppressing ineffective ones.
  • Offline Transferability Across Models: AgentBrew-14B achieves 20.2 average Acc / 55.3 Score and improves over Qwen3-14B by +11.5 Acc / +12.5 Score.On Notion, AgentBrew-14B reaches 14.2 / 69.8 versus 14.2 / 70.1 for AgentBrew-32B.
  • Offline Transferability Across Models: A single offline distillation round can be reused to train smaller models without additional environment interaction.The reported transfer result attributes improvement to the quality of AgentBrew’s distilled signal rather than model capacity alone.

5 Related Work

Related work addresses tool-use training through task-defined trajectory collection or iterative optimization in simulated environments. These approaches emphasize obtaining high-quality task-trajectory pairs and finer-grained credit than trajectory-level rewards provide.

  • Training Data for Tool-Use Agents: The prevailing forward approach defines tasks, executes them with strong models, and retains successful task-trajectory pairs for training.This line of work depends on collecting and filtering trajectories against predefined task outcomes.
  • Policy Optimization for LLM Agents: On-policy methods train agents through iterative rollouts and environment rewards, while credit-assignment methods seek finer-grained signals than sparse trajectory-level rewards.The related methods include GRPO, DAPO, WebRL, DigiRL, and information-gain-based approaches.

6 Conclusion, Limitation, and Future Work

AgentBrew trains tool-use agents offline from raw interactions, using retrospective instructions and per-action credit to retain signal that filtering would discard. The paper reports scope and methodological limitations while suggesting learning from production user logs.

  • Conclusion: PMI-based credit assignment decomposes action contributions after retrospective task inference, enabling training from noisy trajectories.
  • Limitations: AgentBrew may degrade on extremely long or chaotic trajectories and PMI may not capture all causal effects in highly entangled action sequences.
  • Future Work: A proposed future direction is learning from unlabeled, noisy production user logs by recovering intent and identifying key session actions.
  • Pipeline: The pipeline has three stages: raw experience collection, offline experience distillation, and offline PMI-weighted policy training.
  • Scope: The evaluation covers GitHub, Notion, and PostgreSQL, spanning code collaboration, knowledge management, and relational database administration.

C Task Proposal

The task-proposal process builds diverse, realistic workflows from tool descriptions and grounds each generated instruction in entities discovered through a read-only environmental inspection.

  • Category Generation: Task categories are generated from complete tool descriptions to represent realistic human intents and meaningful multi-tool workflows.
  • Category Generation: Categories should cover diverse difficulty levels and differ in tool combinations and reasoning patterns.
  • Category Generation: The procedure generates categories that collectively maximize coverage of the available tool set.
  • Grounding: A single read-only tool call grounds each task in concrete entities or configurations from the live environment.
  • Grounding: Generated tasks follow a workflow pattern, use an example task for style, and contain identifiers confirmed by the inspection.
  • Credit Assignment: PMI credit is computed from reductions in reference-model NLL after successive trajectory prefixes reveal actions and observations.
  • Credit Assignment: Negative credits are clamped, and trajectories with only non-positive credits receive zero normalized weights.
  • Credit Assignment: The mean normalized weight is 0.75 overall, below the uniform baseline of 1.0, because many actions receive near-zero credit.

E Retrospective Task Inference

Retrospective task inference revises each original task so it matches what the trajectory actually completed, using successful state-changing operations as primary evidence.

  • Inference Procedure: Each trajectory is converted into a revised instruction inferred from its reduced tool-interaction trajectory.
  • Inference Procedure: The inference prompt minimally edits the original task and removes or weakens unsupported subtasks.
  • Evidence Rules: Successful state-changing operations receive priority, while retrieval steps are used when they identify objects involved in successful writes.
  • Evidence Rules: Partial completion is represented by retaining only the completed subset of a multi-step task.
  • Evidence Rules: The revised instruction must not claim unsupported outcomes or invent entities, artifacts, or results absent from the trajectory.

F Experiment Details

The experiments use sliding-window memory to manage very long tool-use trajectories while preserving recent details and compressed information about earlier steps.

  • Context Management: Real-world tool-use trajectories can exceed 100K tokens because API responses may contain verbose page content or query results.
  • Context Management: Each step combines compressed long-term summaries of earlier steps with full uncompressed content from the three most recent steps.
  • Context Management: The memory mechanism preserves fine-grained recent details while retaining high-level progress from earlier steps.
  • Summarization: Step summaries record attempted actions, discovered facts, constraints, decisions, and identifiers needed later, while remaining under 500 words.
  • Experimental Control: The same context-management procedure is applied across methods and models so performance differences reflect training approaches rather than memory advantages.

F.2 Training Hyperparameters

Training uses PMI-derived per-action credits as sample weights in a custom weighted cross-entropy objective, with zero-credit samples excluded before optimization.

  • PMI-derived per-action credits weight each training sample in the policy’s supervised fine-tuning objective.The trainer applies each credit as a sample-level weight to token-averaged cross-entropy before batch aggregation.
  • Zero normalized-credit samples are filtered out before training because they contribute no gradient signal.
  • Qwen3-32B is fine-tuned for one epoch on the combined corpus from all three environments.
  • Training uses DeepSpeed ZeRO-Stage 3 on 4×H100 80GB GPUs with a 5 × 10^-6 learning rate and effective batch size of 64.The configuration also uses cosine scheduling, 10% warmup, BF16 mixed precision, gradient checkpointing, and a 20,000-token maximum sequence length.

F.3 Evaluation Protocol

Evaluation uses programmatic verifiers that compare post-execution application state with ground-truth specifications, reporting both full-completion accuracy and average partial completion.

  • Each test task receives a verifier score in [0, 1] based on how closely the resulting environment state matches its ground-truth specification.Verifiers inspect outcomes such as created resources, modified database rows, and updated page content.
  • Acc is the fraction of test tasks with a score of 1.0, while Score is the average per-task score.

G Case Studies

Case studies in GitHub, Notion, and PostgreSQL show how retrospective task inference aligns instructions with observed outcomes and PMI credit assignment highlights useful actions in imperfect trajectories.

  • Environment case studies: The case studies cover three real-world tool-use environments: GitHub, Notion, and PostgreSQL.
  • Core mechanisms: Retrospective task inference rewrites each original instruction into an outcome-aligned task, while PMI credit assignment identifies actions useful for policy learning.
  • GitHub: In the GitHub example, repository inspection and documentation listing support migration-note creation, while adding an issue comment receives +0.087 credit.The aligned task audits acme-corp/ml-serving and publishes migration notes rather than completing every originally requested step.
  • PostgreSQL: The PostgreSQL trajectory partially completes a student-loan compliance dashboard because available status tables lack temporal or monthly fields.
  • PostgreSQL: In the PostgreSQL example, schema inspection and monthly enrollment-table construction receive positive action credits, while a redundant schema check receives only +0.002.The inferred task focuses on creating an enrollment trend dashboard while excluding disabled students and joining through person.name.
Loading 2609.05837v1…