PaperlayerView exact revision on arXiv

Source-linked AI summary

AgentTune: A Unified Training Lifecycle for Tool-Using LLM Agents

R E Zera Marveen Lyngkhoi, Abhivansh Gupta, Vidushee Vats, Ram Mohan Rao Kadiyala, Vinay Kumar Sankarapu, Pratinav Seth

arXiv:2610.13335v1cs.LG

TL;DR

Tool-using agent systems often handle execution, training, evaluation, and recovery in separate tools and representations. AgentTune uses a shared EventLog and composable library components to support these stages. Its strongest task-level evidence is a legal contract-search result, while the paper notes that its evaluations are internal comparisons.

  • Problem

    Agent tooling addresses individual lifecycle stages in isolation, leaving RL rollouts, workflow states, and evaluation traces in different representations.

    Source: §1 · p. 1

  • Method

    AgentTune uses a two-tier EventLog for trajectory projections and provides a shared runtime, RL core, distillation, RAG, decision workflows, evaluation, and a designed recovery path.

    Source: §3.3 · p. 4, §8 · p. 22

  • Results

    GRPO raises held-out CUAD exact match from 60.0% to 69.0% and reduces searches per question from 3.00 to 2.90 in the legal case study.

    Source: §1 · p. 2, §5.1.3 · p. 17

  • Takeaways & Limitations

    The EventLog supports projections across agent execution, training, evaluation, and recovery, while the reported experiments establish representative components separately.

    Source: §8 · p. 22

    The reported numbers are internal comparisons, and the release lacks integration with external agentic benchmarks.

    Source: Evaluation · p. 22

Abstract

from arXiv · show

Tool-using agents increasingly require a common substrate for building, training, evaluating, distilling, and maintaining agentic systems, yet existing tooling typically addresses these stages in isolation. We present AgentTune, a source-available Python library that provides a unified framework for developing and optimizing tool-using agents. At its core, AgentTune introduces a two-tier EventLog representation that preserves agent trajectories while supporting consistent projections for training, evaluation, and auditing. Building on this substrate, the library provides a shared agent runtime, tool-calling reinforcement learning, agentic distillation, retrieval-augmented generation, decision workflows, evaluation utilities, and a designed path toward self-healing agents. These components are exposed through composable interfaces so that agent strategies, training methods, evaluation procedures, and recovery mechanisms can be combined without requiring separate representations or pipelines. We demonstrate the framework across agentic tasks including legal contract search and domain-specific applications, illustrating its use for trajectory-based optimization, tool-use evaluation, and cross-task agent development. AgentTune is released as a source-available library at https://github.com/Lexsi-Labs/AgentTune

1 Introduction

AgentTune addresses the fragmentation between agent execution, training, evaluation, and recovery by using a shared trajectory interface. The paper presents a composable library and reports its strongest task-level evidence in legal contract search.

1 Introduction

  • Unlike AgentTuning [8], AgentTune is a software library for agent development rather than a trajectory fine-tuning method and dataset.The paper locates the broader fragmentation at interfaces where RL rollouts, workflow states, and evaluation traces use different representations.Source: §1 · p. 1
  • GRPO raises held-out CUAD exact match from 60.0% to 69.0% and reduces searches per question from 3.00 to 2.90.The paper identifies this legal contract-search study as its strongest task-level result.Source: §1 · p. 2
  • The two-tier EventLog represents interactions once and projects them into training, evaluation, and audit formats.It accepts trajectories from the runtime, RL rollouts, DECIDE workflows, and evaluation traces, and projects records out to SFT rows, evaluator inputs, and audit records.Source: §1 · p. 2
  • AgentTune combines a shared runtime and training interface with tool-calling RL, agentic distillation, retrieval training, decision workflows, and a designed recovery path.The RL core supports GRPO, PPO, DPO, RLOO, and BCO; DECIDE provides YAML-defined workflows and audit trails.Source: §1 · p. 2

2 Installation and Getting Started

The getting-started material describes installation, runtime requirements, and a scripted quickstart. The example uses the same EventLog across execution and evaluation, and later stages can remain unchanged when the scripted policy is replaced with generation.

2 Installation and Getting Started

  • The source installation command is git clone, cd AgentTune, then pip install -e .; service, docs, dev, and flash-attn are optional extras.The one-command setup can install the agentic spine, RL core, RAG package, extra evaluation metrics, PostgreSQL DECIDE destination, and OpenEnv sandbox.Source: §2 · p. 2
  • The scripted quickstart builds an agent, runs one episode, and evaluates it without a GPU, network, or model download.The core imports PyTorch at package-import time, while training a model with an RL algorithm requires PyTorch and a CUDA GPU.Source: §2 · p. 2
  • Every quickstart call produces or consumes the same EventLog, and replacing the scripted policy with a real generation function leaves later training, distillation, and recovery stages unchanged.The example uses a scripted policy and an in-process tool; the paper points to a real GRPO run in Section 3.5.Source: §2 · p. 3

3 Library Structure

This section describes how AgentTune organizes tool-using agent development around a shared EventLog and composable library components. It covers the architecture, trajectory schema, workflows, and reusable training, evaluation, orchestration, and recovery tools, including configurable safeguards and deployment interfaces.

3.1 Scope and definitions

  • AgentTune reuses TRL optimizers, supplies a trajectory substrate rather than a retrieval-ranking method, and provides neither a security sandbox nor certified decision logic or benchmark-leadership claims.It also inherits TRL’s single-node scaling limits; untrusted code requires a remote sandbox, and the bundled domain logic is plausible rather than regulator-approved.Source: §3.1 · p. 3

3.2 Package layout and high-level architecture

  • Five layers connect interfaces, agent runtime, EventLog, learning and evaluation, and production improvement, whose gated recovery cycle returns audited failures and corrected trajectories to the EventLog.The layers are additive rather than a fixed stack, and evaluation cuts across them; the shared RL core is reached by multiple components, while distillation uses a separate trainer-factory path.Source: §3.1 · p. 3, §3.2 · p. 4

3.3 The EventLog schema

  • EventLog is an ordered list of typed events in light or full tiers, with projections for training, evaluation, and auditing; only full-tier logs support token-mask and SFT-row operations.Light logs record trajectory content, while full logs also carry token spans and log probabilities; the full-only operations raise ValueError on light logs.Source: §3.3 · p. 4, §3.3.1 · p. 4, §3.3.3 · p. 6

3.3.2 Segment-aware, completion-only loss

  • The SFT projection creates one row per full-tier log, applies normal loss to reasoning and final text, double-weights tool calls, and masks tool results and observations.It drops REWARD, TURN_COMPLETE, and MEMORY_OP events, so MemoryReAct trajectories lose their memory operations in this training projection.Source: §3.3.2 · pp. 5–6

3.3.4 Schema properties and their limits

  • EventLog preserves event order and leaves native sources unchanged, but its training-weight and tier guarantees are conditional, and the schema cannot represent several important trajectory structures.Concurrent calls serialize, streaming and human-turn kinds are absent, multi-agent events lack attribution, and rejected search or retry branches flatten into the accepted path.Source: §3.3.4 · p. 6

3.4 The agentic spine

  • The agentic spine combines a strategy and harness through a Project lifecycle that logs steps, collects and evaluates trajectories, trains or distills from projections, and converts failures into training data.Strategies share an init/propose/observe/is_done interface; harness capabilities are checked by conformance tests that detect mismatches such as snapshot restoration that fails to revert state.Source: §3.4 · p. 6, §3.4.4 · p. 7, §3.4.1 · p. 7

3.5 The shared RL core

  • A shared RL core supplies five TRL-backed algorithms with a common multi-turn tool-calling loop, while rollout engines, reward functions, and judges support training and evaluation.Rollouts can use local transformers, vLLM, or hosted APIs; reward combinations are normalized, and judge defenses include score clamping, prompt rotation, relative scoring, and a vocabulary-overlap guard.Source: §3.5 · p. 8, §3.5.4 · p. 9, §3.5.5 · p. 10

3.6 Agentic distillation

  • Agentic distillation filters teacher trajectories for grounding, schema correctness, and answer accuracy, then converts survivors to SFT rows for a caller-supplied trainer.The student’s loss covers its own reasoning, tool calls, and answers, and Project remains independent of GPU and training-stack dependencies.Source: §3.6 · p. 10
  • Agentic distillation supports demonstrations, text-only teacher rollouts, and tool-using teacher rollouts that train students on the teacher’s tool-use policy as well as its prose.The workflow previews training rows without a GPU and checks for empty trajectory collections; its collection-to-distillation path can be exercised without a GPU or API key.Source: §3.6 · p. 10

3.7 Agentic RAG and data synthesis

  • On multi-hop CUAD questions, SQLiteFTSBackend, ChromaBackend, and HybridBackend achieved gold-chunk R@8 of 19.3%, 18.1%, and 21.3%, with top-8 gold-passage hit rates of 83.3%, 81.7%, and 90.0%, respectively.The synthesis pipeline builds a typed graph, samples two-to-five-hop paths, verifies chain-dependency and retrieval necessity, balances hop count with retrieval difficulty, and creates leakage-safe splits. Standalone utilities include curriculum sampling and a trajectory store that logs rollouts, questions, rewards, and per-step thoughts, actions, and observations.Source: §3.7.1 · p. 11, §3.7.2 · p. 11, §3.7.3 · p. 11

3.8 DECIDE: decision workflows as YAML

  • DECIDE compiles YAML templates into LangGraph workflows that can run inference, collection, evaluation, or training, with configurable thresholds before training or deployment.The library ships 24 templates and includes step limits and an append-only audit log.Source: §3.8 · p. 12

3.9 The self-healing closed loop

  • The self-healing loop classifies failures, generates and replays corrected examples, then retrains only after trigger and safety checks and deploys only if held-out and trajectory-evaluation scores pass.A background retrainer runs on a daemon thread while the loop continues ingesting; deployment includes an automatic backup and rollback path.Source: §3.9 · pp. 12–13, Table 6 · p. 13

3.10 Multi-agent orchestration

  • AgentTuneGraph composes rollout and judge nodes into a LangGraph StateGraph and compiles callables for rollout, per-completion GRPO rewards, and standalone trajectory scoring.An optional router selects one rollout node per batch, and duck-typed nodes allow testing with fakes.Source: §3.10 · p. 13

3.11 Evaluation

  • Evaluation utilities cover generation metrics, deterministic trajectory measures, LLM-judged assessment, standard lm-eval benchmarks, and tool-using task shapes.Code-generation evaluation uses a timed subprocess and restricted builtins, but does not enforce filesystem or OS-level confinement.Source: §3.11 · pp. 13–14, Table 7 · p. 14

3.12 Tool library, CLI, and packaging

  • AgentTune provides built-in and wrapped tools, accepts custom tool objects or documented callables, and offers Python SDK, CLI, and optional FastAPI/WebSocket interfaces.Training requires a current Python version and PyTorch with a CUDA GPU; service, docs, development, and flash-attention packages are optional extras.Source: §3.12 · p. 14

4 Case Studies

Section 4 presents representative evidence from training smoke tests, the DECIDE closed loop, and orchestration testability. It also identifies limits on the scope and strength of that evidence.

4 Case Studies

  • Table 12 counts six projections against up to fifteen converters, but instrumentation-cost evidence lacks a controlled integration-effort measurement.Reported GPU results use one NVIDIA RTX PRO 6000 Blackwell Max-Q workstation GPU; the LegalBench-RAG transfer evaluation is internal, with its harness and outputs absent from the public mirror.Source: §4 · p. 14

4.1 Training smoke tests

  • 1/4→4/4 contract rate and 0/4→3/4 correct-label rate across four sampled completions per probe; distillation loss is in-sample.These smoke tests exercise trainer paths end to end and show policy movement, not competitive benchmark performance.Source: Table 8 · p. 15, §4.1 · p. 15

4.2 The DECIDE closed loop end to end

  • 0.33 to 1.00 accuracy over six held-out cases reconstructed from past successful runs; the gate relied on accuracy alone.A self-heal DPO retrain preceded LoRA adapter deployment and reload; the trajectory arm was not exercised.Source: §4.2 · p. 15

4.3 GPU-free testability and dependency footprint

  • Injected models, rollout engines, and runners permit orchestration tests without trainable models; actual RL and model-backed rollout paths require model, GPU, or API dependencies.Tested surfaces include Project lifecycle plumbing, EventLog projections, harness conformance, DECIDE graph compilation, closed-loop triggers, synthesis stages, and deterministic trajectory metrics.Source: §4.3 · p. 15

5 Use Case

The legal case study follows contract search from question synthesis through search-policy training and distillation, then sketches how the pattern could apply beyond legal work. The experiments report improved in-domain and transfer results, alongside student-distillation comparisons.

5.1 Compact legal search agents

  • The case study takes a legal contract-search agent from raw-corpus synthesis through search-policy training to a distilled student.One-shot top-8 hybrid retrieval makes answers reachable for 68.3% of multi-hop CUAD questions, versus 93.3% with perfect retrieval.Source: §5 · p. 15, §5.1 · pp. 15–16, Figure 2 · p. 16

5.1.1 Task and dataset curation

  • From 90 CUAD training contracts, curation retains 3,992 verified questions spanning two to five hops, with gold passages kept disjoint across train and validation.The pipeline begins with 5,361 candidates, retains 4,364 after three checks, then removes 372 at a final answerability gate.Source: §5.1.1 · p. 16, Figure 3 · p. 16

5.1.2 Experimental setup

  • The search policy is a Qwen3.5-9B model trained with GRPO and LoRA, rewarded for answer correctness, evidence coverage, and search frugality.Evaluation uses greedy decoding, and the frozen policy is also tested on three previously unseen LegalBench-RAG corpora.Source: §5.1.2 · pp. 16–17

5.1.3 Comparative analysis

  • Exact match rises from 60.0% to 69.0% on held-out CUAD questions while searches per question fall from 3.00 to 2.90.The frozen policy also improves Doc@8 and reduces searches on all three unseen corpora; MAUD’s Doc@8 gain is a ceiling effect.Source: §5.1.3 · p. 17, Table 9 · p. 17, Table 10 · p. 17
  • On MAUD, Doc@8 moves from 0.98 to 1.00 and searches from 5.0 to 4.5; PrivacyQA Doc@8 rises from 0.83 to 0.98, with ContractNLI between them.The frozen CUAD-trained policy transfers without further training; the transfer figure reports improvements in Doc@8 and fewer searches on each corpus.Source: §5.1.3 · p. 17, Table 10 · p. 17, Figure 5 · p. 18

5.1.4 Agentic distillation

  • Students learn sequential search decisions from complete teacher trajectories, while retrieved tool outputs remain context but are masked from the loss.The 1.5B student can run on-premises without external inference calls; later failures can enter the closed loop as corrective training data.Source: §5.1.4 · p. 17
  • Masked trajectory training reaches 45.8 Pass@1 on CUAD and 55.4 Pass@1 on ECTHR, while unmasked CUAD training has higher recall but lower precision.On CUAD, unmasked training has 58.2 recall versus 52.6 and 39.4 precision versus 48.5; masked training beats it by 4.8 Pass@1 points.Source: §5.1.4 · pp. 17–18, Table 11 · p. 19

5.2 Beyond legal: a general pattern

  • The paper presents the pipeline, search training, distillation, and self-healing loop as a pattern for other regulated-domain workflows.It describes a common setup of a small on-premises model, multi-turn corpus search, grounded-answer and frugality rewards, and failure-based corrective training.Source: §5.2 · p. 19

6 Discussion and Related Work

The discussion positions AgentTune as a lifecycle layer built on existing optimizers and orchestration tools, with one trajectory schema connecting stages. It explains that design choice and identifies how the system differs from the tools it composes with.

6.1 Related work

  • AgentTune adds a multi-turn rollout loop, reward composition, and trajectory schema around TRL trainers, but inherits TRL’s single-node scaling limits.It does not reimplement optimizers; verl and OpenRLHF are identified as cluster-scale alternatives.Source: §6.1 · pp. 19–20
  • AgentTune treats agent strategies as swappable, makes retrieved-token masking a schema-level property, and supports training strategies through a shared core.It also distinguishes its trajectory-and-lifecycle library from AgentTuning [8], which delivers a dataset and models for generalized agent ability.Source: §6.1 · p. 20
  • DECIDE compiles YAML workflows into LangGraph state graphs, while its closed loop turns classified failures into preference data and gates deployment against held-out successful runs.The gate checks procedural regressions such as loops and tool-argument accuracy alongside end-task accuracy.Source: §6.1 · p. 20

6.2 Feature comparison

  • Table 12 compares documented first-class capabilities rather than ranking systems; AgentTune’s lifecycle axis complements tools such as TRL and LangGraph.Connecting six lifecycle stages pairwise could require up to 15 format converters, while one schema uses six projections; AgentTune is source-available, not OSI open source.Source: §6.2 · pp. 20–21, Table 12 · p. 21

6.3 Design decisions

  • A fixed EventLog contract lets users swap strategies, tools, trainers, and judges instead of adopting a monolithic framework’s components.The library supplies defaults, while defining interfaces that accept user-provided components.Source: §6.3 · p. 21
  • The SFT projection creates segment-aware completion-only loss masks from event kinds, so training, distillation, and retraining share one token-ownership definition.Injected trainer, rollout, judge, retraining, and deployment-gate boundaries allow these components to be supplied as parameters.Source: §6.3 · p. 21

7 Limitations

AgentTune’s limits concern scale, incomplete or constrained training and tooling paths, and the scope of its domain validation. Its reported evaluation consists of internal comparisons rather than results calibrated on public agent benchmarks.

7 Limitations

  • The demonstrations are small contract-learning smoke tests, and AgentTune inherits TRL’s single-node scaling limits rather than targeting cluster-scale RLHF.They make weight changes visible and reproducible, but are not competitive public-leaderboard results.Source: §7 · p. 22
  • The standalone SFT path is not wired end to end, and tool-calling training is unavailable through the CLI because it lacks a --tools flag.Agentic RL and trainer_factory-based distillation are the training paths that run.Source: §7 · p. 22
  • ToolRegistry.get(...) can fail in lean environments without langchain_community, while some built-in tools require credentials or browser binaries and lack repository test coverage.RougeMetric and BleuMetric silently return zeros when their optional dependencies are absent.Source: §7 · p. 22
  • A default reward configuration scores well-formed, malformed, and missing tool calls identically, while several collection, routing, and reward paths have documented defects.The working collection path reads each episode’s in-memory PipelineState because the audit-log extraction methods return empty results.Source: §7 · p. 22
  • The RAG synthesis pipeline and trajectory metrics are validated on English document corpora, leaving multilingual and non-contractual domains unvalidated.Source: §7 · p. 22

Evaluation.

  • Every reported number is an internal comparison; the release has no integration with public tool-agent or broader agentic benchmarks.The paper identifies a public benchmark calibration point as the most useful next measurement.Source: Evaluation · p. 22

8 Conclusion

AgentTune centers its lifecycle design on a shared trajectory schema, with runtime, training, distillation, RAG, workflows, and a recovery path organized around it. The paper reports representative components separately, with the legal case study providing its strongest task-level evidence.

8 Conclusion

  • A shared EventLog schema supports projections between trajectory-producing components and training rows, evaluator inputs, and audit records.The library also provides an agentic runtime, TRL-backed tool-calling RL, distillation, RAG synthesis, YAML workflows, and a designed self-healing path.Source: §8 · p. 22
  • The reported experiments establish representative components separately rather than demonstrating the entire lifecycle in one experiment.Source: §8 · p. 22
  • The library is available under the Lexsi Labs Source Available License v1.1.Source: §8 · p. 22

9 Future Work

Future work includes verifying AgentTune beyond English contractual domains, testing naturally occurring queries, and expanding isolation and training capabilities. The paper also identifies external public-benchmark calibration as a particularly valuable addition.

9 Future Work

  • Multilingual and non-contractual domains, as well as naturally occurring user queries, remain to be tested.Source: §9 · p. 22
  • Tool isolation currently runs at process level, with container and remote-host isolation planned.Source: §9 · p. 22
  • The paper identifies scoring one public tool-use benchmark through create_agentic_trainer as the most valuable single addition for calibrating its internal numbers.Cluster-scale RL is outside AgentTune’s current aim, and the standalone core/sft path is not wired end to end.Source: §9 · p. 23

10 Security and Ethical Considerations

The security discussion identifies risks from unrestricted code execution, prompt injection, feedback-loop poisoning, unredacted audits, and unvalidated decision templates. The worked examples illustrate recovery, workflow, GRPO, and synthesis components through condensed repository walkthroughs.

10 Security and Ethical Considerations

  • RunBashTool and RunPythonTool execute model-generated code with the training process’s credentials, network access, and filesystem; DictToolHarness has no sandbox.IsolatedTool provides process isolation only and can silently fall back to in-process execution unless require_isolation=True.Source: §10 · p. 23
  • Retrieved text reaches the policy verbatim, and an attacker-influenced retraining buffer can pass an accuracy gate even when its held-out set is poisoned.Source: §10 · p. 23
  • Audit records are unredacted, and the BFSI KYC example carries personal details into both the audit trail and retraining buffer.The synthesis pipeline and trajectory metrics are validated only on English contractual text.Source: §10 · p. 23
  • The domain workflows are demonstrations rather than certified decision systems, and the automatic accuracy gate does not require human approval before deployment.Source: §10 · p. 23

A.1 Self-healing a project’s own trajectories

  • The worked examples condense repository walkthroughs; one recovery example detects failures in collected project trajectories, then classifies, generates corrections, and gates retraining.It can run in Python without a DECIDE audit log.Source: A Worked examples · p. 28, §A.1 · p. 28

A.2 A DECIDE closed-loop daemon

  • The DECIDE closed-loop daemon watches an audit log and runs detection, classification, retraining, gating, and deployment with injected model-boundary callables.Source: §A.2 · p. 29

A.3 Multi-agent GRPO with a judge panel

  • The GRPO example composes two routed rollout policies and two judges with distinct rubrics into a rollout-and-reward pair.Source: §A.3 · p. 29

A.4 Zero-dependency synthesis run

  • The six-stage synthesis pipeline runs on CPU with injected fakes, which can be replaced by GroqLLMClient and BGEM3Embedder for production.The run writes staged CSVs, a GRPO dataset, metrics, and a manifest.Source: §A.4 · p. 30

A.5 Training a document-search agent

The listing wires a lexical search tool into a GRPO training run, where the model can call the tool during rollouts.

A.5 Training a document-search agent

  • Listing 9 presents this setup as an example of wiring retrieval into a tool-calling training run.Source: §A.5 · p. 30
  • A GRPO run on Qwen3-0.6B uses a search tool and grounding and correctness rewards during training.The SearchCorpusTool is passed to the trainer, so the model can call it mid-rollout.Source: §A.5 · p. 30
  • The search tool uses a SQLite full-text backend indexed with document chunks and returns up to three results.Source: §A.5 · p. 30

A.6 Collecting training data from a live pipeline

These examples show how DECIDE runs can become training data, tools can be authored and evaluated, and EventLog records can support training, harness checks, and configurable workflows.

A.6 Collecting training data from a live pipeline

  • CollectRunner converts DECIDE episode state into DPO-, BCO-, or GRPO-shaped training records.The example reads in-memory episode state rather than the audit log.Source: §A.6 · p. 30

A.7 Authoring a tool

  • A custom tool needs an execute method, while its parameter schema supplies the model-facing interface.The tools argument also accepts plain annotated callables.Source: §A.7 · p. 31

A.8 Scoring a policy without a rollout

  • RLEvaluator can score a policy on a static preference dataset without running a rollout or tool loop.DPO metrics can also be computed from precomputed chosen/rejected score pairs.Source: §A.8 · p. 31

B EventLog event kinds

  • The EventLog defines eight event kinds, with REWARD as the only kind with a meaningful scope.Source: Table 13 · p. 31

C Reward-function catalog

D EventLog projections (source)

E Building and checking a harness

F A DECIDE environment config (excerpt)

H Detailed experimental configuration

This material identifies the legal-search configuration, describes optional AuditKIT integration, and summarizes EventLog and related lifecycle components.

H Detailed experimental configuration

I AuditKIT integration

  • AuditKIT can provide a separate evaluation and experiment-management layer alongside AgentTune without coupling evaluation to its training core.Source: I AuditKIT integration · p. 34

Eventlog

  • EventLog provides a two-tier trajectory schema shared across lifecycle stages, with the light tier observational and the full tier carrying token spans and logprobs for training.The described lifecycle also includes a shared RL core, DECIDE workflows, harnesses, rollout engines, and a gated self-healing loop.Source: Eventlog · p. 35
Loading 2610.13335v1…