Source-linked AI summary
Agent Lightning v1.0: Towards Harnessed Agentic RL
Zhiyuan He, Siwei Zhang, Zhiwen Zhou, Yuqing Yang, Yu Kang, Yuge Zhang, Luna K. Qiu, Tin Yan Tsui, Jiahang Xu, Chong Luo
TL;DR
Harnessed agentic RL leaves training systems to reconstruct token-level samples and assign rewards from interactions controlled by deployment-time harnesses. Agent Lightning v1.0 systematically characterizes these challenges and provides a lightweight framework; RL improves Qwen3.5-9B on SWE-bench Verified from 41.8% to 56.4%, a 14.6-point gain.
Problem
Harnessed agentic RL makes sample assembly and advantage assignment difficult because the deployment harness controls interactions while training observes only model calls.
Method
Agent Lightning v1.0 systematically characterizes these challenges through a lightweight framework supporting arbitrary agent harnesses and embedding design choices for training.
Results
14.6 percentage points: RL improves Qwen3.5-9B on SWE-bench Verified from 41.8% to 56.4% using about 6K training examples.
Takeaways & Limitations
The released framework, data pipeline, and scripts support reproducible research on harnessed agentic RL across instruction-following, search, and coding agents.
Takeaways & Limitations
Merging methods involve trade-offs: changing consumed prompts can introduce off-policy stitching, while correctness-preserving approaches may retain redundant computation or require substantially more complex backends.
Abstract
from arXiv · showhide
Modern agents operate inside agent harnesses that manage tools, context, and control flow, making the harness a critical part of the agent system. Our original Agent Lightning introduced a disaggregated architecture that connects arbitrary agents to RL training through an LLM endpoint proxy, an approach later adopted by frameworks such as verl Uni-Agent, AReaL 2.0, slime, and Polar. We refer to this paradigm as harnessed agentic RL, where the deploy-time harness directly participates in model post-training. Harnessed agentic RL differs fundamentally from traditional agentic RL: the harness, rather than the training engine, owns the environment interaction loop, while the trainer observes only sequences of LLM request-response pairs. This introduces challenges in retokenization, sample merging, advantage calculation, loss normalization, and backend scheduling, which can substantially affect training stability and effectiveness. We present Agent Lightning v1.0, a lightweight framework for harnessed agentic RL implemented in approximately 3,500 lines of code. It supports arbitrary agent harnesses and serves as a practical testbed for studying these challenges. We evaluate it on instruction-following, search, and coding agents, and provide a complete reproducible pipeline for coding-agent RL. Using only 6K training examples and modest compute, RL improves Qwen3.5-9B on SWE-bench Verified from 41.8% to 56.4%, a 14.6-point absolute gain. We release the complete workflow and training scripts to facilitate reproducible research on harnessed agentic RL.
Customized Trainer
This section presents the overall framework of Agent Lightning v1.0. No further details about a customized trainer are provided in the supplied passages.
- Figure 1 depicts the overall framework of Agent Lightning v1.0.
1 Introduction
Harnessed agentic RL trains through the deployment-time agent harness, which owns interaction and orchestration while training observes model-call sequences across a service boundary. Agent Lightning v1.0 systematically addresses the resulting training challenges with a lightweight framework supporting arbitrary harnesses.
- Harnessed agentic RL: Harnessed agentic RL keeps the deployment-time harness responsible for context construction, tool execution, and agent–environment interaction while the trainer optimizes resulting model calls.This preserves deployment-time context policies, tool protocols, and execution semantics without requiring the agent loop to be reimplemented in the training system.
- Harnessed agentic RL: Rollouts are exposed as sequences of prompt–response pairs, with intervening harness and environment transitions remaining latent.The harness independently constructs each model-call prompt, and the policy observes only the exact prompt delivered through the LLM API.
- Challenges: Four implementation challenges arise: retokenization and sample merging, advantage calculation, loss normalization, and backend scheduling under dynamic sample counts.These challenges affect token continuity, reward and advantage assignment, optimization weighting, and workload partitioning across fixed GPU workers.
- Challenges: Dynamic sample counts complicate advantages, loss normalization, and scheduling because one rollout may produce multiple samples through retokenization or harness operations such as subagent spawning and context summarization.Sample-level normalization can overweight rollouts producing more samples, while the backend must partition the variable sample set across fixed GPU workers.
- Agent Lightning v1.0: Agent Lightning v1.0 is a complete refactoring implemented in approximately 3,500 lines of code, supporting RL training with arbitrary agent harnesses and serving as a practical testbed.Its training pipeline incorporates design choices for the identified challenges and is evaluated on instruction-following, search, and coding agents.
2 Challenges
Harnessed agentic RL makes the harness responsible for interaction and message state, leaving the trainer with call-level observations rather than a complete rollout. This creates challenges in preserving sampled prompts, merging calls, and keeping optimization invariant to harness-driven sample fragmentation.
- Harnessed agentic RL: The harness owns interaction and state, while the trainer observes only LLM calls, making training-sample assembly a modeling problem.Environment interactions and harness state transitions between calls are not directly visible to the training engine.
- Why Token-Prefix Continuity Breaks: Text-level prompt continuity does not ensure token-level continuity after chat templating and retokenization.The same text can receive different token boundaries, breaking exact token-prefix relations required by RL training.
- Sequence merging: Best-effort merging appends calls only when observed token IDs satisfy exact prefix continuity, preserving rollout prompts while lowering the merge ratio when retokenization drifts.This strategy works with standard dense causal kernels; alternatives trade correctness, reuse, and backend complexity differently.
- Dynamic sample construction: One rollout can produce a variable number of training samples because retokenization, subagent spawning, and context summarization split or replace histories.The resulting sample count is known only after execution and sample construction.
- Optimization across fragmented samples: Rollout-level advantages and token-mean loss normalization avoid letting incidental sample counts change baselines or gradient weighting.Retokenization and harness-internal operations should not alter a rollout’s statistical influence merely by producing more samples.
3 System Design
Agent Lightning v1.0 uses a lightweight control plane to coordinate disaggregated trainers and agent harnesses through declarative rollouts, reconciliation, reliability mechanisms, and flexible execution backends. Its design also supports collocated asynchronous RL, deduplicated requests, self-hosted execution, and rollout-level training choices.
- Control plane: The API Gateway records rollout state and events, while the Rollout Controller reconciles agent executions running as Kubernetes Jobs or local processes.This separates the rollout abstraction from Kubernetes as an execution backend.
- Design principles: Approximately 3,500 lines of code implement clearly separated components with rollout-level reward, advantage calculation, and loss normalization.These choices address challenges identified for harnessed agentic RL.
- Collocated asynchronous RL: Collocated async RL time-shares one GPU pool between rollout and weight updates, pauses requests during switching, and keeps the transition invisible to agent harnesses.The API Gateway stops admitting new requests, completes current ones, and pauses later requests until rollout resumes.
- Collocated asynchronous RL: Roughly a 2x end-to-end speedup over synchronous RL is achieved by collocated async RL while using fewer GPUs and avoiding the slowest-rollout wait.The same GPUs are time-shared between rollout and training rather than split across separate pools.
- Reliability: Repeated model requests with identical prompts are deduplicated during sample assembly, retaining only the most recent call after retries or supersession.This handles retried LLM calls that may produce different responses and therefore cannot be made idempotent.
- Execution infrastructure: Agent Lightning v1.0 runs each agent execution as a standard Kubernetes Job, enabling self-hosted or on-premise compute instead of commercial sandbox services.This avoids recurring commercial sandbox costs and keeps the training stack open-source.
4 Experiments
Agent Lightning v1.0 is evaluated on search, general instruction-following, and coding agents, with detailed emphasis on a reproducible coding-agent pipeline. Across these settings, training improves validation rewards, while coding experiments examine data filtering, reward-hacking safeguards, and rollout-level training choices.
- 4 Experiments: The framework is evaluated in search, general instruction-following, and coding-agent training settings.The search and instruction-following setups follow Search-R1 and LLM-in-Sandbox, while coding data is derived from SWE-smith.
- 4.1 Search Agent: 16.6% absolute gain: search-agent validation reward rises from 25.1% to 41.7%.The search agent uses Llama-3.2-3B-Instruct, GRPO, HotpotQA training data, and exact match as the reward metric.
- 4.2 General Instruction-Following Agent: 18.3% absolute improvement: general instruction-following validation reward increases from 51.9% to 70.2%.The agent uses a computer sandbox, Qwen3-4B-Instruct-2507, RLOO, and an 80%/20% training/evaluation split.
- 4.3 Coding Agent: The coding agent uses Qwen3.5-9B, mini-SWE-agent, and SWE-smith-derived tasks with data filtering and reward-hacking safeguards.The harness interacts with repository environments, executes commands, and produces code changes.
- 4.3.1 Dataset Preprocessing and Filtering: 59,136 tasks from 128 repositories: SWE-smith provides executable software-engineering tasks with problem statements, patches, and tests.Its Docker images occupy 295 GB, compared with 4 TB for R2E-Gym and 6 TB for SWE-Gym.
- 4.3.1 Dataset Preprocessing and Filtering: Approximately 5,000 examples remain after removing empty statements, missing branches, oversized test suites, and tasks solved in all four Qwen3.5-9B rollouts.Retained tasks have both successful and failed model rollouts, providing a model-based difficulty filter.
- 4.3.2 Preventing Reward Hacking: Reward-hacking safeguards disable Git access and restrict outbound networking to whitelisted services.These controls require solving tasks from the problem statement and local information rather than retrieving reference source code.
- 4.3.3 Training Dynamics: 38.2% validation reward: rollout-level advantage plus rollout-level normalization outperforms the 35.0% baseline and 33.1% advantage-only variant at step 128.The full rollout-level variant also produces slower, more stable policy-entropy growth than the advantage-only correction.
5 Related Work
Traditional RL frameworks place the agent’s environment-interaction loop inside the training engine, making independently maintained agent harnesses difficult to reuse. This limitation motivates architectures that can connect existing harnesses to RL training.
- Traditional RL frameworks: Traditional frameworks including verl, AReaL, and slime require agent loops inside the training framework, hindering reuse of mini-SWE-agent, OpenHands, OpenCode, Claude Code, Codex, OpenClaw, and Hermes.These frameworks follow the classic ReAct-style Markov formulation in which the training engine owns environment interaction.
6 Conclusion
The conclusion characterizes harnessed agentic RL, where the deploy-time harness owns environment interaction, and presents Agent Lightning v1.0 as a framework addressing the resulting training challenges. The framework supports arbitrary harnesses, is validated across three agent types, and enables substantial coding-agent improvement with RL alone.
- Harnessed agentic RL places the environment interaction loop in the deploy-time agent harness rather than the training engine.
- The paradigm introduces challenges in retokenization, advantage calculation, loss normalization, and training backend scheduling.
- Agent Lightning v1.0 is an approximately 3,500-line framework supporting arbitrary agent harnesses and embedding rollout-level design choices for these challenges.
- 41.8% to 56.4%: RL alone improves Qwen3.5-9B on SWE-bench Verified, using the released data pipeline and reward-hacking safeguards.
A Detailed System Design
This appendix details Agent Lightning v1.0’s three system components: the API Gateway, Rollout Controller, and Customized Trainer.
- System components: Agent Lightning v1.0 comprises three detailed system components: the API Gateway, Rollout Controller, and Customized Trainer.The appendix describes these components introduced in Section 3.
A.1 API Gateway
The API Gateway is a lightweight, stateful central service that stores rollouts, models, and events while exposing them through a minimal API. Its proxy decouples agent execution from RL training, enabling arbitrary harnesses and independently provisioned resources.
- API Gateway: The API Gateway stores rollouts, models, and events in a single stateful service exposed through a minimal API.It is the central component of Agent Lightning v1.0.
- API Gateway: A rollout represents one agent execution with a unique ID, input, status, and user-defined metadata.Statuses progress through queuing and running to succeeded or failed; multiple rollouts may originate from one training example.
- API Gateway: Models identify registered LLM inference endpoints, which the Gateway uses to route agent-harness requests.The trainer registers each model by name and address.
- API Gateway: Events attach arbitrary data to rollouts, including model requests with token IDs and log probabilities and terminal scalar rewards.Users can also define custom event types.
- API Gateway: The proxy automatically attributes harness LLM calls to rollouts and records their request data for later training export.Harnesses connect by pointing an OpenAI-compatible client to the proxy, while training and execution resources can run independently or in different locations.
A.2 Rollout Controller
The Rollout Controller orchestrates agent execution on Kubernetes by fetching active rollouts, launching and monitoring tasks, and reporting status through the API Gateway. It supports both Kubernetes and local-process backends, with rollout status treated as the ground truth under best-effort eventual consistency.
- Controller overview: The Rollout Controller periodically fetches active rollouts, launches agent tasks, monitors them, and reports status through the API Gateway.Its primary backend is the K8s Reconciler, with a Local Reconciler available for debugging.
- K8s Reconciler: The K8s Reconciler creates Kubernetes Jobs from user-provided templates, watches Job updates, and periodically lists managed Jobs to recover missed events.This follows the standard Kubernetes controller pattern and propagates terminal states with low latency.
- Local Reconciler: The Local Reconciler launches agents in a local process pool and relies on periodic polling because it directly owns process handles.No separate watch mechanism is needed for this backend.
- State Consistency: The API Gateway’s rollout status is the ground truth, while Kubernetes execution state may lag because of network failures or delayed updates.The K8s Reconciler retries synchronization on its next cycle, providing best-effort eventual consistency once communication resumes.
A.3 Customized Trainer
The Customized Trainer connects VERL’s training backend to the API Gateway by registering rollouts, collecting terminal events, and assembling training samples. Its components implement rollout-aware sample processing and expose trajectory data for diagnosing failures.
- Customized Trainer: At each step, the trainer registers current-batch rollouts, waits for termination, retrieves model_request and reward events, and assembles them into training samples.The lightweight trainer is built on VERL and connects the training backend to the API Gateway.
- Dedicated Sample Adapter: The adapter merges consecutive model requests only when the later prompt exactly token-level matches a prefix of the earlier request and response.This preserves a simple API Gateway without a server-side request buffer and keeps training consistent with deployment.
- Dedicated Sample Adapter: The adapter computes baselines and advantages at the rollout level, treating rollouts as the unit for advantage calculation.The paper identifies rollout-level computation as the more principled choice.
- Dedicated Sample Adapter: The rollout-level token-mean loss normalizes training so every rollout carries equal weight regardless of its sample count.This implements the loss-normalization approach discussed in Section 2.
- Trajectory Monitoring: Trajectory monitoring exposes rollout inputs, statuses, model requests, rewards, token and turn statistics, and custom events for diagnosing reward hacking, runaway trajectories, and silent failures.Execution logs are retained in Kubernetes for manual or AI-assisted inspection.