Agent Lightning v1.0: Towards Harnessed Agentic RL

1 Introduction

• In harnessed agentic RL, the deployment harness constructs each model prompt and owns context, tools, control flow, and interaction, while training observes request–response pairs.This preserves deployment-time context policy, tool protocols, and execution semantics without reimplementing the agent loop in the RL framework.Source: §1 · p. 2
• Retokenization can make a prior response’s token IDs differ from those in a later prompt, breaking token continuity and preventing safe merging of consecutive calls.This can happen even when the text is unchanged.Source: §1 · p. 2
• A harness rollout can produce a dynamic number of training samples through retokenization, subagent spawning, or context summarization, complicating reward and advantage assignment.Traditional agentic RL instead maps each rollout to a unique training sample.Source: §1 · p. 3
• Dynamic samples make loss normalization nontrivial; sample-level normalization can overweight rollouts producing more samples and may destabilize training.A rollout may map to multiple training samples, so batch sample counts vary.Source: §1 · p. 3
• Backend scheduling must partition the sample set, known only after harness execution, into training steps and mini-batches while balancing work across fixed GPU workers.The number and parallel configuration of training GPUs remain fixed.Source: §1 · p. 3
• Agent Lightning v1.0 systematically characterizes harnessed-RL challenges in an approximately 3,500-line framework that serves as a practical testbed for studying them.The framework supports RL training with arbitrary agent harnesses and embeds design choices for the identified challenges.Source: §1 · p. 3
• 14.6% absolute gain on SWE-bench Verified: RL improves Qwen3.5-9B from 41.8% to 56.4% using only 6K training samples and modest compute.The authors release the complete workflow and training scripts for reproducibility.Source: §1 · p. 3

2.1 Retokenization and Sample Merging

• Because text-level prefix continuity can fail after retokenization or response transformation, Agent Lightning merges calls only when exact token-prefix matches preserve the prompts actually consumed.Template rendering, decode–retokenize drift, and inference-time response processing can break token continuity even when the underlying text is unchanged; best-effort merging preserves correctness but may reduce prefix reuse.Source: Figure 3 · p. 4, §2.1 · pp. 5–6

2.2 Advantage Calculation

• Only 36% of coding-agent rollouts remain a single training sample on average, yielding 2.4 samples per rollout; the authors favor rollout-level advantages so incidental splits do not change group baselines.They note that subagent spawning and context summarization can also split rollouts, and that better credit assignment among a rollout’s samples remains future work.Source: §2.2 · pp. 6–7

2.3 Loss Normalization

• Because sample counts can reflect incidental splitting, the authors prefer rollout-level token-mean loss; sample-level sequence means can overweight rollouts with more samples, while token means can become unstable with many long negative samples.The rollout-level token mean averages each rollout’s pooled response-token loss uniformly across rollouts.Source: §2.3 · p. 8

2.4 Training Backend Complexity

• Since workload size is known only after sample construction, the backend must map variable workloads to fixed workers while preserving rollout and prompt-group provenance and keeping each rollout within one optimizer update.Splitting a rollout across updates can expose its samples to different policy versions, creating within-rollout policy skew.Source: §2.4 · pp. 8–9

3 System Design

• The API Gateway stores rollout state and events, the Rollout Controller launches executions, and the Customized Trainer registers rollouts and assembles training samples from recorded events.The controller can run agent tasks as Kubernetes Jobs or local processes; harnesses connect through the LLM endpoint proxy, allowing training and execution resources to be provisioned independently.Source: §3 · p. 9
• Agent Lightning v1.0 uses approximately 3,500 lines of code and incorporates rollout-level reward and advantage calculation and rollout-level loss normalization.The framework presents itself as lightweight, with each component having a clear responsibility.Source: §3 · p. 9

3.1 Collocated Async RL

• Collocated async RL begins updates after enough rollout data is collected, while the API Gateway drains current requests and pauses new ones until rollout resumes.The phase switch is invisible to the agent harness.Source: §3.1 · p. 10, Figure 6 · p. 10
• Roughly 2x end-to-end speedup over synchronous RL is achieved by collocated async RL while using fewer GPUs.Rollout and training share the same GPU pool, avoiding the wait for the slowest rollout without requiring separate machine pools.Source: §3.1 · p. 10

3.2 Network Issues

• Idempotent API Gateway endpoints support safe retries, while training-sample assembly keeps only the most recent model request for each identical prompt.Earlier matching requests are discarded as retried or superseded calls.Source: §3.2 · p. 10

3.3 Kubernetes Integration

• The Rollout Controller schedules each agent execution as a standard Kubernetes Job, enabling self-hosted or on-premise compute without a commercial sandbox provider.This also keeps the entire training stack open-source.Source: §3.3 · p. 11

3.4 Monitoring

• The Customized Trainer records training and validation rollouts alongside Kubernetes pod logs, enabling AI agents to identify rollout problems such as reward hacking.The paper reports finding reward-hacking examples with this monitoring system.Source: §3.4 · p. 11

4 Experiments

• The experiments cover search, general instruction-following, and coding agents, using Search-R1 and LLM-in-Sandbox setups and SWE-smith-derived coding data.The coding agent uses mini-SWE-agent as its harness and Qwen3.5-9B as its policy model.Source: §4 · p. 11, §4.3 · p. 12

4.1 Search Agent

• Search-agent validation reward rises from 25.1% to 41.7%, an absolute 16.6% gain.Mean training reward also increases steadily.Source: §4.1 · p. 11, Figure 7 · p. 12

4.2 General Instruction-Following Agent

• General instruction-following validation reward improves from 51.9% to 70.2%, an absolute 18.3% improvement.Batch-level training reward is noisy, while validation reward trends upward.Source: §4.2 · pp. 11–12, Figure 8 · p. 12

4.3.3 Training Dynamics

• With rollout-level advantage and normalization, validation reward reaches 38.2% at step 128, versus 35.0% for the baseline and 33.1% with only rollout-level advantage.The combined variant’s policy entropy grows more slowly and remains more stable than with only the rollout-advantage fix; its rollouts yield 2.41 training samples on average, and 36% remain a single fully merged row.Source: Figure 9 · p. 13, §4.3.3 · p. 14

4.3.1 Dataset Preprocessing and Filtering

• For coding-agent training, filtering SWE-smith tasks and applying a model-based difficulty filter yields approximately 6,000 training examples and 400 test examples.The difficulty filter removes tasks solved in all four Qwen3.5-9B rollouts, retains mixed-outcome tasks, and adds 1,000 tasks that fail in all four rollouts.Source: §4.3.1 · p. 13

4.3.2 Preventing Reward Hacking

• The coding-agent safeguards disable Git commands and hide .git, while a Kubernetes network policy blocks general outbound access except to whitelisted services.These measures address reward-hacking behaviors that obtain reference source code instead of solving the task from the problem statement and local information.Source: §4.3.2 · p. 13

4.3.3 Training Dynamics

• On SWE-bench Verified, the coding-agent checkpoint improves from 41.8% to 56.4% at step 208.The checkpoint uses Rollout-level Advantage + Rollout-level Norm.Source: §4.3.3 · p. 14

5 Related Work

• Traditional RL frameworks such as verl [8], AReaL [9], and slime placed the agent loop inside the training framework, requiring independently maintained harnesses to be reimplemented there.Examples of such harnesses include mini-SWE-agent, OpenHands, OpenCode, Claude Code, Codex, OpenClaw, and Hermes.Source: §5 · p. 14
• Agent Lightning [11] introduced a disaggregated LLM-endpoint architecture later adopted by verl Uni-Agent, AReaL 2.0 [13], slime v0.3.0, and Polar [14].These frameworks make different, sometimes conflicting, choices for retokenization, advantage calculation, and loss normalization with dynamic training samples per rollout, and commonly rely on commercial sandbox services.Source: §5 · p. 14
• Agent Lightning v1.0 runs on a self-hosted Kubernetes cluster and uses approximately 3,500 lines of code as a compact, transparent testbed.It is validated on the search-agent, instruction-following, and coding-agent settings of Search-R1 [19], LLM-in-Sandbox [20], and SWE-smith [21], respectively.Source: §5 · p. 14

6 Conclusion

• Harnessed agentic RL places the environment interaction loop in the deploy-time agent harness rather than the training engine, creating challenges in retokenization, advantage calculation, loss normalization, and backend scheduling.Source: §6 · pp. 14–15
• Qwen3.5-9B improves on SWE-bench Verified from 41.8% to 56.4%, a gain of 14.6 percentage points, using about 6K training examples.The released data pipeline and reward-hacking safeguards support this RL-only improvement.Source: §6 · p. 15
• Agent Lightning v1.0 is an approximately 3,500-line framework for arbitrary agent harnesses that embeds rollout-level design choices and releases a complete data pipeline, codebase, and scripts for reproducible research.It is validated on search, instruction-following, and coding agents.Source: §6 · p. 15

• The appendix describes the API Gateway, Rollout Controller, and Customized Trainer, the three components introduced in Section 3.Source: A Detailed System Design · p. 19

A.1 API Gateway

• The lightweight, stateful API Gateway stores rollouts, models, and events and exposes them through a minimal API.Figure 11 summarizes these objects and their relationships.Source: §A.1 · p. 19
• A rollout is one agent execution with a unique ID, input, metadata, and status; GRPO [17] can generate multiple independent rollouts from one training example.Rollout status progresses through queuing and running to succeeded or failed.Source: §A.1 · p. 19
• The trainer registers named LLM endpoints with the gateway, which routes harness requests to the corresponding inference server.The proxy forwards calls to registered endpoints, and each call’s prompt and response token IDs and log probabilities are recorded for training.Source: §A.1 · p. 19
• By default, events record each LLM interaction as a model_request with prompt and response token IDs and log probabilities, plus a scalar reward typically reported at rollout end.Users can also define custom event types.Source: §A.1 · p. 19
• The gateway decouples RL training from agent execution: harnesses connect by directing their LLM endpoint to the proxy, while training and execution resources can run independently and in different locations.The trainer creates rollouts and collects trajectories; the proxy embeds the rollout ID in each request so calls are attributed automatically.Source: §A.1 · pp. 19–20

A.2 Rollout Controller

• The Rollout Controller fetches active rollouts from the API Gateway, launches and monitors agent tasks, and reports their status through the Gateway API.Its primary backend is the K8s Reconciler; a Local Reconciler is also available for debugging.Source: §A.2 · p. 20
• The K8s Reconciler creates Kubernetes Jobs from user-provided templates and synchronizes their status using API watches plus periodic recovery listings.It creates a Job for each queued rollout that lacks one; the watch propagates terminal states with low latency, while periodic listings recover missed events.Source: §A.2 · p. 20
• The Local Reconciler runs agents in a local process pool and uses periodic polling because it owns the process handles directly.It does not need a separate watch mechanism.Source: §A.2 · p. 20
• The API Gateway’s rollout status is ground truth, and synchronization with Kubernetes guarantees only best-effort eventual consistency.Kubernetes execution state may lag during network failures or delayed updates; the K8s Reconciler retries synchronization until the two sides converge after communication resumes.Source: §A.2 · p. 20

Customized Trainer

• At each training step, the VERL-based trainer registers the batch’s rollouts, waits for terminal states, then retrieves model-request and reward events to assemble training samples.This connects the training backend to the API Gateway.Source: §A.3 · p. 21

Dedicated Sample Adapter

• The sample adapter merges consecutive requests only when the later prompt is an exact token-level prefix match of the earlier request and response.The API Gateway maintains no server-side request buffer, keeping training consistent with deployment.Source: §A.3 · p. 21
• The adapter computes baselines and advantages at the rollout level.The paper describes rollout-level calculation as the more principled choice.Source: §A.3 · p. 21
• The adapter uses rollout-level token-mean loss normalization so every rollout has equal weight regardless of its sample count.This implements the rollout-level token-mean loss discussed in Section 2.Source: §A.3 · p. 21

Trajectory Monitoring

• Trajectory monitoring exposes rollout inputs, status, requests, rewards, token and turn statistics, and custom events to help diagnose reward hacking, runaway trajectories, and silent failures.Execution logs are kept in Kubernetes, and rollouts can be inspected manually or with AI agents.Source: §A.3 · p. 21