PaperlayerView exact revision on arXiv

Source-linked AI summary

Polar: Agentic RL on Any Harness at Scale

Binfeng Xu, Hao Zhang, Shaokun Zhang, Songyang Han, Mingjie Liu, Jian Hu, Shizhe Diao, Zhenghui Jin, Yunheng Zou, Michael Demoret, Jan Kautz, Yi Dong

arXiv:2605.24220v1cs.DC

TL;DR

Agentic RL harnesses can be difficult to integrate as training environments, especially when they are complex or opaque. Polar proxies their model calls and reconstructs token-faithful trajectories through an asynchronous rollout service. It improves the same 4B base model under all four evaluated harnesses, with a 22.6-point absolute pass@1 gain under Codex.

  • Problem

    Agentic RL targets complex, heterogeneous harnesses, but integrating them can require framework-specific adaptation or prescribed interfaces that are difficult for opaque systems.

    Source: §1 · p. 2

  • Method

    Polar proxies an unchanged harness’s model API traffic, reconstructs token-faithful trajectories, and separates rollout stages behind asynchronous service boundaries.

    Source: §1 · p. 3, §3.4 · p. 7

  • Results

    Polar improves the same 4B base model under all four evaluated harnesses, with a 22.6-point absolute pass@1 gain under Codex.

    Source: §4.1 · p. 10

  • Takeaways & Limitations

    Polar lets rollout scale independently from training and inference while preserving the behavior of non-standard harnesses.

    Source: §5 · p. 12

    Broadcasting outcome rewards to every per-request trace caused reward hacking associated with noisy credit assignment, and session normalization or PRM-style credit assignment is outside this work’s scope.

    Source: §4.1 · p. 10

Abstract

from arXiv · show

Reinforcement learning for language agents increasingly depends on custom harnesses that manage long-running context, multi-turn tool use and multi-agent orchestration. However, porting these harnesses into RL environment interfaces remains difficult and often loses important training signals. We bridge this gap with polar, a rollout framework for scalable asynchronous RL over arbitrary agent harnesses. Polar treats the agent harness as a black box: it proxies LLM API calls, records token-level model interactions, and reconstructs token-faithful trajectories for training. Each rollout node efficiently manages runtime prewarming, agent execution, trajectory reconstruction, and evaluation in parallel, exposing asynchronous service endpoints that can be consumed by independent trainers at scale. This decoupled design makes Polar agnostic to agent harnesses, training infrastructure, and RL algorithms while improving compute utilization for long-running agent workloads. We validate polar by training agents on software-engineering tasks with popular coding harnesses. Using simple GRPO, polar improves Qwen3.5-4B by 22.6, 4.8, 0.6 and 6.2 points on SWE-Bench Verified with the Codex, Claude Code, Qwen Code and Pi harnesses, respectively. We further demonstrate Polar for offline data generation over custom harnesses and ablate trajectory reconstruction strategies. Polar rewrites its preceding work, Prorl Agent, and has been registered as one of NeMo Gym environments.

1. Introduction

Agentic RL targets complex, heterogeneous harnesses with long-running workflows, making the training target itself a systems challenge. Polar uses model API traffic as a black-box interface to capture token-level interactions and reconstruct trajectories, while separating rollout work behind asynchronous services.

1. Introduction

  • Agentic RL targets complex software systems with heterogeneous environments, external tools, and long-running workflows, sometimes implemented in different languages or as closed-source binaries.Unlike a standardized training target, these systems make the training target itself a central systems challenge.Source: §1 · p. 2
  • Existing integrations can require framework-specific agent adaptation or prescribed interfaces, and may remain difficult or infeasible for increasingly complex or opaque harnesses.Some systems reduce integration burden with tracing and LLM-call capture but still require agents to conform to prescribed interfaces.Source: §1 · p. 2
  • Polar trains agents as black boxes by proxying their LLM calls and turning captured model interactions into trajectories and rewards.The harness runs unchanged; the proxy captures prompts, sampled tokens, log probabilities, and responses.Source: §1 · pp. 2–3
  • Polar separates runtime setup, execution, trajectory reconstruction, evaluation, and trainer callbacks behind asynchronous service boundaries.Its contributions also include token-faithful trajectory reconstruction and validation on coding harnesses, with offline SFT data generation using a custom harness.Source: §1 · p. 3

2. Related Work

Prior work addresses agent RL systems, instrumentation, evaluation, and token fidelity through different integration points. Polar focuses on using the model endpoint to capture and train from existing harnesses while leaving their internal behavior unchanged.

2.1. Agent RL Systems

  • ProRL Agent introduced a service boundary for multi-turn rollouts, while Polar changes the integration contract to launch native harnesses and observe them externally.SkyRL-Agent focuses on efficient training once tasks fit its environment abstractions; Polar targets running pre-existing harnesses unchanged.Source: §2.1 · p. 3
  • PRIME-RL and Slime address policy-optimization and inference scaling, whereas Polar supplies a rollout substrate for asynchronous trainers using heavier harnesses.Polar is not a replacement trainer.Source: §2.1 · pp. 3–4

2.2. Low-Intrusion Agent Instrumentation

  • Agent Lightning and rLLM collect agent execution through tracing or tracked clients, while Polar uses the provider API endpoint as its minimum integration point.The proxy records token-level fields and supports harnesses implemented as command-line programs, package-managed tools, or binaries.Source: §2.2 · p. 4

2.3. SWE Task Evaluation and Benchmark

  • Harbor evaluates agents in containers and converts native logs into trajectories, while Polar proxies model traffic to produce trainer-consumable token IDs, log probabilities, loss masks, and rewards.SWE-bench and SWE-Gym provide software-engineering workloads that combine expensive setup, sparse patch-level rewards, and long-tail execution.Source: §2.3 · p. 4

2.4. Token Fidelity and Retokenization Drift

  • Polar preserves behavior-policy token fidelity because transcript retokenization can produce token IDs different from those used during generation.Generated assistant tokens come from inference responses, interstitial tokens from canonical prompt tokenization, and the loss mask marks only behavior-policy tokens as trainable.Source: §2.4 · p. 4

3. Polar

Polar targets tasks run through existing harnesses, which call a model endpoint while using tools and managing context. Its service separates task coordination from gateway execution, and the gateway proxy observes harness model calls for downstream trajectory construction.

3. Polar

  • Polar targets harness-driven tasks involving tools, file editing, sub-agents, or context management, with execution outcomes evaluated as rewards.The rollout system must preserve native interactions as trainer-ready traces, including prompt context, sampled tokens, loss masks, rewards, and provenance.Source: §3 · pp. 4–5

3.1. Architecture

  • The rollout server expands task requests into independent sessions, while gateway nodes execute sessions, capture model calls, build trajectories, and evaluate outputs.The server persists terminal results and accepts gateway callbacks; gateways host the proxy endpoint used by harnesses.Source: §3.1 · p. 5

3.2. Harness and Proxy Capture

  • Polar routes native harness model calls through a gateway proxy by configuring the harness’s model base URL to point to the gateway.Gateway-level worker pools isolate runtime setup, ready buffering, harness execution, and post-run trajectory and evaluation work; prewarming proceeds off the critical path.Source: Figure 3 · p. 5, §3.2 · p. 6

1. Detect the provider API.

Polar identifies provider API formats from each request’s path and headers.

1. Detect the provider API.

  • Request paths and headers distinguish Anthropic Messages, OpenAI Chat Completions, OpenAI Responses, and Google generateContent-style calls.Source: §1 · p. 6

2. Normalize the request.

A provider transformer normalizes harness requests into the OpenAI Chat Completions format used by local inference servers, adding training fields as needed.

  • The transformer maps request roles, content parts, tool definitions and choices, stop controls, and generation parameters into the OpenAI Chat Completions shape.Source: §2 · p. 6
  • It adds training-specific fields, including logprobs=true.Source: §2 · p. 6
  • The normalized request format is the one consumed by local inference servers.Source: §2 · p. 6

3. Capture token-level data.

The gateway records model-call data needed to reconstruct training samples, including token IDs and inference log probabilities.

  • Each completion record stores request and response messages, prompt and sampled response token IDs, the finish reason, and inference-backend log probabilities.Source: §3 · p. 6
  • The record preserves both the request messages and the response messages for each captured call.Source: §3 · p. 6
  • Prompt token IDs, sampled response token IDs, and backend log probabilities are captured alongside the completion.Source: §3 · p. 6

4. Return the provider shape.

Polar keeps the proxy below the agent framework and separates rollout stages behind asynchronous services. Its trajectory builders can consolidate append-only calls while preserving sampled-token training fidelity.

4. Return the provider shape.

  • The proxy returns the harness-expected response schema; streaming requests use a non-streaming upstream response and a synthetic provider-shaped stream.This captures tokens while retaining compatibility with harnesses expecting server-sent events.Source: §4 · p. 6

3.2.1. Harness Adapter

  • Small harness adapters can configure providers and tools or skills, while common runtime operations let tasks switch between Docker and rootless Apptainer.Source: §3.2.1 · p. 6, §3.2.2 · p. 6

3.3. Asynchronous Rollout Staging

  • Stage-isolated execution separates runtime startup, harness execution, evaluator setup, testing, and teardown so their costs do not block one another.INIT prepares runtimes, READY buffers initialized runtimes, RUNNING executes the harness, and POSTRUN constructs trajectories, evaluates, and tears down.Source: §3.3 · p. 6, §3.3.1 · p. 6

3.3.2. Evaluator prewarm and timeouts.

  • Evaluator runtimes can be prepared during agent execution, and a timed-out harness still enters post-run so captured partial traces retain terminal timeout status.Source: §3.3.2 · p. 6

3.4. Trajectory Reconstruction

  • The trajectory builder converts an ordered completion session into a trajectory, with custom strategies registered alongside per-request and prefix-merging builders.Source: §3.4 · p. 7

3.4.1. Per Request

  • Per-request reconstruction emits one trace per completion, preserving individual calls but potentially fragmenting a coding problem into hundreds of short samples.Source: §3.4.1 · p. 7

3.4.2. Token Faithful Prefix Merging

  • Prefix merging reconstructs longer traces only where conversation histories are append-only, while compaction, subagent, and parallel-branch boundaries form separate chains.Each chain uses sampled assistant tokens as trainable tokens and masks canonical interstitial tokens.Source: §3.4.2 · pp. 7–8, Figure 4 · p. 7
  • The asynchronous pipeline lets the rollout server keep inferencing while the trainer waits for evaluated trajectory groups; Figure 5b reports fewer updates and faster training with prefix merging.Source: Figure 5 · p. 8

4. Experiments

Polar is evaluated for online GRPO across four coding harnesses and for offline generation of supervised fine-tuning trajectories. The experiments report benchmark gains across harnesses, a trajectory-merging efficiency comparison, and a filtered SFT corpus.

4.1. SWE-Gym GRPO on Coding Harnesses

  • 22.6 points on SWE-Bench Verified with Codex, 4.8 with Claude Code, 0.6 with Qwen Code, and 6.2 with Pi are the gains over the Qwen3.5-4B base model after Polar GRPO.The base-to-trained scores are 3.8% to 26.4% for Codex, 29.8% to 34.6% for Claude Code, 34.6% to 35.2% for Qwen Code, and 34.2% to 40.4% for Pi.Source: §4.1 · p. 10, Table 1 · p. 10
  • 5.39× faster over three training steps, prefix_merging reduces trainer updates from 1,185 to 218 and wall-clock time from 189.5 to 35.2 minutes.Average rollout-GPU utilization is 87.7% with prefix_merging versus 20.4% with per_request.Source: §4.1 · p. 10
  • Per-request outcome-reward broadcasting produced significant reward hacking, which the authors attribute to noisy credit assignment without session normalization or an advanced process reward model.Session normalization and PRM-style credit assignment are outside this work’s scope and are on the roadmap.Source: §4.1 · p. 10

4.2. Offline Data Generation

  • A single 8×H100 SGLang serving job hosting Qwen3.5-122B-A10B (TP=8, max_model_len=32,768) drove pi-coding-agent v0.67.68 against 1,638 SWE-Gym instances across seven repositories.Each task ran in its own Apptainer SIF; the per-task timeout was 3,600 seconds.Source: §4.2 · p. 11
  • 504 of 1,638 attempts entered the SFT corpus, a 30.8% acceptance rate, after patches passed all FAIL_TO_PASS tests while keeping PASS_TO_PASS tests green.The run cost roughly 64 GPU-hours; repository acceptance varied, exceeding 50% for getmoto/moto and falling below 20% for data-frame and dataflow workloads.Source: §4.2 · p. 11, Table 2 · p. 11
  • The released corpus contains long multi-turn trajectories, averaging 104 messages and 51 assistant turns per session, with a long tail above 200 turns.It uses an Apache-2.0 license and a repository-stratified 90/10 train/test split.Source: §4.2 · p. 11

5. Conclusion

Polar moves the integration boundary to the model endpoint: the harness runs normally while a proxy observes token-level traffic and the rollout service reconstructs trajectories and rewards. The design separates rollout scaling from training and inference while preserving non-standard harness behavior.

5. Conclusion

  • Polar treats test-time environments as a first-class part of RL, rather than requiring them to be ported into the trainer.Its endpoint proxy observes token-level model traffic, and the rollout service converts completed executions into trainable trajectories and rewards.Source: §5 · p. 12

A.1. Framework Comparison

This appendix compares rollout-system design criteria and presents representative Polar configurations, traces, and service endpoints. The comparison distinguishes first-class support from partial or absent support, while the examples show how task submission and trainer-facing data are organized.

A.1. Framework Comparison

  • The framework comparison evaluates async RL support, async rollout staging, rollout-as-a-service, and native-harness agnosticism, marking first-class, partial or planned, and absent primary support.The criteria distinguish continued rollout generation with staleness handling, independently scheduled execution stages, a durable task API, and harness use without reimplementation.Source: Table 3 · p. 15
  • The comparison characterizes SkyRL, Agent Lightning, rLLM, and OpenClaw-RL by their respective environment, workflow-observability, gateway, and agent-recipe boundaries relative to Polar’s opaque native-harness execution.The cited comparison notes partial support for several adjacent mechanisms rather than treating them as absent.Source: §A.1 · p. 15

A.2. SWE-Gym GRPO Hyperparameters

  • The SWE-Gym GRPO hyperparameter table reports policy-optimization and rollout parameters from the example configuration, but omits cluster topology and worker placement.The payload and trace examples are distinct from this table’s hyperparameter scope.Source: Table 4 · p. 15

A.3. Representative Task Payload

  • Polar’s representative task payload specifies a runtime, Codex harness, prefix_merging builder, SWE-Bench evaluator, and patch-submission configuration.The evaluator can refresh the runtime and invokes the configured patch command.Source: §A.3 · p. 16

A.4. Representative Trace

  • The trainer-facing trace example includes token IDs and log probabilities, prompt and response messages, reward, and session and harness provenance.The shown example records the builder as prefix_merging and the harness as codex.Source: §A.4 · pp. 16–17

A.5. Service API Summary

  • The service API supports non-blocking task submission, polling task status and results, status inspection, session callbacks, and gateway registration and heartbeats.A separate gateway surface supports session creation, status, deletion, and provider-style model requests.Source: §A.5 · p. 17

Related papers

Loading 2605.24220v1…