Source-linked AI summary
ArenaRL: Scaling RL for Open-Ended Agents via Tournament-based Relative Ranking
Qiang Zhang, Boli Chen, Fanrui Zhang, Ruixue Ding, Shihang Wang, Qiuchen Wang, Yinfeng Huang, Haonan Zhang, Rongxiang Zhu, Pengyong Wang, Ailin Ren, Xin Li, Pengjun Xie, Jiawei Liu, Ning Guo, Jingren Zhou, Zheng-Jun Zha
TL;DR
Open-ended agent tasks lack objective ground truth, making noisy pointwise reward scores inadequate for distinguishing strong trajectories. ArenaRL uses process-aware pairwise ranking and tournament-based optimization, with strong results across agent and writing benchmarks while retaining setting-specific limitations.
Problem
Open-ended agent tasks lack objective ground truth and involve subjective, multi-dimensional quality judgments, limiting conventional RL reward evaluation.
Method
ArenaRL replaces pointwise scalar scoring with process-aware intra-group pairwise ranking and tournament-based advantage estimation.
Results
ArenaRL substantially outperforms standard RL baselines across travel planning, deep research, and open-ended writing tasks.
Takeaways & Limitations
Seeded single-elimination ranking preserves advantage-estimation accuracy with O(N) complexity, offering an efficient alternative to O(N^2) exhaustive comparison.
Takeaways & Limitations
Anchor-based ranking has O(N) complexity but can miss subtle differences between exploratory samples, creating ambiguity among suboptimal solutions.
Abstract
from arXiv · showhide
Reinforcement learning has substantially improved the performance of LLM agents on tasks with verifiable outcomes, but it still struggles on open-ended agent tasks with vast solution spaces (e.g., complex travel planning). Due to the absence of objective ground-truth for these tasks, current RL algorithms largely rely on reward models that assign scalar scores to individual responses. We contend that such pointwise scoring suffers from an inherent discrimination collapse: the reward model struggles to distinguish subtle advantages among different trajectories, resulting in scores within a group being compressed into a narrow range. Consequently, the effective reward signal becomes dominated by noise from the reward model, leading to optimization stagnation. To address this, we propose ArenaRL, a reinforcement learning paradigm that shifts from pointwise scalar scoring to intra-group relative ranking. ArenaRL introduces a process-aware pairwise evaluation mechanism, employing multi-level rubrics to assign fine-grained relative scores to trajectories. Additionally, we construct an intra-group adversarial arena and devise a tournament-based ranking scheme to obtain stable advantage signals. Empirical results confirm that the built seeded single-elimination scheme achieves nearly equivalent advantage estimation accuracy to full pairwise comparisons with O(N^2) complexity, while operating with only O(N) complexity, striking an optimal balance between efficiency and precision. Furthermore, to address the lack of full-cycle benchmarks for open-ended agents, we build Open-Travel and Open-DeepResearch, two high-quality benchmarks featuring a comprehensive pipeline covering SFT, RL training, and multi-dimensional evaluation. Extensive experiments show that ArenaRL substantially outperforms standard RL baselines, enabling LLM agents to generate more robust solutions for complex real-world tasks.
1 Introduction
Open-ended agent tasks lack objective ground truth, making pointwise reward modeling vulnerable to discriminative collapse. ArenaRL addresses this with process-aware relative ranking, efficient tournaments, and full-cycle benchmarks, outperforming standard RL baselines across diverse tasks.
- Open-ended tasks lack objective ground truth because correctness is subjective and multi-dimensional.
- Pointwise scalar rewards can collapse subtle trajectory differences into unstable signals, hindering optimization.
- ArenaRL replaces pointwise scoring with process-aware intra-group relative ranking that evaluates outcomes, reasoning coherence, and tool use.
- ArenaRL substantially outperforms SFT, GRPO, and GSPO across travel planning, deep research, and open-ended writing benchmarks.
- Seeded single-elimination tournaments achieve high-accuracy advantage estimation with O(N) complexity, reducing the cost of exhaustive pairwise comparison.
- Open-Travel and Open-DeepResearch provide full SFT →RL training →multi-dimensional evaluation pipelines for open-ended agents.
2 Related Work
Prior agent benchmarks and RL methods largely target deterministic tasks with verifiable outcomes. Open-ended agents instead require evaluation and training infrastructure for subjective, multi-dimensional objectives.
- Existing agent benchmarks commonly test deterministic goals such as reaching target pages or producing correct code.
- Open-ended tasks lack a single gold solution and involve trade-offs among reasoning soundness, constraint satisfaction, and practical usefulness.
- Existing open-ended benchmarks are predominantly static test suites that support post-hoc evaluation rather than systematic training–evaluation pipelines.
- RL methods such as GRPO and DAPO succeed with clear ground truth but remain challenging to extend to open-ended tasks without objective outcomes.
3 Preliminary
ArenaRL formulates open-ended agent behavior as trajectory generation and replaces noisy pointwise evaluation with process-aware pairwise comparison. Tournament ranking then produces relative signals for policy optimization.
- 3.1 Task Definition: An agent trajectory interleaves chain-of-thought, tool invocations, environmental feedback, and a final answer.
- 3.1 Task Definition: The RL objective aligns behavior with reward while regularizing divergence from a reference policy through KL control.
- 3.2 Pointwise Scalar Evaluation: Pointwise reward modeling treats observed scores as true trajectory utility corrupted by noise, creating discriminative collapse among high-quality responses.
- 3.2 Pointwise Scalar Evaluation: As group variance vanishes, score normalization amplifies judge noise into gradients that can cause stagnation or degeneration.
- 3.3 Process-Aware Pairwise Evaluation: ArenaRL jointly evaluates trajectory pairs with a process-aware rubric covering reasoning consistency, tool-call precision, and final-answer reliability.
- 3.3 Process-Aware Pairwise Evaluation: Bidirectional evaluation swaps trajectory order to eliminate positional preference in pairwise scoring.
- 3.4 Tournament-Based Ranking: ArenaRL uses intra-group relative ranking and investigates five tournament topologies to balance computational cost with ranking fidelity.
4 Methodology: ArenaRL
ArenaRL ranks trajectories relative to one another through tournament-based, process-aware comparisons, then converts those rankings into normalized advantages for policy optimization. Its seeded single-elimination design targets the efficiency–resolution trade-off while alternative tournament topologies provide different computational costs and ranking procedures.
- ArenaRL framework: ArenaRL constructs a dynamic arena that replaces pointwise rewards with relative quality rankings among trajectories sampled from the current policy.The framework systematically investigates five tournament topologies for deriving robust advantage signals.
- Tournament topologies: Round-robin ranking compares every trajectory with all other N −1 trajectories, but its O(N^2) complexity makes it impractical for large online-training groups.The paper uses round-robin as a gold standard for evaluating the fidelity of more efficient topologies.
- Anchor-Based Ranking: Anchor-based ranking compares each exploratory trajectory individually with a deterministic reference trajectory, using high-entropy sampling for exploration diversity.The anchor is generated with greedy decoding, while the remaining trajectories are sampled at higher entropy.
- Anchor-Based Ranking: Anchor-based ranking has O(N) complexity but cannot resolve subtle differences between exploratory samples, creating ambiguity among suboptimal solutions.It measures how samples compare with the anchor rather than directly comparing exploratory samples with one another.
- Seeded Single-Elimination: Seeded Single-Elimination uses preliminary anchor-based scores to assign seeds, reducing premature collisions between high-quality trajectories before bracket elimination.The elimination phase arranges pairings by seed rankings, with winners advancing and losers eliminated.
- Seeded Single-Elimination: Seeded Single-Elimination preserves O(N) complexity through N −1 seeding comparisons and N −1 tournament comparisons, ranking primarily by survival depth.Trajectories eliminated in the same round are further ordered by accumulated average scores from previous matches.
- Alternative topologies: Double-elimination removes trajectories after two defeats, while Swiss uses fixed-round dynamic pairings and ranks with total wins plus Buchholz scores at O(N log N).These topologies offer alternatives to single elimination with different loss-handling and pairing structures.
- Ranking-Based Policy Optimization: ArenaRL converts tournament ranks into normalized advantage signals before optimizing the policy, producing increasingly strong reasoning and planning behaviors on open-ended tasks.Ranks use 0 for the highest-ranked trajectory; rank-based rewards are standardized within each group before policy optimization.
5 Benchmarking Open-Ended Agency
Open-Travel and Open-DeepResearch are full-cycle benchmarks for open-ended agentic tasks, built through staged data construction, training-data preparation, quality control, and multidimensional evaluation.
- The benchmarks target realistic business scenarios involving long-horizon planning, tool coordination, autonomous information retrieval, and report generation.
- Stage I: Benchmark Data Collection: Stage I collects expert-refined queries and complete tool-use trajectories from high-performing closed-source models as reference baselines.The collection process selects 50 queries for each of five Open-Travel subtasks and 100 Open-DeepResearch queries.
- Stage II: Large-Scale Training Data Construction: Stage II expands representative seed queries with multiple LLM query generators, using trajectory data for SFT while reserving other queries for RL.
- Stage III: Trajectory Data Quality Control: Stage III applies rule-augmented LLM inspection to assess tool-use effectiveness, conversational correctness, and final-answer consistency, rewriting failed trajectories.
- The released datasets contain 2,600 SFT and 1,626 RL samples for Open-Travel, and 2,662 SFT and 2,216 RL samples for Open-DeepResearch.
- Evaluation: Evaluation uses two independent closed-source LLM judges for pairwise trajectory-and-answer comparisons, reporting criterion-specific and averaged win rates.DeepResearch additionally reports valid generation rate because long contexts can cause context overflow and invalid final answers.
6 Experiments
Experiments select Seeded Single-Elimination as ArenaRL’s tournament topology and compare it with RL and closed-source baselines across agent and writing benchmarks. ArenaRL achieves stronger benchmark performance, while group-size scaling, evaluator agreement, and training curves support the method’s robustness.
- Tournament Topology Analysis: Seeded Single-Elimination reaches a 32.5% average win rate versus 32.9% for Round-Robin while using O(N) pairwise comparisons.The comparison uses group size N = 8 and number of groups K = 8.
- Main Results: ArenaRL achieves a 41.8% average win rate on Open-Travel, compared with 16.4% for GRPO and 17.2% for GSPO.
- Main Results: ArenaRL reaches a 64.3% win rate and 99% valid generation rate on Open-DeepResearch, while the SFT baseline has a 32% valid generation rate.
- Main Results: Across three open-ended writing benchmarks, ArenaRL outperforms GRPO by 6.70% and GSPO by 7.27% in overall average score.
- Ablations: ArenaRL performance improves monotonically with group size, and even N = 2 achieves a 20.8% average win rate versus 16.4% for SFT.
- Evaluation and Training Dynamics: LLM and human evaluations show 73.9% overall agreement, while direct RL training rises from score 0 at step 0 to 71% at step 160.
- Case Studies: On deterministic POI search, the ArenaRL-tuned model improves search accuracy by 75% to 83% over baseline.
7 Conclusion
ArenaRL shifts open-ended-agent reinforcement learning from pointwise scalar scoring to intra-group relative ranking. Its seeded single-elimination topology balances advantage-estimation accuracy with computational efficiency.
- ArenaRL shifts open-ended-agent reinforcement learning from pointwise scalar scoring to intra-group relative ranking.
- The framework systematically investigates five tournament topologies for relative advantage estimation.
- Seeded single-elimination strikes a favorable balance between advantage-estimation accuracy and computational efficiency.
- Evaluations across travel planning, deep research, and open-ended writing find robust advantage signals and incentivized efficient reasoning and planning.
A Implementation Details.
The implementation uses a TRL- and DeepSpeed-based cold-start phase, followed by ArenaRL reinforcement learning with task-specific group configurations. Evaluation compares candidate outputs against baseline outputs using Qwen3-Max and Claude-4-Sonnet.
- Cold-start phase: Cold-start training uses TRL with DeepSpeed ZeRO-3 for 3 epochs on 32 × NVIDIA H20 GPUs.The learning rate is 2 × 10−5.
- Cold-start phase: The cold-start batch size is 1 per GPU.
- RL phase: ArenaRL masks environmental-feedback tokens from the loss to focus optimization on reasoning quality.
- RL phase: RL uses group size N = 16 and K = 8 for Open-Travel and writing, while Open-DeepResearch uses N = 8 and K = 4.The smaller Open-DeepResearch configuration is intended to enhance training efficiency.
- Evaluation: Qwen3-Max and Claude-4-Sonnet evaluate Open-Travel and Open-DeepResearch by comparing candidate outputs with baseline outputs for each test sample.
B Writing Benchmarks
The evaluation adopts three complementary writing benchmarks covering professional writing, real-user tasks, and ultra-long text generation. Together, they assess constraint compliance, response quality, summarization, creativity, stylistic fidelity, and coherence at scale.
- The evaluation uses WritingBench, HelloBench, and LongBench-Write as three complementary benchmarks.
- WritingBench: WritingBench evaluates domain-specific writing proficiency and controllability across six professional and creative domains.It emphasizes compliance with complex, multidimensional constraints.
- HelloBench: HelloBench evaluates diverse in-the-wild tasks derived from real user queries.The analysis covers open-ended QA, long-form summarization, and heuristic text generation.
- LongBench-write: LongBench-write evaluates coherent ultra-long texts, including outputs on the order of 10,000 words.It assesses topical consistency and global coherence as output length scales.
C Benchmarking Tool Annotation
The benchmarks expose tool interfaces for deep research and travel planning. Open-DeepResearch uses Google search with long-page summarization, while Open-Travel provides six tool types for location search, navigation, recommendations, and intercity transport.
- The benchmark section outlines the tools used by Open-DeepResearch and Open-Travel.
- Open-DeepResearch: Open-DeepResearch performs web search through the Google API and summarizes parsed webpages exceeding 2,500 characters with Qwen3-Max.A summarization model is integrated to prevent excessive context length.
- Open-Travel: Open-Travel annotates six tool types for travel planning.
- Open-Travel: Search poi and around search retrieve location information, while get navigation provides route planning across walking, driving, and public transit.
- Open-Travel: Universal search supplies city-level recommendations, and search flights and search train tickets return intercity transport options with prices, locations, and times.Train results distinguish direct from transfer routes.
- Open-Travel: The travel tools use Amap services for location, nearby search, and navigation, while universal search uses the Bailian search engine and flight and train outputs are simulated.
- Algorithm 1 presents the seeded single-elimination tournament procedure as a trade-off between training efficiency and advantage-estimation accuracy.
E Prompts
The appendix presents prompts for open-ended tasks and the seeded single-elimination procedure used to estimate trajectory advantages. The algorithm seeds trajectories, runs elimination rounds, assigns hierarchical tiers, and computes advantages.
- E Prompts: The appendix states that it illustrates all prompts used in the paper.
- E.3 Open-ended Writing Task Prompt: Open-ended writing prompts adapt to whether the query is Chinese or English.
- F Case Study: ArenaRL takes a trajectory group, an arena judge, and returns standardized advantages.
- F Case Study: The algorithm obtains initial scores and sorts trajectories descendingly to create ordered seeds.
- F Case Study: The tournament maintains a buffer of winners and losers while processing paired trajectories through elimination rounds.
- F Case Study: Seeded pairings compare high- and low-ranked trajectories before initializing elimination tiers.
- F Case Study: Losers are archived into tiers and winners advance, with the champion entering the final tier.
- F Case Study: Tiers are sorted by accumulated scores before the procedure performs advantage calculation.
F.1 Case Study of SFT model
The SFT case study evaluates a two-person, two-day Chengdu hiking request focused on natural scenery and photography within a 600 RMB-per-person budget. Its trajectory shows repeated information gathering, while the resulting itinerary specifies transportation, hiking activities, costs, and safety guidance.
- F.1 Case Study of SFT model: The case concerns a two-day weekend hiking trip from Chengdu to Qingcheng Back Mountain for two people, emphasizing scenery, photography, and a 600 RMB-per-person budget.
- F.1 Case Study of SFT model: The SFT trajectory begins by locating Qingcheng Back Mountain and planning transportation and hiking routes with search_poi.
- F.1 Case Study of SFT model: The trajectory then gathers transportation, route, attraction, and Chengdu location information through repeated search steps.
- F.1 Case Study of SFT model: The final itinerary recommends a Chengdu–Qingchengshan high-speed train, scenic-area transport, a two-day hike, and a 600 RMB-per-person budget.
- F.1 Case Study of SFT model: The itinerary includes route attractions such as Wulong Valley, Youyicun, Baiyun Cave, Cuiying Lake, Feiquan Valley, and Tai’an Ancient Town.
- F.1 Case Study of SFT model: Safety guidance covers steep trails, scarce toilets, lower mountain temperatures, heavy rain, and phone charging.
F.2 Case Study Following ArenaRL
The ArenaRL case study shows an agent progressively collecting attraction information, constructing a route, and requesting navigation for Qingcheng Back Mountain. The resulting guide provides transportation, tickets, and a detailed itinerary for the same travel query.
- F.2 Case Study Following ArenaRL: The ArenaRL case uses the same two-day Qingcheng Back Mountain hiking request for two people with natural scenery, photography, and a 600 RMB-per-person budget.
- F.2 Case Study Following ArenaRL: The reasoning trajectory gathers basic travel information and attraction locations through universal_search and repeated search_poi calls.
- F.2 Case Study Following ArenaRL: The agent searches for Youyicun, Cuiying Lake, and Laifeng Pavilion before assembling a route through the main scenic spots.
- F.2 Case Study Following ArenaRL: The planned route starts and ends at Tai’an Ancient Town and passes through Wulong Valley, Youyicun, Laifeng Pavilion, and Cuiying Lake.
- F.2 Case Study Following ArenaRL: The resulting guide describes high-speed-train or car access, scenic-area shuttle transport, entrance tickets, and a day-one hiking itinerary.