PaperlayerView exact revision on arXiv

Source-linked AI summary

Let It Flow: Agentic Crafting on Rock and Roll, Building the ROME Model within an Open Agentic Learning Ecosystem

Weixun Wang, XiaoXiao Xu, Wanhe An, Fangwen Dai, Wei Gao, Yancheng He, Ju Huang, Qiang Ji, Hanqi Jin, Xiaoyang Li, Yang Li, Zhongwen Li, Shirong Lin, Jiashun Liu, Zenan Liu, Tao Luo, Dilxat Muhtar, Yuanbin Qu, Jiaqiang Shi, Qinghui Sun, Yingshui Tan, Hao Tang, Runze Wang, Yi Wang, Zhaoguo Wang, Yanan Wu, Shaopan Xiong, Binchen Xu, Xander Xu, Yuchi Xu, Qipeng Zhang, Xixia Zhang, Haizhou Zhao, Jie Zhao, Shuaibing Zhao, Baihui Zheng, Jianhui Zheng, Suhang Zheng, Yanni Zhu, Mengze Cai, Kerui Cao, Xitong Chen, Yue Dai, Lifan Du, Tao Feng, Tao He, Jin Hu, Yijie Hu, Ziyu Jiang, Cheng Li, Xiang Li, Jing Liang, Xin Lin, Chonghuan Liu, ZhenDong Liu, Zhiqiang Lv, Haodong Mi, Yanhu Mo, Junjia Ni, Shixin Pei, Jingyu Shen, XiaoShuai Song, Cecilia Wang, Chaofan Wang, Kangyu Wang, Pei Wang, Tao Wang, Wei Wang, Ke Xiao, Mingyu Xu, Tiange Xu, Nan Ya, Siran Yang, Jianan Ye, Yaxing Zang, Duo Zhang, Junbo Zhang, Boren Zheng, Wanxi Deng, Ling Pan, Lin Qu, Wenbo Su, Jiamang Wang, Wei Wang, Hu Wei, Minggang Wu, Cheng Yu, Bing Zhao, Zhicheng Zheng, Bo Zheng

arXiv:2512.24873v3cs.AIcs.CL

TL;DR

Open-source agent development lacks a scalable, end-to-end ecosystem. The paper introduces ALE and develops ROME within it; ROME achieves consistent performance across diverse agentic benchmarks.

  • Problem

    Open-source agentic crafting lacks a scalable, end-to-end ecosystem, while prior approaches rely on limited demonstrations or ad hoc reinforcement-learning recipes.

    Source: §1 · p. 3

  • Method

    The paper introduces ALE, integrating ROLL, ROCK, and iFlow CLI, and develops ROME using curated tool-grounded trajectories and a staged training pipeline.

    Source: §1 · p. 3, §4 · p. 31

  • Results

    ROME achieves consistent performance across diverse agentic benchmarks, including 57.4% accuracy on SWE-bench Verified.

    Source: §1 · p. 3

  • Takeaways & Limitations

    The authors present ALE as a foundation for streamlining the development and deployment of agentic LLMs.

    Source: §4 · p. 31

    Existing terminal benchmarks are limited in scale, reliability, and diagnostic resolution, and task shortcuts can undermine conclusions about correctness, robustness, and generalization.

    Source: Motivation and Limitations of Existing… · pp. 27–28

Abstract

from arXiv · show

Agentic crafting requires LLMs to operate in real-world environments over multiple turns by taking actions, observing outcomes, and iteratively refining artifacts. Despite its importance, the open-source community lacks a principled, end-to-end ecosystem to streamline agent development. We introduce the Agentic Learning Ecosystem (ALE), a foundational infrastructure that optimizes the production pipeline for agentic model. ALE consists of three components: ROLL, a post-training framework for weight optimization; ROCK, a sandbox environment manager for trajectory generation; and iFlow CLI, an agent framework for efficient context engineering. We release ROME, an open-source agent grounded by ALE and trained on over one million trajectories. Our approach includes data composition protocols for synthesizing complex behaviors and a novel policy optimization algorithm, Interaction-Perceptive Agentic Policy Optimization (IPA), which assigns credit over semantic interaction chunks rather than individual tokens to improve long-horizon training stability. Empirically, we evaluate ROME within a structured setting and introduce Terminal Bench Pro, a benchmark with improved scale and contamination control. ROME demonstrates strong performance across benchmarks like SWE-bench Verified and Terminal Bench, proving the effectiveness of ALE.

ROCK & ROLL & IFLOW & DT Joint Team

The section identifies ROCK, ROLL, iFlow CLI, Terminal Bench Pro, and iFlow-ROME; Figure 1 is described as an overview of ALE and ROME performance.

ROCK & ROLL & IFLOW & DT Joint Team

1 Introduction

The paper presents ALE as an end-to-end ecosystem for agentic crafting, where LLMs plan, act, and self-correct through multi-turn interaction with real-world environments. It uses ALE to develop ROME and argues that agentic progress requires co-designing training infrastructure, executable environments, and evaluation protocols.

Motivation

  • The paper frames agentic crafting as a shift from one-shot generation toward grounded, iterative workflows and calls for collaborative benchmarks, standardized execution environments, and reproducible training pipelines.It identifies co-design of training infrastructure, executable environments, and evaluation protocols as central to complex agentic settings.Source: §1 · pp. 3–4

Agentic Learning Ecosystem

  • ALE unifies data generation, agent execution, and policy optimization through ROLL, ROCK, and iFlow CLI.ROLL supports multi-environment RL rollouts and chunk-aware credit assignment; ROCK provides secure sandboxed execution and trajectory synthesis, execution, and validation; iFlow CLI orchestrates structured prompts and packages agents for real-world workflows with user-feedback APIs.Source: §1 · p. 3

ROME training

  • ROME is an open-source Qwen3-MoE agent trained through a curated trajectory-composition workflow and a staged pipeline from continuous pre-training to SFT and RL.ROCK's isolation and permission controls support security, safety, and validity checks on synthesized multi-source, multilingual, tool-grounded trajectories.Source: §1 · p. 3
  • IPA assigns policy credit to semantic interaction chunks rather than individual tokens to stabilize long-horizon RL and strengthen long-context agentic crafting.The report presents IPA as a component of ROME's training pipeline.Source: §1 · p. 3

Empirical results

  • ROME achieves 57.4% accuracy on SWE-bench Verified and 24.7% on Terminal-Bench v2.0, and performs competitively on Terminal Bench Pro under stricter contamination control and improved domain balance.The report also says ROME approaches the performance of larger models exceeding 100B parameters on terminal-centric tasks and is stably deployed in iFlow CLI.Source: §1 · p. 3

2 Agentic Learning Ecosystem: ROME Wasn’t Built in a Day

ALE combines ROLL, ROCK, and iFlow CLI into infrastructure for agentic crafting, supporting the pipeline from training to deployment. ROLL handles post-training, ROCK manages sandboxed environments, and iFlow CLI manages agent context and workflows.

2.1 System Overview

  • ROLL, ROCK, and iFlow CLI jointly support agentic RL training: ROLL issues environment calls, ROCK executes them in sandboxes, and iFlow CLI coordinates context between LLM responses and observations.The paper characterizes the combined infrastructure as efficient, fault-tolerant, and scalable.Source: §2.1 · p. 4

Agentic Training Pipeline.

  • ROLL separates inference, environment interaction, reward computation, and parameter updates into specialized worker roles that can scale independently in distributed execution.A controller coordinates heterogeneous workers and their deployment and lifecycle management.Source: Agentic Training Pipeline · pp. 4–5

Fine-grained Rollout.

  • ROLL enables sample-level pipelining by decomposing rollout into LLM generation, environment interaction, and reward computation, rather than processing each phase as one full batch.Asynchronous reward computation and per-sample lifecycle control let users decide when and where each phase runs.Source: Fine-grained Rollout · p. 5

Asynchronous Training.

  • ROLL’s asynchronous pipeline buffers completed trajectories, constrains per-sample policy staleness, and overlaps rollout with training after synchronizing updated weights.Samples that exceed the asynchronous-ratio constraint are discarded to preserve model accuracy.Source: Asynchronous Training · p. 5, Figure 3 · p. 5

Train–Rollout Multiplexing.

  • Dynamic train–rollout GPU multiplexing shifts resources from rollout to training when the buffer is ready, then returns them to rollout to reduce bubbles and improve utilization.This addresses long-tail rollout latency and bursty training demand that leave GPUs underused under static partitioning.Source: Train–Rollout Multiplexing · pp. 5–6, Figure 3 · p. 5

2.3 Environment Execution Engine: ROCK

  • ROCK provides framework-agnostic, sandboxed environment management through a client-server architecture, with Admin orchestration, Worker nodes, Rocklet proxies, and an image registry.Its Sandbox and GEM APIs support environment lifecycle and agent interaction; ROCK also supports multi-agent setups, fault isolation, and large-scale scheduling.Source: §2.3 · pp. 6–7, API Interfaces · p. 7

Agent Native Mode.

  • Agent-native mode proxies requests containing iFlow CLI’s complete conversation context to ROLL inference during training or external APIs during deployment, keeping context handling consistent without duplicating agent logic.The mode also supports multiple agent frameworks across data synthesis, training, and evaluation.Source: Agent Native Mode · pp. 7–8

2.4 Agent Framework: iFlow CLI

3 Agentic Model: ROME is Obviously an Agentic ModEl · 3.1 Data Composition · Task Construction and Formalization.

ROME is presented as an ALE-grounded agentic model built through a systematic data and training pipeline. Its data design progresses from foundational coding and reasoning to executable, feedback-rich tasks and trajectories that support planning, execution, and adaptation.

3 Agentic Model: ROME is Obviously an Agentic ModEl

  • ROME’s ALE-based development combines data acquisition and synthesis, Agentic CPT, SFT, IPA reinforcement learning, and a comprehensive benchmark suite.Source: §3 · p. 10

3.1.1 Agent Competencies as a Blueprint for Data Design

  • The data blueprint targets task understanding and planning, action and execution, and interaction and adaptation through a two-tier curriculum from foundational skills to closed-loop behavior.Basic Data builds coding and reasoning without full environment orchestration; Agentic Data adds executable settings and runtime signals for planning, execution, and iterative refinement.Source: §3.1.1 · pp. 10–11

3.1.2 Code-Centric Basic Data Composition

  • The code-centric corpus draws on approximately one million GitHub repositories, project-level code samples, and filtered, explicitly linked closed-Issue/merged-PR pairs.Repositories are selected using activity signals; Issues and PRs are filtered to retain clear problem–solution correspondence.Source: §3.1.2 · p. 11

Task Construction and Formalization.

  • Issue–PR pairs support five task types: localization using AGENTLESS (Xia et al., 2024) modified-file targets, search-and-replace repair, test generation, SWE-RL (Wei et al., 2025) feedback-edit trajectories, and CoT reasoning.Rejection sampling retains localization samples covering all modified files and filters repair and test-generation samples by sequence-level similarity to golden patches.Source: Task Construction and Formalization · pp. 11–12
  • The initial corpus exceeded 200B tokens, then hygiene and quality checks distilled it to 100B tokens for continuous pre-training and post-training.Source: Task Construction and Formalization · p. 12

3.1.3 Agentic Data Composition

  • Agentic data packages executable specifications, pinned environments, and verifiable feedback, while trajectories record multi-turn tool use, edits, and runtime signals for long-horizon revision and recovery.Instances include prompts, Dockerfiles, build/test commands, and unit tests; trajectories may include reasoning traces and capture stateful dependencies and partial-failure recovery.Source: §3.1.3 · p. 12
  • Because open-source artifacts are sparse and noisy for agentic needs, the authors propose general tool-use data plus four-stage programming-centric synthesis, with multi-agent verification filtering unreliable or ambiguous executions.The synthesis strategy targets high-fidelity, verifiable instances and diverse trajectories at scale.Source: §3.1.3 · pp. 12–13

General Tool-Use Data Construction. · Programming-Centric Data Construction. · Data Filtering: Multi-Stage Filtering Pipeline for Rigorous Testing.

The data pipeline builds broad tool-use and programming-focused agentic datasets, then filters execution traces to reduce reward corruption and support training on complex behaviors.

General Tool-Use Data Construction.

  • Basic tool-use data is synthesized from parsed task dialogues into standardized calls with execution feedback across four single- or multi-turn, single- or multi-tool settings.Curated documentation supports tool selection and parameter grounding; API and MCP traces ground calls in actual execution environments.Source: General Tool-Use Data Construction · p. 13
  • Interactive tool-use tasks use an e-commerce sandbox and synthesized systems such as file and billing environments, with state- and schema-grounded tasks and outcome validation.LLM-simulated users add realism, while checks validate call syntax and whether post-interaction outcomes match task expectations; the corpus establishes planning, tool-selection, and state-tracking prerequisites.Source: General Tool-Use Data Construction · p. 13

Programming-Centric Data Construction.

  • Programming task drafts draw on linked PRs and Issues, code and terminal workflows, repositories spanning additional languages, and creative task variants filtered for feasibility.The explore agent uses more than 20,000 repositories and separates PR fix patches from test patches for independence and reproducibility.Source: Programming-Centric Data Construction · p. 13
  • The instance builder creates Docker-pinned tasks and iteratively validates image function, compilation, unit tests, and semantic alignment between tests and task instructions.A separate review agent checks specification fidelity, implementation completeness, solvability, test coverage, and false positives using an independent external language model.Source: Programming-Centric Data Construction · pp. 13–14
  • Trajectory agents run diverse scaffolds and LLMs on validated instances, then assess completion with unit tests and analyze tool use, loops, redundancy, and task alignment.Successful trajectories record planning, reasoning, tool calls, file edits, and environment interactions for training across languages, ecosystems, and application scenarios.Source: Programming-Centric Data Construction · p. 14
  • 76K instances and trajectory records totaling 30B tokens combine broad tool handling with environment-pinned programming supervision for execution fidelity and adaptive iteration.The combined datasets aim to move models from basic tool literacy toward specialized, high-confidence agentic capabilities.Source: Programming-Centric Data Construction · p. 14

Data Filtering: Multi-Stage Filtering Pipeline for Rigorous Testing.

  • The four-stage filtering pipeline gates traces through heuristic checks, an LLM judge, sandbox execution, and expert audits to curate high-confidence data and minimize reward corruption.It targets false positives and false negatives from brittle tests, ambiguous specifications, and incomplete ground-truth checks that can induce optimization drift.Source: Data Filtering: Multi-Stage Filtering… · pp. 14–15

3.1.4 Safety-Aligned Data Composition

  • Training rollouts revealed unsolicited unsafe tool and code actions, including internal-network probing, reverse SSH tunneling, and GPU cryptocurrency mining; the authors organize general-security risks as safety and security, controllability, and trustworthiness.They assembled security scenarios and red-team injections across prompts, repositories, and tools, then generated golden trajectories without these issues for post-training.Source: §3.1.4 · pp. 15–16

3.2 Training Pipeline … Introduction of Training Stages.

The training pipeline combines continual pre-training, two-stage supervised fine-tuning, and reinforcement learning to develop agentic capabilities and prepare the policy for long-horizon optimization. Its training data and objectives are designed to improve behavioral reliability and reduce noise from execution failures.

3.2 Training Pipeline

  • The pipeline combines agentic continual pre-training, two-stage SFT, and IPA-based reinforcement learning, with a reformulated SFT objective addressing execution-failure gradient noise and inefficient learning.IPA trains and samples REINFORCE at the semantic interaction-chunk level toward long-horizon success.Source: §3.2 · p. 16

Stage I: Mastery of Atomic Tasks.

  • Stage I trains on approximately 500B tokens of structured code tasks and broader reasoning and tool-use data to establish coding, reasoning, and interactive behaviors.It uses next-token prediction with a 32M-token global batch and a constant learning rate of 3 × 10−5.Source: Stage I: Mastery of Atomic Tasks · pp. 16–17

Stage II: Emergence of Agentic Solver.

  • Stage II trains on approximately 300B synthesized trajectories from teacher models interacting with sandbox environments, including successful executions and corrected failures to support recovery and long-horizon planning.Training hyperparameters remain consistent with Stage I except for weight decay, which is linearly annealed from 0.1 to 0.01.Source: Stage II: Emergence of Agentic Solver · p. 17

3.2.2 Anchoring Reinforcement Learning in Reliable Policy Regions via Supervised Fine-Tuning

  • Two-stage SFT replaces naive single-step SFT, which elicits basic multi-turn tool use but does not master complex logic and state transitions, with a procedure addressing gradient noise and inefficient sample use.The stages are heuristic-guided naive SFT followed by adaptive valuable-data revisiting.Source: §3.2.2 · p. 17, Introduction of Training Stages · p. 18

Introduction of Training Stages.

  • Naive SFT analysis finds that verbose overthinking degrades efficiency and tool use, ungrounded reasoning encourages repetitive tool calls, and test-passing demonstrations can still contain logical or semantic errors.Source: Introduction of Training Stages · p. 17
  • High-quality programming examples, particularly Python, improve cross-domain generalization, while multilingual data preserves reasoning consistency without degrading tool-use performance.Source: Introduction of Training Stages · p. 17
  • The million-scale SFT dataset contains 70% agentic, 15% reasoning-intensive, and 15% general-purpose instruction data across approximately 15 languages, with filtering to remove overthinking and low-quality trajectories.Filtering removes redundant tool calls, incomplete interactions, self-repair loops, and fake positives, then ranks remaining trajectories with an LLM-as-Judge.Source: Introduction of Training Stages · pp. 17–18
  • Adaptive Stage 2 revisits high-confidence trajectories using verified interactions, expert-audited demonstrations, and preference-refined samples to produce supervision better aligned with downstream RL.Its quality controls emphasize verifiability, style consistency, and reproducibility, combining executability constraints with soft scoring of efficiency and strategic coherence.Source: Introduction of Training Stages · p. 18

Error-Masked Training Enhances Training Stability. · Task-Aware Context Masking Ensures Training Efficiency. · Loss Formulation of the Whole SFT Training Objective.

The SFT objective suppresses loss from failed interactions and context irrelevant to a subtask, combining execution correctness and task relevance in its mask. The RL pipeline curates verifiable instances and optimizes structured interaction chunks for agentic training.

Error-Masked Training Enhances Training Stability.

  • Error-masked training zeros token-level losses for turns with tool-execution errors, preventing gradients from reinforcing failure-prone behaviors.It uses runtime execution feedback so updates come from executable, semantically valid trajectories.Source: Error-Masked Training Enhances… · p. 18

Task-Aware Context Masking Ensures Training Efficiency.

  • Task-aware context masking retains turns relevant to the current subtask and masks redundant, similar, or pruned historical turns.Pattern-based heuristics such as tool-call triggers and loop-entry markers identify task-specific decision boundaries.Source: Task-Aware Context Masking Ensures… · pp. 18–19

Loss Formulation of the Whole SFT Training Objective.

  • The SFT objective uses a binary interaction-level mask factorized into error and task-relevance components; only error-free, relevant turns contribute to loss.The objective is defined over each dialogue state before a turn and the model response at that turn.Source: Loss Formulation of the Whole SFT… · p. 19

3.2.3 Prepare Training Instance for Reinforcement Learning

  • The RL candidate pool contains approximately 60K instances from human-annotated synthesized examples and expert-designed long-horizon software-engineering tasks.Instances are selected for verifiable execution outcomes and sufficient task complexity.Source: §3.2.3 · p. 19
  • Difficulty-based filtering retains approximately 2K moderately difficult RL instances and excludes unstable environments and misaligned task specifications.Difficulty is estimated using pass rates from strong open-source baselines and the SFT model; tests are withheld until evaluation.Source: §3.2.3 · p. 19

3.2.4 Towards Efficient and Scalable Agentic Reinforcement Learning

  • IPA treats interaction chunks—not individual tokens or full trajectories—as its optimization unit to improve credit assignment and training stability in multi-turn agentic settings.A chunk is consecutive agent-environment communication contributing to a high-level subgoal and ending with a tool call; IPA builds on a REINFORCE variant.Source: §3.2.4 · p. 19
  • IPA also introduces a sampling strategy intended to reduce low-quality trajectory rollouts and improve sample efficiency.Figure 8 depicts the IPA training pipeline.Source: §3.2.4 · pp. 19–20, Figure 8 · p. 20

REINFORCE as a powerful baseline. · Adapt REINFORCE to the off-policy training.

The paper uses REINFORCE as a simple sequence-reward baseline, then adapts it for off-policy agentic training with geometric-mean importance ratios and different updates for positive and negative trajectories.

REINFORCE as a powerful baseline.

Adapt REINFORCE to the off-policy training.

  • In industrial-scale asynchronous agentic training, REINFORCE degrades as high off-policy ratios make the old policy increasingly outdated, creating distribution shift and a biased objective.This contrasts with its effectiveness on single-turn reasoning tasks.Source: Adapt REINFORCE to the off-policy… · p. 20
  • The method replaces continued multiplication in Truncated Importance Sampling with a geometric mean to make importance ratios more robust to low-probability tokens.The passage attributes the geometric-mean approach to Zheng et al. (2025b) and Zhao et al. (2025).Source: Adapt REINFORCE to the off-policy… · p. 20
  • Importance sampling corrects the off-policy objective, but its naive form can yield high-variance gradients and unstable updates, motivating Truncated Importance Sampling.The paper notes that inference–training engine mismatches are not included in this analysis.Source: Adapt REINFORCE to the off-policy… · p. 20
  • Uniform Truncated Importance Sampling clips positive and negative samples alike, whereas the adapted method applies it only to negative samples to avoid interfering with positive-sample gradients.The passage follows TOPR (Roux et al., 2025) for this asymmetric treatment.Source: Adapt REINFORCE to the off-policy… · p. 20
  • The adapted objective combines return-weighted supervised updates for positive trajectories with clipped importance-sampling updates for negative trajectories, aiming to avoid brittleness and policy collapse.The negative-sample update addresses distribution shift from large-scale negative sampling that can squeeze probability onto useless tokens.Source: Adapt REINFORCE to the off-policy… · pp. 20–21

Handle the inference-training mismatch. · Dynamic trajectory filtering for data refinement.

IPA organizes multi-turn agentic training around interaction chunks, refining credit assignment and rollout sampling while addressing inference–training divergence and unreliable trajectories.

Handle the inference-training mismatch.

  • Inference and training engines can diverge in execution backends, quantization, or batching, creating hidden instability even with shared parameters; token-level masking excludes tokens above the importance-weight threshold.The mismatch is attributed to the dominant agentic-model training paradigm rather than a particular engine.Source: Handle the inference-training mismatch · p. 21

Dynamic trajectory filtering for data refinement

  • Dynamic filtering rejects trajectories with environmental failures or illegal tool calls, then resamples from the same initial state with the current policy to maintain valid batch construction.Unreliable rewards on noisy interactions can inject misleading gradients and trigger policy collapse.Source: Dynamic trajectory filtering for data… · p. 21

3.2.4.2 Modeling Multi-Turn Agentic Task as Chunked MDP

  • The Chunked MDP treats each variable-length interaction chunk, typically ending in a tool call, as an action, aligning credit assignment with environmental transitions rather than token or sentence granularity.States encode interaction history; transitions reflect agent generation and stochastic tool responses, and positive reward is sparse, arriving when all unit tests pass.Source: §3.2.4.2 · p. 22

3.2.4.3 Reconstruct Training Objective via Chunk-Level Optimization

  • IPA aligns importance sampling and mismatch masking with chunk-level optimization, using a geometric-mean ratio to damp outlier-token effects and relaxing the masking constraint to preserve stability.Fine-grained token weights can misalign with outcome-driven rewards, while token-level policy gradients use state distributions induced by the inference policy.Source: §3.2.4.3 · pp. 22–24
  • Chunk-level discounted returns avoid token-level signal decay and improve training stability and task performance, as shown in Figure 10.All tokens in a chunk share its return weight; discounting by chunks shortens the effective horizon and suppresses early ineffective attempts relative to outcome-proximate chunks.Source: Figure 10 · p. 23, §3.2.4.3 · p. 23

3.2.4.4 Rollout Paradigm Refinement via Chunk-Level Initialized Resampling

  • Sequential Rollback starts resampling near the end of an expert-like trajectory and moves toward earlier chunks, reducing exploration burden and enabling progressive learning of crucial forks.The reported results show more positive training rollouts and improved test performance over naive sampling on a challenging task.Source: §3.2.4.4 · pp. 24–25, Figure 12 · p. 25
  • Parallelized Initialization samples from multiple anchor chunks at once to avoid Sequential Rollback’s costly backward search; a hybrid IL/RL objective uses expert chunks as a fallback when resampling yields no positive trajectories.The method combines imitation learning on expert chunks with chunk-level reinforcement learning on resampled chunks.Source: §3.2.4.4 · pp. 25–26
  • IPA’s chunk-level resampling improves generalization on challenging tasks and enables ROME, a 30B MoE model, to achieve capabilities comparable to a 480B agentic model.Figure 13 reports improved performance with Chunk-Level Initialized Resampling; the text attributes ROME’s capabilities to IPA.Source: Figure 13 · p. 26, §3.2.4.4 · p. 26

3.3 Experiments and Benchmark

The evaluation spans tool use, general agentic capabilities, and terminal-based execution as measures of real-world deployability. The section also introduces Terminal-Bench Pro to address limitations in existing benchmark coverage, difficulty calibration, and contamination control.

3.3.1 Evaluation Setup

  • The evaluation framework covers tool use, general agentic capabilities, and terminal execution as practical measures of real-world agent deployability.These dimensions range from tool calling to long-horizon, environment-grounded task completion.Source: §3.3.1 · pp. 26–27
  • Tool-use benchmarks include TAU2-Bench, BFCL-V3, and MTU-Bench, while general capabilities are assessed with BrowseComp-ZH, ShopAgent, and GAIA.The tool-use tasks assess tool selection, invocation, and coordination; the general capability suite targets complex queries requiring multi-step reasoning and decision-making.Source: §3.3.1 · p. 26
  • Terminal execution is tested on Terminal-Bench 1.0 and 2.0, SWE-bench Verified, SWE-Bench Multilingual, and the extended benchmark.These benchmarks require goal-directed workflows in executable environments.Source: §3.3.1 · pp. 26–27

3.3.2 Terminal Bench Pro: A More Rigorous and Fine-Grained Benchmark for Terminal Agents

  • Terminal-Bench Pro is introduced to provide larger-scale, domain-balanced coverage, calibrated difficulty, and stronger safeguards against data leakage.Figure 14 characterizes the benchmark and compares it with other benchmarks; the supplied passage does not provide the comparison values.Source: §3.3.1 · p. 27, Figure 14 · p. 27

3.3.1 Evaluation Setup

  • Evaluations use temperature 0.7, top-p 0.8, top-k 20, output limits of 65,536 tokens, and context limits of 262,144 tokens; CLI tasks share iFlow CLI, and Pass@1 is Avg@3.An asterisk marks scores from official reports or public leaderboards.Source: §3.3.1 · p. 27

Motivation and Limitations of Existing Benchmarks.

Existing terminal benchmarks are limited in scale, reliability, and diagnostic resolution, weakening aggregate rankings and domain-level evaluation. Sparse test suites can also let agents pass through shortcuts rather than demonstrate correctness, robustness, or generalization.

Motivation and Limitations of Existing Benchmarks.

  • Terminal Bench 1.0 has 80 tasks and Terminal Bench 2.0 has 89, leaving aggregate rankings sensitive to idiosyncratic instances and wide confidence intervals.The small task counts also limit the scale of evaluation.Source: Motivation and Limitations of Existing… · p. 27
  • Tasks sensitive to external network conditions introduce environment-induced variance unrelated to agent capability, reducing reproducibility and complicating performance attribution.Source: Motivation and Limitations of Existing… · p. 27
  • Some sub-domains have only one to three tasks, producing unstable domain-level estimates and substantial category-wise pass@1 variance across benchmarks.Examples include one games task and three machine-learning tasks.Source: Motivation and Limitations of Existing… · p. 27
  • Broad tasks with sparse test suites have low coverage, allowing agents to exploit underspecified requirements or unintended shortcuts and weakening conclusions about correctness, robustness, and generalization.Source: Motivation and Limitations of Existing… · pp. 27–28

Design and Construction.

Terminal Bench Pro is designed for fine-grained, rigorous evaluation of terminal-based agents through broad domain coverage, expert validation, and reproducible environments. Its 400-task dataset spans eight CLI domains, while the evaluation overview reports ROME’s agentic performance advantage at comparable or smaller parameter budgets.

Design and Construction.

  • Each benchmark instance includes an executable test file and reproducible environment to eliminate nondeterminism from external network or system dependencies.This deterministic evaluation setup is one of the benchmark’s three stated design principles.Source: Design and Construction · p. 28
  • Tasks span eight CLI domains informed by GitHub issue discussions, are authored from scratch by programmers, and undergo independent expert review for clarity, reference quality, and test coverage.The domains are data processing, games, debugging, system administration, scientific computing, software engineering, machine learning, and security.Source: Design and Construction · p. 28
  • The benchmark contains 400 tasks—200 public and 200 private—uniformly distributed across eight domains, with high test coverage, task diversity, and low evaluation variance.These properties are presented as a reliable basis for systematic assessment of terminal-based agentic systems.Source: Design and Construction · p. 28

3.3.3 Evaluation Results

  • ROME shows an agentic performance advantage at comparable or smaller parameter budgets, surpassing the performance ceiling typically observed in standard-sized models.Figure 15 summarizes average scores on general agentic and code agent benchmarks against total and activated parameters.Source: §3.3.3 · p. 28, Figure 15 · p. 28

Evaluation on Terminal-Based Benchmarks. · Evaluation on Tool-Use Benchmarks. · Evaluation on General Agentic Benchmarks.

Across terminal, tool-use, and general-agent evaluations, ROME posts strong benchmark results against comparable and larger models, while Terminal Bench Pro exposes persistent difficulty for all evaluated agents.

Evaluation on Terminal-Based Benchmarks.

  • 41.50% on Terminal-Bench 1.0, 24.72% on Terminal-Bench 2.0, 57.40% on SWE-Bench Verified, and 40.00% on SWE-Bench Multilingual; ROME outperforms normal-sized models across all four.The benchmarks require iterative reasoning, tool use, error recovery, and state maintenance across multiple interaction steps.Source: Evaluation on Terminal-Based Benchmarks · p. 29
  • 41.50% on Terminal-Bench 1.0 exceeds Qwen3-Coder-480B-A35B-Instruct (37.92%) and DeepSeek-V3.1 (38.75%), and is comparable to Qwen3-Coder-Plus (39.58%) and Kimi-K2 (39.25%).On SWE-Bench Verified, ROME scores 57.40%, above GLM-4.5 Air (56.20%), Gemini-2.5 Flash (28.73%), and GPT-OSS-120B (43.93%).Source: Evaluation on Terminal-Based Benchmarks · p. 29
  • All evaluated models, including ROME and large-scale baselines, score low on Terminal Bench Pro, whose stricter criteria, deeper horizons, and complex dynamics expose error compounding and brittle planning.The results indicate substantial headroom for future research on realistic, high-difficulty terminal tasks.Source: Evaluation on Terminal-Based Benchmarks · pp. 29–30

Evaluation on Tool-Use Benchmarks.

  • 49.46% average across six tool-use tests puts ROME ahead of Qwen3-Coder-30B-A3B (40.87%) and Devstral Small 2 (39.35%).ROME scores 62.45% on MTU-Bench (Single-Turn), above Gemini-2.5 Flash (45.93%) and GPT-OSS-120B (54.16%), and scores 62.28% on Tau2-Bench Retail and 50.50% on Tau2-Bench Airline.Source: Evaluation on Tool-Use Benchmarks · p. 30, Table 3 · p. 30
  • ROME remains competitive with much larger tool-use models, scoring 49.46% average versus Qwen3-Coder Plus (47.41%) and DeepSeek-V3.1 (49.94%).On MTU-Bench (Single-Turn), ROME scores 62.45%, exceeding DeepSeek-V3.1 (61.71%); ROME activates 3B parameters versus 37B for DeepSeek-V3.1 and 35B for Qwen3-Coder 480B.Source: Evaluation on Tool-Use Benchmarks · p. 30, Table 4 · p. 30

Evaluation on General Agentic Benchmarks.

  • GAIA tests everyday multi-tool queries, BrowseComp-ZH tests Chinese multi-hop web search with evidence aggregation, and ShopAgent tests e-commerce assistance across multi-step interactions.ShopAgent assesses preference inference, product retrieval and comparison, reasoning over structured attributes, and adaptation to evolving user intent.Source: Evaluation on General Agentic Benchmarks · pp. 30–31
  • 25.64% average on general-agent benchmarks exceeds Qwen3-Coder-30B-A3B-Instruct (15.69%) and Devstral Small 2 (16.30%).ROME also outperforms Gemini-2.5 Flash (22.66%), GLM-4.5 Air (24.78%), Qwen3-Coder-Plus (23.99%), and Qwen3-Coder-480B-A35B-Instruct (23.88%), and performs close to Kimi-K2.Source: Evaluation on General Agentic Benchmarks · p. 31, Table 5 · p. 31, Table 6 · p. 31
  • ROME scores 34.53% on ShopAgent Single-Turn and 29.61% on Multi-Turn, substantially surpassing other normal-sized models and exceeding GLM-4.6 on the task.The Multi-Turn setting is more challenging because users can revise or refine intentions, requiring intent clarification and adaptive planning.Source: Evaluation on General Agentic Benchmarks · p. 31, Table 5 · p. 31

4 Conclusion

The conclusion presents ALE as an end-to-end ecosystem for developing agentic LLMs and ROME as evidence of its practical potential. Evaluations across benchmarks, including Terminal Bench Pro, underscore ROME’s strong performance.

4 Conclusion

  • ALE integrates ROLL, ROCK, and iFlow CLI to optimize the complete agent-LLM production pipeline, with IPA anchoring its training pipeline.The conclusion frames this infrastructure as groundwork for streamlining the development and deployment of agentic LLMs.Source: §4 · p. 31
  • ROME is an open-source agent built on ALE and trained on extensive trajectories, demonstrating the potential of the ecosystem-based approach.The conclusion says ALE addresses challenges that have impeded practical implementation and adoption of agents in the open-source community.Source: §4 · pp. 31–32
  • ROME shows strong performance across diverse contexts in evaluations using multiple benchmarks, including the newly proposed Terminal Bench Pro.The authors present these results as evidence of ALE’s practicality and effectiveness.Source: §4 · pp. 31–32

6 Appendix

The appendix evaluates ROME on real-world tasks and reports advantages over evaluated baselines, including larger models. It also presents representative case studies showing stronger execution and visual/page quality than comparable-size models, with results competitive with large-scale models.

6.1 Real-world Case Study and Subjective Evaluation

  • The evaluation uses 100 distinct tasks from de-identified iFlow CLI user logs, scored by 20 blinded experts whose majority votes determine overall win rates.The rubric covers functionality and interaction, layout and style, code quality and robustness, structural and semantic correctness, and innovation and prompt understanding; it compares ROME with two similarly sized and two large-scale models.Source: §6.1 · p. 34, Table 7 · p. 34
  • On the 100-task benchmark, ROME outperforms all evaluated baselines in overall task execution quality, including larger Qwen3-Coder-Plus and GLM-4.6.The pairwise win-rate matrix reports blinded majority-vote comparisons; the passage provides no specific win-rate values.Source: §6.1 · p. 34, Figure 16 · p. 34
  • In two representative cases, ROME shows stronger task execution and visual/page quality than comparable-size models, with results competitive with large-scale models.The examples are Sleep Management System Generation and Solar System Modeling, shown in Figures 17 and 18.Source: §6.1 · pp. 34–35, Figure 17 · p. 36, Figure 18 · p. 37

Related papers

Loading 2512.24873v3…