BIRD-RL: Scaling Agentic Reinforcement Learning over Stateful Data-Centric Environments

1 Introduction

• SQL issue resolution needs root-cause analysis and iterative database interaction, while container-per-rollout isolation creates spatial inefficiency and scaling bottlenecks.Stateful operations can include data manipulation and schema modification.Source: §1 · p. 2
• BIRD-RL decomposes SQL agent learning into four progressive stages, from reasoning imitation to interactive agentic reinforcement, to address instability from direct end-to-end learning.The stages are reasoning imitation, tool-use imitation, adaptive reasoning reinforcement, and interactive agentic reinforcement.Source: §1 · p. 2
• Trajectory-scoped execution combines a worker-partitioned database pool and customized agent loop to reuse replicas while preserving isolated stateful interactions.The design supports high-concurrency efficiency through orchestrated database-replica reuse.Source: §1 · pp. 2–3
• BIRD-RL also supports BIRD-ZENO, a unified multi-task model trained for SQL debugging and query generation, and releases the SIX-GYM-SQLITE training environment.The paper presents these as contributions beyond the BIRD-TALON debugging model family.Source: §1 · p. 3
• 44.40% and 48.00% Success Rates (SR) for BIRD-TALON-7B and 14B on BIRD-CRITIC-SQLITE were competitive with Claude-Opus-4.6 Agent (48.20%).The infrastructure improved spatial efficiency by 12.8×, while BIRD-ZENO-7B achieved 63.90% Execution Accuracy on the BIRD development set.Source: §1 · p. 3

2.1 Task Definition

• Text-to-SQL maps a natural-language question and database schema to a predicted query, while issue resolution revises a supplied query to address a described problem and preserve user intent.The issue-resolution inputs are the problem description, schema, and query with an issue.Source: §2.1 · p. 3

2.2 Tool-Centric Agent RL Loop

• A tool-centric loop samples tool invocations from the current history, appends returned observations, and ends at a final answer or the step budget.Tool observations are conditioned on the invocation and environment state.Source: §2.2 · p. 3
• Container-per-rollout isolation is prohibitively expensive for large mutable databases, but unbound replica reuse risks lost state continuity and cross-trajectory interference that destabilizes on-policy RL.Each trajectory must retain earlier database effects for later hypothesis testing.Source: §2.2 · pp. 3–4

3 BIRD-RL: Efficient Agentic RL Curriculum for SQL Agent Training

• The four-stage curriculum decomposes learning from static reasoning to stateful interaction to avoid instability from direct end-to-end optimization over a large interactive action space.The final stage uses trajectory-scoped infrastructure for scalable, contamination-free RL.Source: §3 · p. 4

3.1 Stage 1: Reasoning Imitation

• Stage 1 uses supervised fine-tuning on reasoning-and-query outputs to establish a schema-grounded reasoning prior and reduce naive exploration in interactive scenarios.The structured prior is intended to support more efficient execution and prevent unnecessary context expansion.Source: §3.1 · p. 4

3.2 Stage 2: Multi-Turn Tool-Use Imitation

• Stage 2 trajectory fine-tuning uses executable SQL actions for database inspection, hypothesis testing, and root-cause localization, with separate exploration and submission tools.The teacher-generated training set contains 4,502 verified trajectories.Source: §3.2 · p. 4

3.3 Stage 3: Adaptive Reasoning Reinforcement

• Stage 3 adds single-turn reasoning-focused RL between imitation and multi-turn exploration to encourage adaptation beyond rigid offline patterns.GRPO optimizes candidate outputs using execution-based rewards that measure test-suite correctness.Source: §3.3 · pp. 4–5

3.4 Stage 4: Stateful Interactive Agentic Reinforcement

• Stage 4 optimizes multi-turn trajectories for autonomous tool-augmented exploration, assigning execution-based reward only to the final solution and no process-level rewards.The trajectory-level likelihood ratio is computed over the full trajectory, and the reference policy comes from Stage 3.Source: §3.4 · p. 5

4 Trajectory-Scoped Persistent Execution Infrastructure

• Trajectory-scoped binding preserves each rollout’s database state across tool calls, addressing state continuity needed for SQL debugging.The infrastructure combines a worker-partitioned database pool with a customized stateful agent loop.Source: §4 · p. 5

4.1 Worker-Partitioned Database Pool

• The pool reuses pre-allocated database replicas, partitions them across rollout workers, and tracks availability and reset state to prevent cross-trajectory contamination.Each template database has K working replicas; modified replicas are reset before reuse.Source: §4.1 · pp. 5–6

4.2 Customized Stateful Agent Loop

• The agent loop binds a stable trajectory identifier to one replica until termination, then defers reset of modified replicas until their next allocation.A registry routes calls to the assigned replica; timeout reclamation handles abnormal rollouts.Source: §4.2 · p. 6

5.1 Experiment Settings

• Evaluation covers SQLite and multi-dialect SQL debugging plus BIRD-SQL generation, using Success Rate for debugging and Execution Accuracy for generation.The experiments use BIRD-TALON 7B and 14B debugging models and a mixed-task BIRD-ZENO-7B model, implemented with VERL on 8×H100 GPUs.Source: §5.1 · p. 6

5.2 Overall Performance

• Multi-turn agentic inference raises average overall SR by only 3.87%, while the strongest proprietary model reaches 48.80% SR.General-purpose models average 46.65% SR on query-like issues, falling to 38.00% on management and 36.79% on personalization issues.Source: §5.2 · p. 7
• 48.00% overall SR for BIRD-TALON-14B approaches Claude-Opus-4.6 Agent’s 48.20%; BIRD-TALON improves its 7B and 14B backbones by 18.20% and 14.20%, respectively.The comparison is on BIRD-CRITIC-SQLITE; Table 2 reports detailed baseline comparisons.Source: §5.2 · p. 7, Table 2 · p. 7

5.3 Analysis of Database Reuse Efficiency.

• 12.8× lower measured online lifecycle I/O than per-rollout containers comes with a one-time 69 GiB offline pool cost under the evaluated SQLite setting.Only 9.25% of practical trajectories execute state-altering queries and trigger a database reset; the reported I/O comparison excludes offline pool materialization.Source: §5.3 · p. 8, Table 3 · p. 8

5.4 Results on Heterogeneous SQL Tasks.

• BIRD-ZENO-7B maintains debugging capabilities while matching strong RL-based generation specialists comparably on BIRD-SQL development, supporting unified training across both tasks.The data-mixture study adopts a debug-heavy 3:1 ratio, which yields the strongest issue-resolution performance while retaining competitive generation capability.Source: §5.4 · pp. 8–9, Table 4 · p. 8, Figure 2 · p. 9

5.5 Case Study: Effectiveness of the Adaptive Reasoning Reinforcement Stage

• On a Stage 3 case, both agents identify the aggregation error, but only the agent with Stage 3 incorporates feedback into correct reasoning and actions.The comparison contrasts the agent with and without Adaptive Reasoning Reinforcement.Source: §5.5 · p. 9, Table 5 · p. 9

5.6 Ablation Study

• Removing curriculum stages reduces performance; excluding agentic RL produces larger drops on management (↓9.33) and personalization (↓7.10) than query-like issues (↓0.33).The study also compares execution-only reward with proxy-reward variants; the sparse, execution-based reward performs better.Source: §5.6 · pp. 9–10, Table 6 · p. 10

• Automatic correction loops are increasingly used in text-to-SQL agents, while debugging the full spectrum of real-world database-administration SQL operations remains largely unexplored [43] [29] [36].Source: §6 · p. 10
• Recent text-to-SQL work has pivoted from prompt-based systems toward multi-agent architectures [11] [35] [39] [36] [47] [6].Source: §6 · p. 10
• Prior SQL-domain RL frameworks steer query synthesis, whereas this work targets dynamic, stateful issue resolution requiring multi-step diagnosis and repair [33] [56] [28] [8] [37].Source: §6 · p. 10

• BIRD-RL is a four-stage agentic reinforcement-learning curriculum for SQL issue resolution.Source: §7 · p. 10
• Trajectory-scoped persistent execution supports isolated stateful interactions through orchestrated database-replica reuse.Source: §7 · p. 10
• The work releases BIRD-TALON for SQL debugging and BIRD-ZENO, a unified multi-task model trained with a principled data-mixing strategy.Source: §7 · p. 10
• The authors present BIRD-RL as a scalable framework for data-centric agent training and highlight multi-objective training potential for real-world SQL applications.Source: §7 · p. 10

A SIX-GYM-SQLITE: Training Environment Construction

• SQLite's in-process design avoids client-server overhead and supports high-throughput parallel rollouts; SIX-GYM-SQLite fills the lack of a dedicated SQLite training environment.Source: A SIX-GYM-SQLITE: Training Environment… · p. 16

A.1 SIX-GYM-SQLite

• The adapted task suite combines original SIX-GYM instances with tasks synthesized from newly collected Stack Overflow posts, retaining 5,000 instances.Source: §A.1 · p. 16
• Database-domain and instance separation, source-post exclusion, and teacher access restricted to training examples are used to prevent training–evaluation contamination.Source: §A.1 · p. 16
• Only 3 of 500 evaluation tasks (0.6%) were flagged as highly similar to training tasks; excluding them left the main conclusions unchanged.Source: §A.1 · p. 16
• The dataset documentation reports coverage of diverse SQL operations and application contexts, alongside task, token-length, evaluation-script, and database statistics.Source: §A.1 · p. 16

B.1 Workflow Orchestration of the Customized Stateful Agent Loop

• The stateful agent loop binds a database replica to each trajectory, lazily resets it when dirty, and releases it after termination.The pseudo-code describes allocation, SQL execution on the bound replica, and marking mutated databases for reset.Source: §B.1 · p. 16, §C.1 · p. 19

B.2 Auxiliary Proxy Reward Designs

• Auxiliary proxy rewards provide denser signals only for trajectories that fail every execution test, leaving any partially passing trajectory primarily optimized by execution reward.Source: §B.2 · pp. 16–17
• The partial-correctness design rewards executable syntax and schema linking, which compares schema items in predicted and gold SQL.Its additional partial reward combines execution validity with a schema-linking score.Source: §B.2 · p. 17
• Other proxy variants score failed trajectories with proprietary-LLM trajectory and solution judgments or a heuristic rubric for execution and checklist-based bug identification.Source: §B.2 · pp. 17–18

C Experiment Details

• The experiment-details section introduces the methods and implementation conventions used in the experiments.Source: C Experiment Details · p. 18

C.1 Alias of LLMs

• The alias list includes SQL-generation baselines such as Think2SQL-7B, SQL-R1-7B, and Arctic-Text2SQL-R1-7B, as well as proprietary-model aliases.Source: §C.1 · p. 18

C.2 Model Implementation Details

• Training uses four stages: two supervised-fine-tuning stages followed by two GRPO stages, with five repeated trials reported for experimental results.Stage 4 uses asynchronous rollouts of up to five assistant turns; its rollout size is four unless otherwise specified.Source: §C.2 · p. 19

C.3 Compared Methods

• Single-turn SQL-debugging baselines receive the issue description, schema, and buggy query, then produce a repair without database interaction.The evaluated models include GPT-5.4-Pro, Claude-Opus-4.6, Claude-Sonnet-4.5, Gemini-3.1-Pro, GLM-4.7, Kimi-K2.5, MiniMax-M2-1, and Qwen3-Coder-480B.Source: §C.3 · p. 20
• SQL-ACT provides multi-turn agents with executable SQL actions and database observations, continuing until they submit a repaired query.Unlike tool-based agents restricted to predefined operations, SQL-ACT uses SQL itself as the action space; BIRD-TALON uses the same scaffold at inference.Source: §C.3 · p. 20
• Table 1 includes Claude-Opus-4.6, Claude-Sonnet-4.5, and Qwen3-Coder-480B in the multi-turn setting; Table 2 also evaluates untrained Qwen2.5-Coder base models in SQL-ACT.The Qwen2.5-Coder models are placed directly into the interactive scaffold without BIRD-RL training.Source: §C.3 · p. 20
• Text-to-SQL comparisons cover RL methods using execution feedback, partial rewards, reasoning, or multi-turn database interaction.The compared approaches include Think2SQL [33], SQL-R1 [28], ReEx-SQL [8], Reasoning-SQL [37], Arctic-T2S-R1 [56], and SQL-Trail; SQL-Trail uses iterative interaction, adaptive turn allocation, and rewards balancing correctness with exploration efficiency.Source: §C.3 · pp. 20–21

D.1 Case Study for Stage 3

• With Stage 3, the agent verifies a repair that pre-aggregates expenses per budget line, while the agent without Stage 3 proposes an erroneous MAX(budget.amount) fix.Both agents identify join-induced budget inflation, but only the Stage 3 agent infers that an event may contain multiple budget lines.Source: §D.1 · p. 21, Figure 4 · p. 21

D.2 Comparison of Training-Time Database Lifecycle I/O

• A dynamic pool leases clean replicas to trajectories, tracks availability and modifications, and defers resetting a modified replica until its next lease.The naive per-rollout baseline instead incurs O(NR S̄) I/O; the pool releases replicas after trajectories complete rather than re-instantiating them.Source: §D.2 · pp. 21–22
• The pool’s lifecycle overhead is O((B + q/k)R S̄), versus O(NR S̄) for per-rollout containers, under the stated uniform-request assumption for bounded instantiation.Here q = Q/R captures reset-triggering rollouts relative to R, while k is the I/O reduction factor of in-place reset versus full materialization; about 9.25% of practical rollouts alter state.Source: §D.2 · pp. 21–22
• The implementation pre-materializes the pool offline to avoid allocation bottlenecks from skewed concurrent demand, leaving online I/O dominated by resets.Replica acquisition uses O(1) pool lookups; the reported offline storage cost is 69 GiB.Source: §D.2 · p. 22
• 12.8× lower measured online lifecycle I/O than per-rollout containers comes with a one-time 69 GiB pre-materialized storage cost, excluded from the online I/O measurement.The practical setting uses B = 16 and R = 16, reserving BR/2 replicas per database; observed capacity avoids pool exhaustion.Source: §D.2 · p. 22

D.3 Generalization for Different SQL Dialects

• BIRD-TALON generalizes to unseen SQL dialects despite training only on SQLite tasks, with the passage attributing this to learned debugging logic and retained pretrained dialect knowledge.Figure 5 presents the unseen-dialect success-rate result, but the supplied passages give no numerical values.Source: Figure 5 · p. 22, §D.3 · p. 23

D.4 Test-time Scaling Analysis

• Additional test-time samples consistently benefit BIRD-TALON-14B’s oracle pass@k, suggesting its policy offers candidates for verifier- or reranker-based scaling.The analysis samples up to 16 trajectories at temperature 0.7 and top-p 0.95; Table 10 reports the test-time scaling results.Source: §D.4 · p. 23, Table 10 · p. 23

E Limitations and Future Work

• The work focuses on SQL as a representative stateful data-centric task; extending BIRD-RL to data cleaning, transformation, workflow construction, and analytical decision-making is future work.These tasks would likewise involve reasoning over persistent states, execution feedback, and large data.Source: E Limitations and Future Work · p. 23
• BIRD-RL currently uses execution-based rewards for functional correctness; future work may combine execution signals with semantics-aware verification while addressing reward hacking.Source: E Limitations and Future Work · p. 23

F Ethics Statement

• The authors report using publicly available databases and resources, avoiding sensitive or personally identifiable information, and generating SQL queries rather than open-ended natural language.Source: F Ethics Statement · p. 23

G Broader Impact

• The authors frame trajectory-scoped infrastructure as a way to reduce isolated-rollout costs and heterogeneous-task training as a path toward unified SQL agents for generation, diagnosis, verification, and repair.They describe possible longer-term benefits including reduced manual debugging effort and improved developer productivity.Source: G Broader Impact · pp. 23–24

H Prompts and Examples

• The appendix presents prompts for debugging and generation, baseline debugging, Stage 1 reasoning imitation and verification, reward scoring, and an example BIRD-TALON trajectory.Figures 6–17 cover these prompts and the example trajectory.Source: H Prompts and Examples · p. 24