PaperlayerView exact revision on arXiv

Source-linked AI summary

ProcArena: A Multi-Scenario Benchmark for LLMs on Direct and Interactive PL/SQL Development from Natural Language

Hang Zhang, Chaokun Wang, Yuzhi Pan, Ziyao Zhong, Shuo Cao, Yue Xue, Zeyu Huang, Xingwei Zhou, Fang Niu, Bofan Xie, Guanchen Ge, Leqi Zheng, Ziyang Liu, Xiannian Cao, Pengcheng Ge

arXiv:2609.06527v1cs.CLcs.AIcs.DB

TL;DR

Existing NL-to-PL/SQL benchmarks do not comprehensively evaluate diverse development scenarios, direct and interactive interaction, and multiple dialects. ProcArena addresses this gap with an execution-based benchmark and controlled interaction protocol, finding best average scores of 62.2% in Direct and 57.8% in Interactive modes. Its scope is procedural SQL extensions across Oracle PL/SQL, PostgreSQL PL/pgSQL, and related dialects.

  • Problem

    Existing NL-to-PL/SQL work primarily generates code from complete requirements, leaving diverse scenarios, interactive development, and multi-dialect evaluation insufficiently covered.

  • Method

    ProcArena constructs executable Direct tasks through iterative logic enhancement and scenario-specific adapters, derives paired Interactive tasks through knowledge integration and requirement perturbation, and uses a controlled Solver–User Simulator protocol.

  • Results

    62.2% and 57.8% are the best average scores achieved across Direct and Interactive modes, respectively, in the evaluation of seven models.

  • Takeaways & Limitations

    Realistic NL-to-PL/SQL development remains challenging, with lower best average performance in Interactive than Direct evaluation.

  • Takeaways & Limitations

    The paper uses PL/SQL to include Oracle PL/SQL, PostgreSQL PL/pgSQL, and related procedural SQL dialects, and some interactive requirements require mismatches detectable from the database environment.

Abstract

from arXiv · show

Large language models (LLMs) have shown strong potential for translating natural-language (NL) requirements into PL/SQL programs, attracting increasing attention from the database community. However, existing NL-to-PL/SQL efforts primarily focus on directly generating PL/SQL from complete NL requirements. In practice, PL/SQL development involves diverse scenarios, such as from-scratch development, code modification, debugging, and optimization, and may require either direct generation or multi-turn interaction. Yet, no comprehensive benchmark evaluates multi-scenario, direct and interactive, and multi-dialect NL-to-PL/SQL development. In this paper, we present ProcArena, an execution-based benchmark covering both Direct and Interactive modes. ProcArena comprises 3,998 executable tasks over 157 databases, spanning nine development subscenarios in PostgreSQL and Oracle. We construct challenging Direct tasks through Iterative Logic Enhancement and scenario-specific adapters, and derive paired Interactive tasks through Knowledge Integration and Requirement Perturbation while preserving executable targets. We further design a controlled Solver-User Simulator protocol that allows models to clarify user intent and inspect the database environment without exposing hidden execution feedback. Evaluating seven language models, we find that the best average scores are only 62.2% and 57.8% in Direct and Interactive, respectively, demonstrating that realistic NL-to-PL/SQL development remains challenging, particularly in interactive settings.

1 INTRODUCTION

ProcArena addresses the lack of a benchmark covering diverse NL-to-PL/SQL development scenarios, dialects, and both direct and interactive interaction. It constructs executable tasks and evaluates seven models, whose best average scores remain 62.2% in Direct and 57.8% in Interactive.

  • Motivation: Existing NL-to-PL/SQL benchmarks mainly generate code from complete requirements and do not cover diverse development scenarios or realistic interactions.Prior NL-to-SQL benchmarks address declarative SQL rather than procedural PL/SQL development.
  • Benchmark scope: ProcArena evaluates NL-to-PL/SQL across multiple development scenarios in Direct and Interactive modes for PostgreSQL and Oracle.The benchmark is presented as the first of its kind to combine these dimensions.
  • Construction: Iterative logic enhancement adds procedural logic while preserving executability and semantic consistency before scenario-specific adapters construct Direct tasks.Adapters rewrite requirements and prepare supporting artifacts while preserving the gold PL/SQL code.
  • Construction: Knowledge integration and requirement perturbation derive paired Interactive tasks, while the Solver–User Simulator protocol coordinates user and database-environment interactions.The protocol is designed to support realistic multi-turn development without exposing the offline evaluator.
  • Benchmark scope: 3,998 executable tasks cover nine development scenarios and seven evaluated models.The evaluation also analyzes generation across subscenarios, construction components, and the User Simulator.
  • Results: 62.2% and 57.8% are the best average scores in Direct and Interactive modes, respectively.These results indicate that the benchmark remains challenging even for the evaluated models.

2 PROBLEM FORMULATION

ProcArena formalizes NL-to-PL/SQL as producing executable PL/SQL from requirements, database information, and optional attachments. Direct tasks submit once, whereas Interactive tasks permit Solver-initiated exchanges with users and the database environment before submission.

  • 2 PROBLEM FORMULATION: An NL-to-PL/SQL task provides a natural-language requirement, database description, and optional task attachment to a Solver.The database description contains schema and metadata.
  • 2 PROBLEM FORMULATION: The Solver produces a candidate PL/SQL code from the task input, using an LLM and an action space.The formulation represents the candidate within the space of PL/SQL codes.
  • 2.1 Direct NL-to-PL/SQL: Direct tasks provide the requirement, database description, and attachment, then allow only one code submission.The Direct Solver action space is limited to submit.
  • 2.2 Interactive NL-to-PL/SQL: Interactive tasks model a multi-turn collaboration among a Solver, User Simulator, and database environment.The Solver receives the initial task input from the User Simulator and may continue interacting before submission.
  • 2.2 Interactive NL-to-PL/SQL: Interactive requirements may be underspecified, while paired tasks preserve the database description and attachment from the Direct task.Interactive construction changes the requirement view and keeps the other listed task elements unchanged.
  • 2.2 Interactive NL-to-PL/SQL: The Solver can clarify information through the User Simulator and inspect the database through the environment.These interaction targets have separate action spaces.
  • 2.2 Interactive NL-to-PL/SQL: The Solver initiates each interaction turn, selecting actions from the task input and accumulated interaction history.Nonterminal actions return responses that are appended to the history.
  • 2.2 Interactive NL-to-PL/SQL: After zero or more nonterminal turns, the LLM generates and submits a candidate PL/SQL code, terminating the task.Submission follows generation based on the initial input and interaction history.

3 BENCHMARK CONSTRUCTION

ProcArena is constructed by increasing procedural complexity, adapting tasks to nine development scenarios, and transforming Direct tasks into Interactive tasks that require Solver–user interaction. The pipeline is designed to preserve semantic consistency and executable targets.

  • Pipeline: The construction pipeline first seeds PL/SQL codes, enhances their procedural logic, and rewrites NL descriptions to remain semantically consistent.This establishes the enhanced NL-to-PL/SQL pairs used by later task construction stages.
  • Direct construction: Scenario adapters transform enhanced pairs into Direct tasks spanning nine development scenarios.The adapters rewrite requirements and prepare scenario-specific task materials.
  • Interactive construction: Knowledge injection and requirement perturbation transform Direct tasks into Interactive tasks that require Solver–user interaction for correct completion.The construction pipeline derives interactive tasks from the direct benchmark instances.

3.1 Seed Task Construction

ProcArena builds validated seed tasks by iteratively adding business rules to existing NL-to-PL/SQL pairs while updating code and requirements together. The final pairs are execution-validated and equipped with call suites for later task construction.

  • Each enhancement round proposes a business rule, implements it in PL/SQL, integrates the snippet, and updates the requirement to describe the resulting code.
  • After N rounds, ProcArena executes the final code and uses a held-out validation LLM to check the constructed pair.
  • Seed tasks combine an NL requirement, PL/SQL code, database description, initial state, construction model, enhancement rounds, and repair budget.
  • The validated pair and execution assets become the seed task for subsequent Direct and Interactive construction.

3.2 Direct Task Construction

Direct tasks represent nine PL/SQL development subscenarios spanning synthesis and editing, constructed from validated seeds with scenario-specific requirements and attachments. Adapters preserve the executable target while applying validity, non-triviality, alignment, and scenario-specific checks.

  • Scenario coverage: User-request analysis identifies five Synthesis and four Editing subscenarios for Direct PL/SQL development.Synthesis accounts for 48.7% of analyzed posts, while Editing accounts for 38.4%.
  • Synthesis: Synthesis ranges from from-scratch generation to document-, example-, template-, and component-based code construction.
  • Editing: Editing covers functional modification, performance optimization, code repair, and code completion.
  • Adapter construction: Scenario adapters rewrite requirements and add documents, examples, templates, reusable components, or existing code while preserving the validated seed behavior.
  • Validation: Candidates must pass artifact-validity, non-triviality, reference-alignment, and scenario-specific checks before materialization.Reference alignment is verified through execution against the reference code.
  • Direct evaluation: Direct solvers receive only the scenario-specific requirement, database description, and attachment, with gold code, calls, and initial state hidden.The Solver submits once and cannot use interaction or environment tools before submission.

3.3 Interactive Task Construction

Interactive tasks derive from Direct tasks by making requirements realistically underspecified through knowledge-based rewrites, omissions, and contradictions. A controlled protocol lets Solvers query users or inspect the environment while preserving executable targets and withholding execution feedback.

  • Requirement construction: Interactive construction applies Knowledge Injection and Requirement Perturbation to model implicit knowledge, omitted information, and conflicting constraints.
  • Task preservation: Both perturbation stages modify only the requirement, preserving attachments, schema, reference code, call suite, and initial state; B2 remains Direct-only.
  • Knowledge Injection: Knowledge Injection replaces explicit statements with domain terminology, logical constraints, or commonsense conventions while preserving the scenario contract and reference code.
  • Requirement Perturbation: Information Omission removes necessary but recoverable details whose absence can permit behavior different from the reference code.
  • Requirement Perturbation: Contradiction Injection creates either incompatible requirement statements or conflicts between the requirement and database environment, requiring user interaction for intended resolution.
  • Interaction protocol: The protocol offers environment inspection, ask_user, ask_selector, and submit actions, with every environment or user-facing action consuming budget.Submit terminates the episode without execution-accuracy feedback, preventing refinement against the hidden oracle.

4 PROCARENA STATISTICS

ProcArena provides broad, executable coverage across modes, dialects, scenarios, databases, and procedure complexity. Its 3,998 tasks include paired Interactive episodes for nearly all applicable Direct tasks, with requirements transformed 5.75 times per Interactive task on average.

  • Corpus scope: 3,998 tasks span PostgreSQL and Oracle, nine Direct subscenarios, and eight paired Interactive subscenarios.The benchmark covers synthesis and editing across complete and underspecified requirements.
  • Scale and coverage: ProcArena contains 1,879 Interactive episodes across 157 databases, compared with prior test splits of 209–300 Direct tasks in one dialect and scenario.
  • Difficulty: ProcArena procedures average 32.1 statements and 18.6 cyclomatic complexity, exceeding the reported averages for PLForge Hard and WeBridge.
  • Dialect distribution: The corpus includes 1,060 PostgreSQL and 1,059 Oracle Direct tasks, plus 940 PostgreSQL and 939 Oracle Interactive episodes.B2 has no Interactive task because it uses runtime scoring after semantic-equivalence checking.
  • Reference procedures: Reference procedures average 39.0 lines, 4.0 parameters, 6.3 calls, and 4.9 tables written across the corpus.
  • Interactive transformations: Interactive requirements contain 5.75 transformations per task on average, covering Knowledge, Contradiction, and Omission transformations.The pipeline changes only requirements; B3 has zero Knowledge transformations.

5 EXPERIMENTS

ProcArena evaluates seven language models across Direct and Interactive PL/SQL development, revealing modest accuracy, mode-dependent difficulty, optimization trade-offs, and distinct interaction and failure patterns.

  • 5.2 Overall Results: 62.2% is the strongest pooled Direct execution accuracy, while the strongest Interactive accuracy is 57.8%.Gemini-3.1 Pro leads both modes; GPT-5.5 follows at 61.6% and 57.0%.
  • 5.2 Overall Results: A5 Orchestration is hardest for every model, with no Direct score above 49.8% or Interactive score above 36.7%.The task requires composing existing procedures into a transactional workflow while respecting their signatures.
  • 5.2 Overall Results: Interactive difficulty differs by category: Gemini-3.1 Pro falls from 59.8% to 48.1% on Synthesis but rises from 66.2% to 73.5% on Editing.A2 and B3 also improve interactively because database grounding can outweigh missing requirement information.
  • 5.4 Interaction Analysis: 98.2% is GLM-5.3’s highest retention, whereas DeepSeek V4 Pro retains 64.5%; token and turn costs do not determine retention.GPT-5.5 retains 92.6% using 38.5k tokens and 7.1 turns, while GLM-5.2 uses 123.8k tokens over 19.8 turns and retains 80.2%.
  • 5.5 Error Analysis: Control flow is the largest Interactive failure category for six models at 17.6%–31.2%, followed by State effect at 7.9%–15.4%.DeepSeek V4 Pro instead shows unusually high Schema and Contract rates of 20.1% and 27.9%.

6 RELATED WORK

Related work covers NL-to-PL/SQL, Text-to-SQL, and code language models, while ProcArena broadens evaluation through scenario, interaction, and dialect coverage rather than proposing a new specialized model.

  • NL-to-PL/SQL: PLForge introduced NL-to-PL/SQL datasets, models, and execution-based evaluation but primarily studied direct generation from complete requirements.ProcArena instead broadens evaluation across multiple development scenarios and interaction modes.
  • NL-to-PL/SQL: ProcArena and PLForge focus on stored procedures, leaving UDFs and triggers outside their current scope.ProcArena expands evaluation along other dimensions, including scenario and interaction coverage.
  • Text-to-SQL: Text-to-SQL research includes schema-aware parsers, intermediate representations, constrained decoding, domain training, and structural constraints.These approaches target SQL generation and controllability through model or workflow design.
  • Text-to-SQL: Agentic Text-to-SQL workflows use planning, feedback, multi-agent collaboration, and user-in-the-loop clarification, while BIRD-INTERACT evaluates dynamic stateful multi-turn SQL tasks.The cited work remains focused on SQL statements rather than dialogue-based alignment for PL/SQL development.
  • Code Language Models: General-purpose code language models achieve broad coverage from massive polyglot data but can underperform on specialized low-resource languages such as PL/SQL.ProcArena complements model-specialization research rather than introducing a new specialization method.

7 CONCLUSION

ProcArena provides an executable, multi-scenario benchmark for evaluating NL-to-PL/SQL development in Direct and Interactive modes. Across seven models, performance remained limited, with failures concentrated in procedural capabilities absent from declarative SQL.

  • 3,998 executable tasks span nine Direct subscenarios and eight paired Interactive subscenarios across two PL/SQL dialects.
  • Iterative Logic Enhancement, live validation, and scenario-specific adapters construct Direct tasks, while Knowledge Integration and Requirement Perturbation derive paired Interactive tasks.
  • 62.2% execution accuracy in Direct and 57.8% in Interactive were the strongest results across seven evaluated models.
  • Failures concentrated in control flow and persistent state, error types that declarative SQL does not have.
  • The released benchmark, interaction traces, and construction pipeline support reproducible evaluation.
Loading 2609.06527v1…