PLSQLBench: Benchmarking LLM Systems for Executable Procedural Database Programming

1 Introduction

• General-purpose code-generation, text-to-SQL, and interactive database benchmarks leave executable procedural-program generation, modification, debugging, and repair underexplored.Source: §1 · p. 1
• The benchmark workflow tests generation, modification, debugging, validation, and repair of executable PL/SQL programs while preserving conversational context.Source: Figure 1 · p. 2, §1 · p. 2
• PLSQLBENCH evaluates executable PL/SQL programs across enterprise-style Spider 2 tasks, Spider-derived schema-grounded tasks, and MBPP-derived procedural problems in single- and multi-turn settings.The tasks span varying levels of database grounding and procedural complexity, unlike benchmarks focused on standalone declarative queries.Source: §1 · pp. 1–2
• Its execution-based protocol combines strict and partial-credit metrics for single-turn tasks with dynamic multi-turn evaluation under accumulated conversation context.Source: §1 · p. 2
• Eight proprietary and open-weight LLMs, including tool-augmented database agents, show persistent challenges in schema use, PL/SQL fidelity, procedural reasoning, exception handling, and cross-turn consistency.The introduction reports these challenges without giving quantitative results.Source: §1 · p. 2

Code-generation benchmarks

• General code-generation benchmarks test program synthesis and execution, but not database programs dependent on schemas, persistent objects, transactions, and procedural runtime behavior.The surveyed benchmarks range from short-program tasks to multilingual, contamination-resistant, library-use, and repository-level evaluation.Source: §2 · p. 2

Text-to-SQL and database interaction

• Text-to-SQL benchmarks expanded to enterprise schemas, workflows, conversation, and debugging, but remain SQL-centered; PLSQLBench evaluates executable PL/SQL combining procedural logic with schema-grounded SQL.The related work describes Spider, BIRD, BEAVER, Spider 2.0, SParC, CoSQL, BIRD-INTERACT, and BIRD-CRITIC as examples of these settings.Source: §2 · pp. 2–3

Enterprise workflow benchmarks

• Enterprise workflow benchmarks capture interaction and information access, but not executable procedural database programming; PLSQLBench evaluates read-only PL/SQL in single- and multi-turn settings.Its tasks cover complex enterprise-style schemas, simpler schema-grounded database tasks, and MBPP-derived procedural problems across varying grounding and procedural complexity.Source: §2 · p. 3

3.1 Benchmark Composition

• PLSQLBENCH combines five subsets from Spider 2.0 Lite, Spider 1.0, and MBPP/MBPP+, spanning enterprise-style schema-grounded tasks, simpler Spider tasks, and procedural programming problems.Spider2-ST and Spider2-MT use Oracle-normalized Spider 2.0 Lite schemas; Spider-PLSQL uses Spider 1.0 databases, and the MBPP subsets adapt programming problems to PL/SQL.Source: §3.1 · p. 3
• 2,865 instances comprise 2,594 single-turn tasks and 271 multi-turn conversations spanning 978 turns.Tasks avoid persistent inserts, updates, or deletes to support safe, repeatable execution.Source: §3.1 · p. 3

3.2 Source Normalization and Schema Setup

• Schema-grounded prompts provide table definitions, column names, types, and other task-referenced objects, while source adaptation differs across MBPP, Spider 1.0, and Spider 2.0 Lite.MBPP-derived tasks use PL/SQL functions or procedures without persistent tables, with Oracle types supplied when needed; Spider 1.0 retains its questions and Oracle schemas, and Spider 2.0 Lite tasks are curated over Oracle-dialect schemas.Source: §3.2 · pp. 3–4

3.3 Spider 2.0 Question Generation and Prompt Construction

• Spider2-ST and Spider2-MT prompts cover PL/SQL code generation, schema-grounded code repair, and interactive development across turns.Interactive requests extend, modify, debug, or repair earlier code while preserving prior context.Source: §3.3 · p. 4

3.4 Execution-Based Evaluation

• Spider2-ST and Spider2-MT evaluation executes tests against reference and generated answers, comparing generated outputs with captured reference outputs.For multi-turn tasks, reference and generated answers are run in turn order in separate clean states, preserving each sequence’s prior-turn state without mixing answers.Source: §3.4 · p. 4
• MBPP tests are translated into self-contained PL/SQL blocks, while Spider-PLSQL uses Spider 1.0 gold-SQL result sets as expected outputs instead of reference PL/SQL.MBPP evaluation runs translated tests in Oracle without executing Python code or canonical solutions; Spider-PLSQL compares emitted rows with gold results using a bounded set of column permutations.Source: §3.4 · p. 4

3.5 Quality Control

• Spider2-ST and Spider2-MT undergo multi-stage checks for clarity, schema consistency, executable answers, valid tests, metadata, and required PL/SQL features.Multi-turn examples are also checked for preserved prior intent and realistic incremental changes; ambiguous or invalid examples are revised or removed.Source: §3.5 · p. 4

4.1 Experimental Setting

• The evaluation covers eight proprietary and open-weight models plus two Codex CLI database agents using GPT-5.4-Mini or GPT-5.6-Sol backbones.Spider2-ST and Spider2-MT use database-disjoint development and held-out test splits; MBPP+-PLSQL’s translated PL/SQL tests are withheld for evaluation.Source: §4.1 · p. 5

4.2 Evaluation Metrics

• Generated PL/SQL is evaluated with execution-based unit tests using Mean Test Pass@1 for partial credit and Suite Pass@1 for all-tests-pass correctness.For multi-turn tasks, prior requests and generated responses are carried into later turns; Turn Suite Pass@1 scores each turn, while Episode Pass@1 requires every turn to succeed.Source: §4.2 · p. 5

4.3 Experimental Results

• GPT-5.4 leads overall at 64.96% test mean Pass@1, followed by Claude-Opus-4.8 at 63.49%; Gemma-4-31B is the strongest open-weight model at 62.26%.GPT-5.6-Sol leads on MBPP, Claude-Opus-4.8 on Spider-PLSQL, and GPT-5.4 on both Spider2 variants; no model dominates across task families, and the best remains below 65%.Source: §4.3 · p. 5
• Codex DB Agent improves every Spider2 result: Mean Test Pass@1 rises 6.9–9.8 points for GPT-5.4-Mini and 12.0–14.5 points for GPT-5.6-Sol.Strict correctness also improves: Spider2-ST Suite Pass@1 rises 1.9 and 9.7 points, while Spider2-MT Episode Pass@1 rises 4.8 and 19.1 points; MBPP+ changes by +6.2 and −0.7 points, respectively.Source: §4.3 · pp. 5–6

4.4 Error Analysis

• Procedures with incorrect logic or output form the largest failure class, accounting for 11,015 failures (82.5%), including 9,953 output mismatches and 1,062 runtime failures.Common errors include incorrect filters, lookups, joins, ordering, aggregation, formatting, or edge-case handling.Source: §4.4 · p. 6
• Invalid or incomplete PL/SQL artifacts account for 1,777 failures (13.3%), including 943 syntax or parser failures and 834 invalid compiled objects.Examples include incomplete procedure bodies, unclosed blocks, incorrect declarations, and non-executable explanatory text.Source: §4.4 · p. 6
• Interface, signature, or schema-grounding mismatches account for 566 failures (4.2%); Spider2-MT contributes 525 (92.8%) of these.Errors include missing entry points or package members, wrong parameter signatures, and invalid object or column references.Source: §4.4 · p. 6

5 Conclusion

• 64.96% was the best overall Mean Test Pass@1, with substantial drops under stricter metrics, showing that procedural database programming remains challenging.The conclusion calls for more reliable reasoning over executable database semantics, dialect constraints, and iterative database development.Source: §5 · p. 6
• PLSQLBench is presented as the first benchmark specifically designed to evaluate LLM systems on executable PL/SQL across varying database-grounding and procedural-complexity levels.It combines schema-grounded and procedural tasks, execution-based tests, and single- and multi-turn workflows—capabilities not directly assessed by conventional code-generation or text-to-SQL benchmarks.Source: §5 · p. 6
• Error analysis identifies persistent failures in procedure semantics, PL/SQL artifact construction, and interface or schema grounding.These failures motivate more reliable reasoning over executable database semantics, dialect constraints, and iterative database development.Source: §5 · p. 6

• PLSQLBench covers only a subset of production database development, omitting larger schemas, legacy dependencies, performance constraints, permission boundaries, and deployment requirements.Source: Limitations · p. 7
• PLSQLBench supports only read-only PL/SQL, preserving reproducible evaluation on a fixed database state while excluding write workflows that are difficult to isolate amid DML, DDL, commits, rollbacks, and session side effects.Source: Limitations · p. 7
• Passing all execution-based tests does not guarantee program correctness under all inputs because correctness is bounded by test coverage.Source: Limitations · p. 7
• PLSQLBench is intended for test-only evaluation without a large benchmark-specific training set, so results measure generalization to its schemas, task formats, and tests rather than adaptation through benchmark-specific tuning.Source: Limitations · p. 7

Ethical Considerations

• PLSQLBench evaluates model reliability rather than promoting unsupervised production use, since incorrect PL/SQL can produce wrong reports, break interfaces, or create unsafe operational behavior.Evaluation uses isolated benchmark schemas, database-backed tests, and read-only tasks that avoid persistent data modifications.Source: Ethical Considerations · p. 7
• The benchmark excludes private customer and production data; human review, schema checks, compilation and execution validation, and test-based quality control mitigate risks from LLM-assisted task creation.Results are intended to guide safer AI-assisted development, with human review, access controls, and production testing still necessary for real deployments.Source: Ethical Considerations · p. 7

A Direct Generation vs. Tool-Augmented Agent

• Table 3 reports Mean Test Pass@1 (%) for direct generation versus Codex DB Agent with medium reasoning, including MBPP+ test-split and Spider2 single- and multi-turn results.The caption says bold marks the better setting for each backbone; it does not provide the underlying values in the supplied passage.Source: Table 3 · p. 9

B.1 Metadata-Guided Curation

• Task metadata specifies each example’s source, database, target PL/SQL constructs, reasoning types, difficulty, and turn count, guiding prompt, reference, test, and annotation creation.Reference answers must match the request, compile and run under Oracle-dialect DDL/DML, use required constructs, and stay consistent across turns; metadata fields are checked for accuracy and completeness.Source: §B.1 · p. 9

B.2 Spider 2 Normalization to Oracle Dialect

• Spider 2.0 Lite databases from SQLite, Snowflake, and BigQuery were converted into Oracle-compatible schema and data artifacts; BigQuery GEOGRAPHY coordinate tuples were replaced with NULL.This enables consistent Oracle instantiation for executable PL/SQL generation, repair, and interactive development tasks.Source: §B.2 · pp. 9–10
• Identifier normalization truncates names beyond Oracle’s 128-character limit, replaces special characters with underscores, and prefixes names beginning with digits or underscores or conflicting with reserved words.Alphabetic prefixes such as C_ for columns and S_ for schemas produce valid identifiers.Source: §B.2 · pp. 9–10
• Source-specific types such as INT, BIGINT, FLOAT, and DATETIME were mapped to Oracle-compatible types including NUMBER, TIMESTAMP, VARCHAR2, CLOB, BLOB, CHAR(1), and DATE.The mapping also uses NUMBER(38).Source: §B.2 · p. 10
• Literal normalization converts date and timestamp formats, rewrites MySQL quote escaping and NULL values, chunks oversized strings into TO_CLOB concatenations, and replaces invalid numeric values with NULL.Oracle DATE, TIMESTAMP, TO_DATE, and TO_TIMESTAMP forms are used for date and timestamp literals.Source: §B.2 · p. 10

B.3 Task Curation Methodology

• Assigned metadata controls each task’s source, database, difficulty, turn count, required PL/SQL constructs, and reasoning types.Constructs include functions, procedures, packages, cursors, exception handling, object types, and dynamic SQL; reasoning includes retrieval, validation, debugging, control flow, aggregation, and state-aware logic.Source: §B.3 · p. 10
• Annotators write prompts for realistic PL/SQL workflows, aligning schema-grounded tasks with relevant tables and columns and pairing MBPP tasks with PL/SQL-compatible inputs, outputs, and Oracle types.Tasks can require data-processing logic but do not require persistent database-state modifications.Source: §B.3 · p. 10
• The benchmark spans code generation, code repair, and multi-turn interactive development, where later requests can modify requirements or request debugging while requiring earlier context to be preserved.Repair tasks must preserve intended behavior; interactive turns may add requirements, change business logic, provide debugging feedback, or request repairs.Source: §B.3 · p. 10
• Difficulty reflects schema complexity, PL/SQL constructs, reasoning demands, and interaction structure, with multi-turn tasks additionally assessed for state tracking, consistent naming, incremental modification, and repair.Advanced tasks may combine multiple tables, complex business logic, object types, dynamic SQL, packages, exception handling, debugging, or cross-object requirements.Source: §B.3 · p. 10
• Each task has a reference solution and executable tests that must match the prompt, compile and run in Oracle DB, and execute deterministically without modification.For multi-turn tasks, solutions and tests are updated consistently so later requirements do not contradict or silently break earlier behavior.Source: §B.3 · pp. 10–11
• Curation tracks source, database, required tables and columns, reasoning types, constructs, difficulty, and turn count to balance benchmark coverage and enable fine-grained error analysis..Source: §B.3 · p. 11

B.4 Spider 1.0 Adaptation

• Spider-PLSQL retains Spider 1.0’s original questions and uses Oracle SQL result sets as execution references, requiring PL/SQL output as one comma-separated row per line via DBMS_OUTPUT.PUT_LINE.The adaptation uses the Oracle-converted release of Spider 1.0 (Oracle Corporation, 2025).Source: §B.4 · p. 11

B.5 MBPP and MBPP+ Python-to-PL/SQL Conversion

• MBPP comprises 974 short, unit-tested Python assignments on programming fundamentals, while MBPP+ expands tests for a subset of tasks (Austin et al., 2021; Liu et al., 2024).The expanded tests address the risk that small test suites miss incorrect implementations.Source: §B.5 · p. 11
• The rule-based converter extracted literal test inputs and expected outputs, inferred consistent types, and generated PL/SQL function scaffolds and unit tests.It rejected non-literal or unsafe test expressions and tasks whose argument types could not be inferred consistently; converted functions need no persistent database tables and may use Oracle object or collection types.Source: §B.5 · pp. 11–12
• 806 of 823 candidate MBPP records and 308 of 323 candidate MBPP+ records passed final Oracle-backed replay validation.Replay validation executed generated PL/SQL tests against replay functions that map converted inputs to expected outputs, checking harness executability and whether each retained record had a function capable of passing its tests.Source: §B.5 · p. 12
• Providing progressively more information about the intended Python computation substantially increased MBPP+ passing rates; sampled hinted generations showed only one apparent shortcut case.The comparison is presented as a diagnostic lower-bound signal. In 50 manually reviewed records, hinted generations otherwise appeared to implement the requested functions rather than replay tests; hinted and no-hint function lengths were similar.Source: §B.5 · p. 13, Table 5 · p. 13
• PL/SQL outputs matched Python-reference checks for 96.8% of output-comparable MBPP records and 89.7% of MBPP+ records.Pass/fail agreement was 96.7% for MBPP and 91.6% for MBPP+, rising to 98.7% and 96.6%, respectively, for conclusive comparisons. The authors treat these heuristic cross-language comparisons as a sanity check, not proof of semantic equivalence, and conclude that the checks support the reliability of the retained benchmark tasks.Source: §B.5 · p. 13

B.6 Spider 2.0 Quality Control Pipeline

• Spider 2.0 quality control combines automatic checks, human review, post-processing, and LLM-as-judge review.Checks cover formatting, metadata, prompt-reference alignment, schema references, compilation, execution, test validity, semantic correctness, PL/SQL quality, construct labels, and table or column claims.Source: §B.6 · p. 13

B.7 Quality Dimensions

• Table 4 summarizes the primary quality dimensions used during review.Source: §B.7 · p. 13

C Metric Definitions

• Each task or turn uses one generated PL/SQL program; compilation, runtime, or required-object creation failures count as zero tests passed.Scores first award partial credit within each task or episode, then average across tasks or episodes.Source: C Metric Definitions · pp. 13–14
• Single-turn reporting distinguishes normalized Mean Test Pass@1 from strict suite-level success, which requires passing every test in a task.Mean Test Pass@1 gives tasks equal weight after normalizing each by its test count.Source: C Metric Definitions · p. 14
• Multi-turn reporting separates Episode Pass, requiring every turn solved, Turn Suite Pass, measuring the fraction of solved turns, and partial-credit Episode Test Pass.Each episode contributes equally to the final multi-turn score, including when episodes differ in test count.Source: C Metric Definitions · p. 14

D.1 Turn Distribution of Spider2-MT

• Spider2-MT has 271 conversations across 978 turns: 155 three-turn, 67 four-turn, and 49 five-turn conversations.The development split contains 208 conversations and 750 turns; test contains 63 conversations and 228 turns.Source: §D.1 · p. 15, Table 6 · p. 15

D.2 Reasoning Type and Difficulty Distribution on Spider2

• Spider2 difficulty labels reflect procedural and database reasoning, from simple retrieval or control flow to advanced packages, dynamic SQL, bulk processing, or coordinated cross-turn behavior.Intermediate tasks combine requirements such as joins, aggregation, cursor iteration, validation, or structured exception handling; Table 7 presents difficulty distributions by Spider2-ST and Spider2-MT dev/test split.Source: §D.2 · p. 15, Table 7 · p. 15
• The reasoning-type distribution reports the top ten types for Spider2-ST and Spider2-MT dev/test splits, ranked by Spider2-MT development frequency.Counts are multi-label, and Spider2-MT counts are over turns.Source: Table 8 · p. 15

D.3 Representative Benchmark Instances

• Spider2-ST examples require schema-grounded functions with specified interfaces, case-insensitive matching, null-input behavior, and explicit exception handling.The example function accepts a city name and returns population text.Source: §D.3 · p. 16
• MBPP-PLSQL converts Python problems into PL/SQL function tasks that use supplied collection types and unit tests rather than persistent database tables.The example asks for a minimum-cost path function based on a cost matrix.Source: §D.3 · p. 16
• Spider2-MT examples test three-turn procedure revision, requiring later turns to preserve prior behavior while adding threshold logic, a join, and object-oriented formatting.The sequence modifies a shared stored procedure interface and lookup behavior across turns.Source: §D.3 · pp. 17–19

E Additional Evaluation Results

• Appendix E adds Single-Turn Suite Pass@1 alongside the main paper’s Mean Test Pass@1; Table 9 reports single-turn suite results, with overall means across two test sets.Codex Agent rows use medium reasoning, and unavailable evaluations are shown as –.Source: E Additional Evaluation Results · p. 20, Table 9 · p. 20

E.1 Multi-Turn Strict Metrics

• Table 10 reports Spider2-MT Episode Pass@1, requiring every conversation turn to pass all required unit tests, and Turn Suite Pass@1, measuring suite-level success at the individual turn level.Source: §E.1 · p. 20, Table 10 · p. 20

E.2 Error Analysis Protocol

• Error analysis treats each single-turn instance and each multi-turn turn as a unit, excludes reference-solution or harness failures, and classifies remaining executions as passes or model failures.Failure categories are wrong logic or output, invalid or incomplete PL/SQL artifacts, and interface, signature, or schema-grounding errors.Source: §E.2 · p. 21
• Oracle diagnostics and expected-versus-actual outputs support category assignment, including compilation, missing-entry-point, argument-signature, and schema-identifier errors.Output mismatches without Oracle exceptions are inspected through expected and actual outputs.Source: §E.2 · p. 21
• For multi-turn data, failures are analyzed by turn while retaining conversation identifiers; aggregate counts are reported by category, dataset, and model, followed by manual review of representative failures.Review avoids cases driven mainly by brittle exact-string evaluation, such as harmless date-format differences.Source: §E.2 · p. 21

E.3 Representative Error Examples

• Figures 5–7 use compact execution evidence to show how failures are identified from generated artifacts and unit-test outcomes.Source: §E.3 · p. 21
• On Spider2-MT instance plsql-multiturn-0053_q01, GPT-5.4 passed 0/3 tests because it emitted only the exception message before required report headers.The reference prints report headers before iterating, so empty-result output should include both report context and the exception message.Source: §E.3 · p. 21, Figure 5 · p. 22
• Figure 6 shows an invalid-artifact failure: the model returns INVALID_REQUEST for a solvable query, stopping execution before semantic comparison.Source: Figure 6 · p. 22
• Figure 7 shows an interface failure: the generated artifact creates only a schema-qualified package body, leaving the required public package procedure uncallable.Source: Figure 7 · p. 22

F.1 Prompt for PL/SQL Code Generation

• Figure 8 presents the prompt for PL/SQL code generation.Source: Figure 8 · p. 23

G Hyperparameters and Infrastructure

• The paper reports model, prompting, inference, and infrastructure settings for reproducibility in Table 11.Source: G Hyperparameters and Infrastructure · p. 24, Table 11 · p. 24