PaperlayerView exact revision on arXiv

Source-linked AI summary

PLSQLBench: Benchmarking LLM Systems for Executable Procedural Database Programming

Marianne Menglin Liu, Leonid Boytsov, Daniel W. Peterson, Pramuditha Perera, Rongguang Wang, Sai Ashish Somayajula, Syed Hamza Rafique, Rohit Saini, Shubham Pathak, Sujeeth Bharadwaj, Tao Sheng, Graham Horwood, Fahad Shah, Ankan Bansal, Sujith Ravi, Dan Roth

arXiv:2608.15931v1cs.CL

TL;DR

Existing benchmarks focus on general-purpose programming or declarative SQL, leaving executable procedural database programming underexplored. PLSQLBench evaluates this capability through executable PL/SQL tasks and finds persistent challenges, with the best overall Mean Test Pass@1 reaching 64.96%.

  • Problem

    Existing AI benchmarks emphasize general-purpose programming and standalone declarative SQL rather than executable procedural database programming.

  • Method

    PLSQLBench combines schema-grounded and procedural PL/SQL tasks across varying database grounding and procedural complexity, evaluated with execution-based tests in single- and multi-turn settings.

  • Results

    The best overall Mean Test Pass@1 is 64.96%, while error analysis reveals persistent failures in procedure semantics, PL/SQL artifact construction, and interface or schema grounding.

  • Takeaways & Limitations

    Procedural database programming remains challenging and involves capabilities not directly assessed by conventional code-generation or text-to-SQL benchmarks.

  • Takeaways & Limitations

    PLSQLBench covers only a subset of real-world database development, omitting larger schemas, legacy dependencies, performance constraints, permission boundaries, and deployment requirements.

Abstract

from arXiv · show

We present PLSQLBench, to our knowledge the first benchmark for evaluating whether LLMs can write executable PL/SQL programs, with correctness measured through execution-based tests. Existing LLM evaluations largely target general-purpose code generation or declarative text-to-SQL, leaving procedural database programming underexplored. PLSQLBench contains 2,865 instances: 2,594 single-turn tasks and 271 multi-turn conversations spanning 978 turns. The benchmark combines complex schema-grounded tasks over enterprise-style Spider 2 databases, simpler schema-grounded tasks derived from Spider, and MBPP-derived procedural problems, covering varying levels of database grounding and procedural complexity. Experiments with eight LLMs reveal recurring difficulties in schema grounding, PL/SQL dialect fidelity, procedural control flow, exception handling, and cross-turn consistency. Tool-augmented LLM agents improve performance on several schema-grounded evaluations, although substantial gaps remain. These results highlight procedural database programming capabilities not directly assessed by conventional code generation or text-to-SQL benchmarks. Our code is available at https://github.com/oracle-samples/plsqlbench.

1 Introduction

PLSQLBENCH addresses a gap in AI evaluation by benchmarking executable procedural database programming in PL/SQL rather than only general-purpose code or standalone declarative SQL. It combines schema-grounded and procedural tasks with executable evaluation, and experiments reveal persistent challenges across database grounding, PL/SQL reasoning, and conversational consistency.

  • Motivation and benchmark: Existing benchmarks primarily assess general-purpose programming or standalone declarative SQL, leaving procedural database programming underexplored.General code-generation benchmarks use languages such as Python, Java, and C++, while text-to-SQL benchmarks focus on standalone queries.
  • Motivation and benchmark: PLSQLBENCH evaluates LLM systems on executable procedural database programming in PL/SQL, including stored procedures, functions, packages, cursors, and exceptions.The benchmark targets PL/SQL as a practically relevant testbed for AI evaluation.
  • Benchmark design: The benchmark uses executable tests to evaluate generation, modification, debugging, and repair of procedural programs while preserving conversational context.It includes both single-turn tasks and multi-turn interactive workflows.
  • Benchmark design: PLSQLBENCH combines enterprise-style Spider 2 tasks, Spider-derived schema-grounded tasks, and MBPP-derived procedural problems across database grounding and procedural complexity.This design contrasts with conventional text-to-SQL evaluations centered on standalone declarative queries.
  • Empirical findings: Eight proprietary and open-weight LLMs, including tool-augmented database agents, exhibit persistent challenges in schema use, PL/SQL dialect fidelity, procedural reasoning, exception handling, and cross-turn consistency.The evaluation protocol combines strict and partial-credit single-turn metrics with dynamic multi-turn evaluation under accumulated conversation context.

2 Related Work

Prior benchmarks evaluate general code generation, schema-grounded SQL, conversational database interaction, or enterprise workflows. PLSQLBench instead targets executable procedural database programming, addressing a gap left by these settings.

  • Code-generation benchmarks: General code-generation benchmarks such as HumanEval and MBPP use natural-language specifications and execution-based unit-test scoring, with MultiPL-E and MXEVAL extending evaluation across languages.These benchmarks assess short program synthesis rather than procedural database programming.
  • Text-to-SQL and database interaction: Text-to-SQL benchmarks progress toward larger databases, enterprise schemas, workflow tasks, conversational querying, execution feedback, and SQL debugging, but remain centered on SQL.Examples include Spider, BIRD, BEAVER, Spider 2.0, Spider2-DBT, SParC, CoSQL, BIRD-INTERACT, and BIRD-CRITIC.
  • Enterprise workflow benchmarks: Enterprise-oriented benchmarks model realistic agent settings, workplace tasks, simulated software companies, and heterogeneous artifact retrieval, but do not evaluate executable procedural database programming.PLSQLBench is positioned as addressing this gap.

3 Dataset Construction and Curation

PLSQLBENCH is curated from three source families into five subsets spanning schema-grounded and procedural PL/SQL tasks. Its construction combines normalization, prompt and reference curation, executable testing, and multi-stage quality control.

  • Dataset composition: Three source families form five subsets: Spider2-ST, Spider2-MT, Spider-PLSQL, MBPP-PLSQL, and MBPP+-PLSQL.Spider2 subsets use Oracle-normalized Spider 2.0 Lite schemas; Spider-PLSQL retains Spider questions; MBPP-derived subsets adapt programming problems and tests to PL/SQL.
  • Dataset composition: 2,865 instances comprise 2,594 single-turn tasks and 271 multi-turn conversations spanning 978 turns.Tasks avoid persistent inserts, updates, and deletes to support safe, repeatable execution.
  • Dataset composition: Spider2-ST contains 407 development and 103 test tasks, while Spider2-MT contains 208 development and 63 test conversations spanning 750 and 228 turns.Both are enterprise-style, schema-grounded subsets using Spider 2.0 Lite databases ported to Oracle.
  • Dataset composition: Spider-PLSQL contains 970 tasks, while MBPP-PLSQL and MBPP+-PLSQL contain 806 and 308 tasks, respectively.The MBPP-derived subsets adapt programming problems and Python unit tests into executable PL/SQL function-generation tasks, sometimes using Oracle object or collection types.
  • Evaluation and quality control: Evaluation uses executable reference outputs for Spider2 tasks, translated self-contained PL/SQL tests for MBPP-derived tasks, and gold-SQL result sets for Spider-PLSQL.Spider2-MT executes reference and generated answers in turn order using separate clean states.
  • Evaluation and quality control: Multi-stage quality control checks Spider2-ST and Spider2-MT examples for clarity, schema consistency, executability, annotation, test validity, and multi-turn intent preservation.Checks include prompt-answer alignment, table and column accuracy, PL/SQL syntax and execution correctness, metadata consistency, and feature coverage.

4 Experiments

Experiments evaluate eight LLMs and two tool-augmented database agents on database-disjoint splits using execution-based correctness metrics. Frontier models lead overall, agents improve Spider2 performance, and most failures arise from incorrect procedure behavior rather than invalid PL/SQL or interface mismatches.

  • Model and Hardware Setup: Eight proprietary and open-weight models, plus two Codex CLI database agents, are evaluated with Oracle SQL/PLSQL skills and database tools.The agents use GPT-5.4-Mini and GPT-5.6-Sol backbones.
  • Data Split: Evaluation uses database-disjoint development and private test splits for Spider2, while MBPP+-PLSQL and Spider2 test sets remain evaluation-only.Translated PL/SQL unit tests are withheld for evaluation, and all tasks from the same database stay within one split.
  • Evaluation Metrics: Execution-based unit tests measure Mean Test Pass@1 with partial credit and stricter Suite Pass@1 requiring every unit test to pass.Multi-turn tasks additionally report Turn Suite Pass@1 and Episode Pass@1, the latter requiring all conversation turns to succeed.
  • Main Results: 64.96% is GPT-5.4’s highest test mean Pass@1, followed by Claude-Opus-4.8 at 63.49%, while Gemma-4-31B leads open-weight models at 62.26%.Gemma-4-31B ranks third overall and outperforms GPT-5.6-Sol at 61.71%.
  • Tool-Augmented Agents: 6.9–9.8 points and 12.0–14.5 points are the Spider2 Mean Test Pass@1 gains from Codex DB Agent for GPT-5.4-Mini and GPT-5.6-Sol, respectively.Under stricter metrics, Spider2-ST Suite Pass@1 rises by 1.9 and 9.7 points, while Spider2-MT Episode Pass@1 rises by 4.8 and 19.1 points.
  • Failure Analysis: 11,015 failures (82.5%) involve wrong procedure logic or output, compared with 1,777 (13.3%) invalid or incomplete PL/SQL artifacts and 566 (4.2%) interface or grounding errors.The largest class includes 9,953 output mismatches and 1,062 runtime failures; Spider2-MT accounts for 525 of 566 interface errors (92.8%).

5 Conclusion

PLSQLBench is presented as the first benchmark specifically designed to evaluate executable procedural database programming in PL/SQL. It combines varied database grounding, procedural complexity, execution-based testing, and single- and multi-turn workflows to assess capabilities beyond conventional code-generation and text-to-SQL benchmarks.

  • PLSQLBench is presented as the first benchmark specifically designed to evaluate LLM systems on executable procedural database programming in PL/SQL.
  • The benchmark spans varying levels of database grounding and procedural complexity.
  • PLSQLBench combines schema-grounded and procedural tasks with execution-based tests and single-turn and multi-turn workflows.
  • The benchmark evaluates capabilities not directly assessed by conventional code-generation or text-to-SQL benchmarks.

Limitations

PLSQLBench captures only part of real-world database development and currently evaluates read-only PL/SQL under fixed-state, execution-based tests. Its results therefore measure generalization to benchmark schemas, formats, and test cases, while remaining bounded by test coverage.

  • PLSQLBench covers only a subset of real-world database development, omitting larger schemas, legacy dependencies, performance constraints, permission boundaries, and deployment requirements.
  • Read-only PL/SQL support omits write-oriented workflows but preserves reproducibility by running predictions against the same fixed database state.Updatable PL/SQL is challenging to isolate because DML may interact with DDL, commits, rollbacks, and session-level side effects.
  • Passing all execution-based tests does not guarantee correctness under all inputs because correctness is bounded by test coverage.
  • Without a large benchmark-specific training set, PLSQLBench is intended for test-only evaluation of model generalization rather than benchmark-specific tuning.

Ethical Considerations … B.2 Spider 2 Normalization to Oracle Dialect

PLSQLBench is designed to measure procedural database programming reliability while emphasizing that deployment requires safeguards and human oversight. Its Spider 2.0 Lite tasks are curated and normalized into executable Oracle-compatible environments through metadata-guided validation and dialect-specific transformations.

  • Ethical Considerations: PLSQLBench evaluates model limitations rather than endorsing unsupervised deployment of generated database code, which can cause incorrect reports, broken interfaces, or unsafe operations.The benchmark uses isolated evaluation environments, while real deployments require human review, access controls, and production testing.
  • A Direct Generation vs. Tool-Augmented Agent: Table 3 compares direct generation with the Codex DB Agent using Mean Test Pass@1 (%) across MBPP+ and Spider2 single- and multi-turn settings.MBPP+ uses its test split, and bold formatting identifies the better setting for each backbone.
  • B.1 Metadata-Guided Curation: 2,865? Each task is initialized with metadata covering its source dataset, target PL/SQL constructs, reasoning types, difficulty, and number of turns.Developers use these fields to construct prompts, reference PL/SQL, tests, and annotations.
  • B.2 Spider 2 Normalization to Oracle Dialect: Spider 2.0 Lite databases from SQLite, Snowflake, and BigQuery were converted into Oracle-compatible schema and data artifacts before task construction and execution.This addressed the heterogeneous SQL dialects of the source databases.
  • B.2 Spider 2 Normalization to Oracle Dialect: Identifiers were normalized to Oracle naming rules by truncating overlong names, replacing special characters, prefixing invalid starts, and avoiding reserved words.Names longer than Oracle’s 128-character limit were truncated; prefixes such as C_ and S_ were used for columns and schemas.
  • B.2 Spider 2 Normalization to Oracle Dialect: Source-specific types and literals were converted into Oracle-compatible representations to prevent schema-loading, execution, and numeric-conversion errors.Examples include mapping INT, BIGINT, FLOAT, and DATETIME to Oracle types, converting date formats with DATE or TO_DATE, and replacing Infinity, -Inf, and NaN with NULL.
  • B.2 Spider 2 Normalization to Oracle Dialect: BigQuery GEOGRAPHY coordinate tuples were replaced with NULL because standard Oracle numeric columns cannot load them without spatial-type conversion.This was an additional normalization step for Spider 2.0 Lite databases originating from BigQuery.
  • B.2 Spider 2 Normalization to Oracle Dialect: The normalization pipeline enables consistent Oracle DB instantiation for executable PL/SQL generation, code repair, and interactive development tasks.It applies across Spider 2.0 Lite schemas and data after dialect-specific conversion.

B.3 Task Curation Methodology · B.4 Spider 1.0 Adaptation

PLSQLBench curates metadata-controlled PL/SQL tasks spanning diverse procedural constructs, reasoning types, task families, difficulty levels, and interaction structures, with executable reference validation. Its Spider-PLSQL adaptation preserves Spider 1.0 questions and evaluates PL/SQL output against Oracle SQL result sets.

  • B.3 Task Curation Methodology: Metadata fields specify each task’s source dataset, target database, difficulty, turn count, required PL/SQL constructs, and reasoning types.Constructs include functions, procedures, packages, cursors, exception handling, object types, and dynamic SQL; reasoning types include retrieval, validation, debugging, control flow, aggregation, and state-aware logic.
  • B.3 Task Curation Methodology: Annotators create prompts satisfying assigned metadata while representing realistic workflows such as reporting, auditing, validation, debugging, and business-rule enforcement.Tasks may require data-processing reasoning without requiring persistent database-state modifications.
  • B.3 Task Curation Methodology: The benchmark includes code generation, code repair, and interactive development tasks, with interactive tasks presenting sequential user requests over multiple turns.Code repair requires preserving intended behavior while fixing incorrect, incomplete, or failing PL/SQL code.
  • B.3 Task Curation Methodology: Difficulty reflects schema complexity, PL/SQL construct coverage, reasoning requirements, and interaction structure, progressing from basic constructs to complex multi-table procedural logic.Advanced tasks may combine business logic, object types, dynamic SQL, package structure, exception handling, and debugging.
  • B.3 Task Curation Methodology: Each task includes a reference PL/SQL solution and deterministic executable tests that compile in Oracle DB, run against the task schema, and match the prompt exactly.For multi-turn tasks, references and tests are updated consistently so later requirements do not contradict or silently break earlier behavior.
  • B.3 Task Curation Methodology: Curation tracks datasets, databases, tables, columns, reasoning types, constructs, difficulty, and turn counts to balance coverage and support fine-grained error analysis.These annotations support coverage across schemas, task types, PL/SQL features, and reasoning categories.
  • B.4 Spider 1.0 Adaptation: Spider-PLSQL adapts Oracle-converted Spider 1.0 by retaining its natural-language questions and using Oracle SQL result sets as execution-accuracy references.Models format PL/SQL output for direct textual comparison with gold-SQL results, using DBMS_OUTPUT.PUT_LINE and one result row per line.

B.5 MBPP and MBPP+ Python-to-PL/SQL Conversion · B.6 Spider 2.0 Quality Control Pipeline · B.7 Quality Dimensions

The benchmark converts MBPP and MBPP+ programming tasks into executable PL/SQL evaluations, validates retained records through replay and Python-reference checks, and applies layered quality control to Spider 2.0 examples. These procedures provide a limited-database-grounded benchmark surface while reviewing formatting, semantics, execution, and other quality dimensions.

  • B.5 MBPP and MBPP+ Python-to-PL/SQL Conversion: 974 MBPP assignments provide short, unit-tested Python programming problems covering numeric, list, string, and standard-library fundamentals.Each assignment asks for a short Python function from a text description, with correctness verified by unit tests.
  • B.5 MBPP and MBPP+ Python-to-PL/SQL Conversion: MBPP and MBPP+ were translated into PL/SQL functions without persistent tables, preserving signatures, inputs, expected outputs, and required Oracle types.The conversion focuses on function interfaces and structured-input representations rather than database-table operations.
  • B.5 MBPP and MBPP+ Python-to-PL/SQL Conversion: The fully automatic rule-based converter parsed test-case ASTs, inferred consistent argument and return types, emitted function scaffolds, and generated equality-based PL/SQL unit tests.It rejected non-literal or unsafe tests and tasks whose argument types were inconsistent across unit tests.
  • B.5 MBPP and MBPP+ Python-to-PL/SQL Conversion: 806 out of 823 candidate MBPP records and 308 out of 323 candidate MBPP+ records passed Oracle-backed replay validation.The validation checked that generated PL/SQL test harnesses were executable and that each retained record had a function satisfying its emitted tests.
  • B.5 MBPP and MBPP+ Python-to-PL/SQL Conversion: Replay validation, Python–PL/SQL behavioral cross-checks, and hinted-generation analysis support the reliability of the retained tasks for evaluating PL/SQL coding with limited database grounding.The validation evidence collectively addresses translation correctness, harness executability, and possible shortcut behavior.
  • B.5 MBPP and MBPP+ Python-to-PL/SQL Conversion: 96.8% of output-comparable MBPP records and 89.7% of MBPP+ records matched Python-reference checks; pass/fail agreement was 96.7% and 91.6%, respectively.Considering only conclusive comparisons increased agreement to 98.7% for MBPP and 96.6% for MBPP+.
  • B.6 Spider 2.0 Quality Control Pipeline: Spider 2.0 quality control combines automatic checks, human review, post-processing, and LLM-as-judge review.Automatic checks cover formatting, metadata, prompt-reference alignment, schema references, compilation, execution, and test validity; human review examines semantics, idiomatic PL/SQL, developer intent, and annotation issues.
  • B.7 Quality Dimensions: Table 4 summarizes the primary quality dimensions used during review.The passage identifies the table as the review framework’s summary of quality dimensions.

C Metric Definitions … D.2 Reasoning Type and Difficulty Distribution on Spider2

PLSQLBench evaluates generated PL/SQL with execution-based, test-level metrics that distinguish partial and complete success for single-turn tasks and multi-turn episodes. Its Spider2 analyses characterize conversation lengths and classify tasks by procedural/database difficulty and reasoning type.

  • C Metric Definitions: Execution-based metrics use one generated PL/SQL program per task or turn, assigning zero tests to compilation, runtime, or object-creation failures.Test-level partial credit is computed within each task or episode before averaging across tasks or episodes.
  • C Metric Definitions: Single-turn evaluation reports task-level test pass rate, strict suite-level success, and Mean Test Pass@1, which weights tasks equally after test normalization.Mean Test Pass@1 averages each task’s fraction of passed unit tests, rather than pooling tests across tasks.
  • C Metric Definitions: Multi-turn evaluation requires every turn for Episode Pass, while Turn Suite Pass counts fully solved turns and Mean Test Pass gives partial credit across episode tests.Each episode contributes equally to the final multi-turn benchmark score despite differing numbers of turns or tests.
  • D.1 Turn Distribution of Spider2-MT: Spider2-MT contains 271 complete conversations spanning 978 turns, with three- to five-turn conversations and 208 development versus 63 test conversations.Three-turn conversations are the largest group, comprising 155 conversations (57.2%).
  • D.2 Reasoning Type and Difficulty Distribution on Spider2: Difficulty labels reflect the procedural and database reasoning required, progressing from simple retrieval or control flow to intermediate multi-requirement tasks and advanced PL/SQL construction.Intermediate tasks may involve joins, aggregation, cursor iteration, validation, or structured exception handling; advanced tasks include packages.
  • D Dataset Statistics Details: Spider2 task statistics are organized across Spider2-ST and Spider2-MT development and test splits, with Spider2-MT difficulty counts measured over turns.The difficulty distribution is summarized in Table 7 for the two split types and their development/test partitions.
  • D.2 Reasoning Type and Difficulty Distribution on Spider2: The Spider2-ST and Spider2-MT splits also report a top-10, multi-label reasoning-type distribution ranked by Spider2-MT development frequency.Table 8 presents counts and within-split percentages, with Spider2-MT counts measured over turns.

D.3 Representative Benchmark Instances … E.2 Error Analysis Protocol

Representative instances span schema-grounded single-turn functions, MBPP-derived function generation, and multi-turn procedure revision, while strict metrics and execution-log analysis assess correctness and failure modes. The examples emphasize exception handling, procedural behavior, cross-turn preservation, and schema grounding.

  • D.3 Representative Benchmark Instances: Spider2-ST tasks require reusable PL/SQL functions with anchored database behavior and explicit exception handling.The representative function accepts p_city_name IN VARCHAR2, returns VARCHAR2 population text, matches city names case-insensitively, and handles NULL input.
  • D.3 Representative Benchmark Instances: MBPP-PLSQL converts programming tasks into PL/SQL function generation without persistent tables, supplying required PL/SQL types and unit tests.One example asks models to implement min_cost for a minimum-cost matrix path using nested NUMBER collection types.
  • D.3 Representative Benchmark Instances: Spider2-MT tests whether models can revise prior PL/SQL artifacts across turns while preserving established interfaces and behavior.The representative three-turn sequence adds threshold branching, then a schema join and object-oriented formatting to the same stored procedure.
  • E Additional Evaluation Results: Additional strict evaluation reports Single-Turn Suite Pass@1, Multi-Turn Episode Pass@1, and Multi-Turn Turn Suite Pass@1 alongside Mean Test Pass@1.The appendix identifies Mean Test Pass@1 as the primary metric and adds strict suite-level measures.
  • E.1 Multi-Turn Strict Metrics: Episode Pass@1 requires every conversation turn to pass all unit tests, whereas Turn Suite Pass@1 measures suite-level success separately for each turn.These metrics are reported for Spider2-MT.
  • E.2 Error Analysis Protocol: Error analysis treats each evaluated single-turn instance and each multi-turn turn as an analysis unit, excluding reference_error rows before categorizing model failures.Remaining executions are separated into passes and failures using statuses, Oracle diagnostics, skipped-execution reasons, and expected-versus-actual outputs.
  • E.2 Error Analysis Protocol: The failure taxonomy distinguishes wrong procedure logic or output from invalid or incomplete PL/SQL artifacts and other model-attributable failures.Categories cover incorrect control flow, exception handling, state updates, formatting, query results, and malformed or non-executable program units.
  • E.2 Error Analysis Protocol: Oracle diagnostics and output comparisons identify compilation, entry-point, argument, schema-grounding, and behavior mismatches, with multi-turn failures retained by conversation identifier.The analysis reports aggregate counts by category, dataset, and model, then manually inspects representative failures while avoiding brittle exact-string artifacts.

E.3 Representative Error Examples · F Prompts · G Hyperparameters and Infrastructure

Representative examples show failures in procedure logic, artifact validity, and required interfaces, while the benchmark prompt emphasizes executable, schema-grounded Oracle PL/SQL. The appendix also reports prompting and infrastructure settings to support reproducibility.

  • E.3 Representative Error Examples: Representative failures span wrong procedure logic, invalid or incomplete artifacts, and interface, signature, or schema-grounding errors.Figures 5–7 use execution evidence and unit-test outcomes to identify these error categories.
  • E.3 Representative Error Examples: 0/3 tests passed when GPT-5.4 emitted only an exception message instead of required report headers for an empty-result case.The generated procedure compiled and looped over records, but raised NO_DATA_FOUND before producing the report context.
  • E.3 Representative Error Examples: 0/1 tests passed when GPT-5.4 returned INVALID_REQUEST instead of executable PL/SQL for a solvable cartoon-title query.The task required joining TVSHOW.TV_CHANNEL with TVSHOW.CARTOON and printing matching titles.
  • E.3 Representative Error Examples: 0/4 tests passed when Gemini-2.5-Flash-Lite created only a schema-qualified package body, leaving the required public package procedure uncallable.The task required PIZZA_RUNNER_UTIL.KPI_DASHBOARD to accept a runner ID and print delivery KPIs.
  • F.1 Prompt for PL/SQL Code Generation: The PL/SQL-generation prompt directs models to produce executable Oracle PL/SQL using only the provided schema context.It explicitly requires schema grounding, Oracle-dialect compliance, and executability while supporting procedural features such as control flow, exceptions, cursors, and packages.
  • F.1 Prompt for PL/SQL Code Generation: The prompt frames PL/SQL as Oracle’s procedural extension to SQL and identifies variables, control flow, exception handling, cursors, packages, object types, and dynamic SQL as supported features.The system prompt structure follows prior SQL-generation system prompts.
  • G Hyperparameters and Infrastructure: The appendix reports model, prompting, inference, and infrastructure settings used in PLSQLBENCH experiments to support reproducibility.These settings are summarized in Table 11.
Loading 2608.15931v1…