Source-linked AI summary
DBcover: A White-box SQL Test Generation Framework for Coverage Improvement
Yankai Rong, Shuang Liu, Jinhao Dong, Qiang Yin, Wei Lu, Jianhua Wang, Xiaoyong Du
TL;DR
RDBMS testing needs higher code coverage despite large codebases and complex execution logic. DBcover combines dynamic SQL-to-path analysis, knowledge-graph context, and two-phase LLM-guided generation, achieving 80.1% coverage on PostgreSQL and 82.3% on MySQL while also working on KingbaseES.
Problem
Improving RDBMS code coverage is important for system quality and robustness, but large codebases and complex execution logic make high coverage challenging.
Method
DBcover uses global and local context in a unified knowledge graph, selects a nearby execution-path seed, and guides an LLM to generate white-box SQL tests.
Results
DBcover achieves 80.1% coverage on PostgreSQL and 82.3% on MySQL, and is effective on the closed-source RDBMS KingbaseES.
Takeaways & Limitations
The framework demonstrates practical applicability across open-source PostgreSQL and MySQL and the commercial RDBMS KingbaseES.
Takeaways & Limitations
The evaluation uses a specific 32B-parameter LLM, and broader testing across model sizes and architectures remains future work.
Abstract
from arXiv · showhide
Relational Database Management Systems (RDBMSs) are the backbone of modern data-intensive applications, making reliability and robustness critical. However, achieving high coverage in RDBMS testing remains challenging because of large codebases and complex execution logic. Traditional fuzzing relies on random SQL generation and cannot capture the correspondence between SQL inputs and internal execution paths, while symbolic execution suffers from prohibitive cost and scalability limitations. We propose DBcover, an LLM-driven white-box SQL test generation framework based on contextual reasoning. DBcover uses lightweight dynamic analysis to extract SQL-to-path correspondence and call graphs as global context, and collects source-level information around target functions as local context. These contexts are organized in a unified knowledge graph for efficient retrieval and reuse. DBcover then performs two-phase test generation: it first selects a semantically relevant seed whose execution path is close to the uncovered target, and then guides the LLM with global and local context to generate SQL test cases that trigger previously uncovered code regions. Experiments show that DBcover achieves 80.1% and 82.3% coverage on PostgreSQL and MySQL, and is also effective on the enterprise RDBMS KingbaseES, demonstrating its practical applicability to closed-source systems.
1 Introduction
DBcover addresses the difficulty of improving RDBMS code coverage by combining lightweight analysis with LLM reasoning over global and local code context. Its two-phase process selects a path-relevant seed and generates SQL tests targeting uncovered regions.
- RDBMS reliability makes code coverage an important measure of testing adequacy and system quality.
- Existing SQL-generation methods improve syntax, semantics, database-state consistency, or coverage collection, but DBMS-oriented testing remains challenging.
- Symbolic analysis cannot practically explore SQL-to-path correspondences at DBMS scale because path explosion makes exhaustive analysis impractical.
- LLM test-generation methods commonly target isolated, bounded components, leaving efficient contextualization of complex RDBMS code as a central challenge.
- DBcover unifies global and local context in a knowledge graph to support contextual SQL test generation.Global context includes SQL-to-path correspondence and call graphs; local context includes target-function source code and dependencies.
- DBcover first selects a seed whose execution path is close to an uncovered region, then uses retrieved context to guide LLM-generated SQL tests.
2 Motivation Example
The motivation example shows why stateful, semantically constrained SQL paths are difficult to reach with ordinary generation. DBcover combines execution traces, call-graph context, source inspection, and targeted seed mutation to cover the previously unreachable branch.
- Motivation Example: DBcover’s generated SQL covers the previously unreachable partition_rbound_datum_cmp line by extending a regression-test seed.
- Runtime Conditions: The target branch requires repeated access to the same range partition, with PARTITION_CACHED_FIND_THRESHOLD set to 16.
- Motivation Example: The seed satisfies range-partition and partition-descriptor conditions but performs only one insertion, so the cache threshold is not reached.
- Motivation Example: The example requires semantically dependent SQL statements and evolving runtime state that syntax-oriented or naive fuzzing does not systematically capture.
- Contextual Reasoning: DBcover organizes regression-test traces, call-graph structure, target source, and dependencies into global and local context for LLM generation.
- Targeted Mutation: DBcover adds 15 insertions to the same range partition, bringing the total to 16 and triggering the cache-optimized branch.
3 Methodology
DBcover constructs a knowledge graph that unifies global SQL-to-path and call-graph context with local target-code information. It then selects nearby seeds and uses a fine-grained LLM pipeline to generate SQL targeting uncovered regions.
- Knowledge Graph Construction: DBcover executes existing regression tests once, records SQL-to-path traces, and stores them in the knowledge graph for reuse.The regression suite supplies the initial seed pool and dynamic execution information.
- Test Cases Generation: DBcover prompts an LLM with the selected seed, relevant execution path, global context, and local target-function context to synthesize new SQL tests.The generation component uses a fine-grained five-stage reasoning pipeline with targeted prompt instructions.
- Knowledge Graph Construction: DBcover builds a code context knowledge graph from SQL queries, execution paths, and functions connected by Execution, Contain, and Call edges.Nodes and edges carry structured properties describing queries, paths, and function code or relationships.
- Knowledge Graph Construction: Global context spans execution paths and call graphs, while local context focuses on the target function or uncovered code lines.The two context scopes are progressively integrated during test generation.
- Seed Selection: Seed selection searches call-graph distances from the uncovered target and retrieves an existing path containing the closest executed function.A configurable depth bound limits the search, and the associated SQL query becomes the seed.
4 Evaluation
The evaluation examines DBcover’s coverage effectiveness, the contribution of its two-phase design, and its applicability to closed-source industrial databases.
- Evaluation: The evaluation addresses coverage improvement against existing generators, the contribution of DBcover’s two-phase design, and effectiveness on closed-source industrial databases.These are defined as research questions RQ1, RQ2, and RQ3.
4.1 Experimental Setup
DBcover is evaluated on MySQL, PostgreSQL, and KingbaseES using shared regression-test seeds and coverage-oriented tooling. The experiments use separate hardware environments for open-source and closed-source systems.
- Instrumentation: Function-level source information, call graphs, and coverage data are collected with Doxygen, LLVM, gcov, lcov, and custom Python scripts.These tools support context extraction, call-relationship construction, and coverage integration.
- Hardware: RQ1 and RQ2 run on a 128-core, 512GB server with two RTX 5880 Ada GPUs, while RQ3 uses an internal 32-core, 64GB environment with two RTX 4090 GPUs.The reported environments differ between open-source-system evaluation and the closed-source industrial-system evaluation.
- Experimental Subjects: The evaluation covers MySQL 8.0.33, PostgreSQL 17.0, and the proprietary industrial DBMS KingbaseES.Including open- and closed-source systems assesses performance across different RDBMS implementations.
- Baselines: The Base configuration measures collective coverage after executing only the initial regression-test cases without generating new tests.This establishes the seed-only coverage reference.
- Seed Inputs: All compared methods use the same seed pool collected from official RDBMS regression-test repositories.The repositories provide broad query coverage and serve as the common initialization basis.
4.2 Comparison with Baselines
DBcover achieves the highest reported line coverage on PostgreSQL and MySQL, while ablations show that both seed selection and fine-grained reasoning contribute to its gains. Remaining coverage is limited by unreachable code and difficult path conditions.
- Comparison with Baselines: 80.1% on PostgreSQL and 82.3% on MySQL are DBcover’s highest reported coverages, exceeding the seed baseline by 11.5% and 11.1%, respectively.SQUIRREL improves by 0.4% and 0.2%, while ShQveL reaches 31.5% and 25.7%.
- Limitations: Most remaining uncovered code lies in routines unreachable through valid SQL during normal execution, including crash recovery, I/O error handlers, and system-management paths.These mechanisms require operating-system-level events outside SQL-interface testing.
- Limitations: Other uncovered regions are SQL-reachable but require complex runtime values, interdependent parameters, system settings, and reasoning beyond current small-parameter LLM capabilities.These architectural and path-condition constraints explain why coverage plateaus around 80%.
4.3 Ablation Study
DBcover’s ablation shows that global context, seed selection, and structured reasoning improve coverage beyond local-code prompting and baseline generation, although gains vary by module.
- Overall ablation results: 2.71% average coverage improvement over DBcovernse and 8.07% over Base shows benefits from both seed selection and fine-grained reasoning.The DBcovernse–Base gap is +5.36%, attributed to local-code-aware prompting alone.
- Module-level analysis: 11.1% and 14.1% coverage gains occur in access/brin over DBcovernse and Base, respectively.BRIN coverage depends on an index, sufficient table data, and a query on the indexed column, forming a stateful dependency chain difficult to infer from source code alone.
- Module-level analysis: 84.9% to 88.0% coverage in utils/adt demonstrates that seed-guided mutations help reach constrained numeric and JSON branches.Seeds provide parseable, type-consistent templates for corner-case inputs such as increased precision or nesting depth.
- Module-level analysis: Optimizer/path, access/nbtree, and tsearch improve by only +0.6%, +0.3%, and +0.2%, respectively.Their remaining uncovered code primarily consists of error-handling routines or assertions.
4.4 Case Study
The case study shows why source-only prompting can produce an invalid direct call, whereas DBcover uses call-graph traversal, seed retrieval, and contextual reasoning to synthesize the correct interface.
- Failure without seed guidance: Source-only prompting generates a syntactically plausible but semantically invalid direct call to interval_lerp.interval_lerp has no SQL interface; PostgreSQL reports that the function does not exist, while percentile_cont is the correct interface.
- DBcover-guided generation: DBcover identifies percentile_cont_multi_final_common as the nearest caller and retrieves its corresponding seed through the knowledge graph.This is the first phase of the two-phase case-study workflow.
- DBcover-guided generation: The LLM then uses structured context to analyze interval_lerp’s trigger condition and generate a correct test case.The second phase combines retrieved knowledge-graph context with structured prompting.
4.5 Evaluation on Enterprise Closed-Source RDBMS
DBcover is evaluated on the closed-source enterprise RDBMS KingbaseES, where it substantially improves coverage in data-type processing and metadata management modules.
- Evaluation setting: KingbaseES evaluation targets data type processing and metadata management, two foundational subsystems of the proprietary database engine.The data-type module includes built-in types and coercion rules, while metadata management governs schema-related metadata.
- Results: 80.0% and 81.4% coverage are reached from initial levels of 50.0% and 68.0% in the two KingbaseES modules.The generated tests exercise edge-case type conversions, nested expressions, and metadata-intensive operations.
- Results: The KingbaseES results demonstrate effective exploration of deep, stateful logic in a proprietary database engine.The evaluation uses an internal environment where source-level instrumentation is permitted under a collaboration agreement.
5 Related Work
Related work spans SQL validity, coverage feedback, and LLM-based generation, while DBcover combines white-box context with LLM reasoning for RDBMS test generation.
- DBMS fuzzing: Traditional DBMS fuzzing improves either syntactic and semantic validity or coverage feedback for meaningful execution differences.These directions include grammar-aware generation, state consistency, and finer-grained coverage collection.
- SQL validity: SQLsmith, SQLancer, Apollo, DynSQL, SQUIRREL, and Griffin generate or mutate SQL while targeting grammar, schema, or semantic validity.Their techniques include syntax-driven traversal, schema-guided AST instantiation, syntax-preserving mutation, and SQL-fragment recombination.
- Coverage feedback: Coverage-oriented tools refine feedback using larger bitmaps, static analysis, or more detailed execution distinctions.Squirrel and SQLRight enlarge AFL’s bitmap, while Ratel addresses coverage-feedback accuracy through static analysis.
- LLM-driven testing: LLM-based testing either augments existing generators or directly synthesizes complete standalone test cases.Examples include ShQveL and LLAMA in augmentation, and Whitefox and CoverUp in direct generation or context-guided testing.
6 Threats to Validity
DBcover’s validity is primarily bounded by generalizability across database architectures and LLM backends. The evaluation covers PostgreSQL, MySQL, and KingbaseES, but uses one 32B-parameter model and plans broader model evaluation.
- Threats to Validity: Generalizability across database architectures and LLM backends is the primary threat to validity.The evaluation addresses database diversity with PostgreSQL, MySQL, and KingbaseES, while broader LLM coverage remains future work.
- Threats to Validity: The evaluation spans PostgreSQL, MySQL, and the commercial closed-source DBMS KingbaseES.PostgreSQL and MySQL represent complex open-source architectures, while KingbaseES adds commercial-system diversity.
- Threats to Validity: The model-dependence evaluation uses a specific 32B-parameter model, with broader architectures and parameter sizes left for future study.The authors report that this model achieved significant gains under path-aware guidance but plan broader evaluation.
7 Conclusion and Future Work
DBcover performs white-box, code-aware SQL test generation through two-layer contextual reasoning and two-phase seed-guided generation. It significantly outperforms state-of-the-art methods on code coverage, and its path-aware tests may support future bug discovery.
- 7 Conclusion and Future Work: DBcover combines global and local context collection with two-phase seed-guided LLM test generation for RDBMS code coverage.Global context supports seed selection, while local context guides accurate SQL generation.
- 7 Conclusion and Future Work: DBcover significantly outperforms state-of-the-art SQL test generation methods on code coverage.
- 7 Conclusion and Future Work: Path-aware SQL tests are compatible with differential testing, metamorphic relations, and crash monitors for future systematic bug discovery.The paper identifies these bug-detection oracles as future work beyond its primary code-coverage metric.