PaperlayerView exact revision on arXiv

Source-linked AI summary

AptMQL-Bench: From Text-to-SQL to Text-to-MQL via Access-Pattern Schema Design and Data-Preserving Migration

Hy Nguyen, Nabi Rezvani, Robin Vujanic

arXiv:2610.02770v1cs.CLcs.DB

TL;DR

Mechanical text-to-MQL benchmark conversions can fail to migrate databases, lose data, and produce inefficient queries as data scales. The paper presents a coding-agent pipeline with human verification that designs schemas around expected access patterns and builds AptMQL-Bench from BIRD. Claude Opus 4.5 reaches 57.38% Soft Execution Accuracy without external knowledge and 70.34% with it, showing that the task remains challenging.

  • Problem

    Mechanical benchmark conversions can fail to migrate databases, lose rows, and yield inefficient MQL queries as data scales.

    Source: §1 · pp. 1–2

  • Method

    A coding-agent pipeline with human verification designs schemas from expected access patterns, validates migration, and translates and optimizes SQL queries into MQL.

    Source: §1 · p. 2

  • Results

    57.38% without external knowledge and 70.34% with it: Claude Opus 4.5’s Soft Execution Accuracy on AptMQL-Bench.

    Source: §8 · p. 8

  • Takeaways & Limitations

    The authors release the benchmark, conversion pipeline, and evaluation scripts to support future research on text-to-MQL generation.

    Source: §8 · p. 8

    Schema designs depend on assumed access patterns, so their representativeness depends on how well those assumptions reflect practical workloads.

    Source: Limitations · p. 9

Abstract

from arXiv · show

Document databases such as MongoDB are core infrastructure for modern applications, and natural-language interfaces to them---text-to-MQL---would let non-experts query complex, semi-structured data without mastering the query language. Progress on this task depends on high-quality benchmarks, which are most practically obtained by converting an existing text-to-SQL benchmark to the document setting. Unfortunately, existing efforts rely on heuristics for mechanical conversion: the document schema mirrors the relational foreign-key graph, and each query mirrors its source SQL. As a result in our experiments, these approaches fail to migrate 6 of 21 BIRD databases outright, silently drop up to 25.9\% of rows on others, and yield schemas whose ground-truth queries run over an order of magnitude slower as the data scales. We instead propose a conversion pipeline, driven by coding agents with human-in-the-loop verification, that designs each document schema from expected access patterns and rewrites queries to be MongoDB-native. Applying it to BIRD, we build an access-pattern-based text-to-MQL benchmark (AptMQL-Bench). It includes 21 document-oriented databases, 3,186 natural-language requests, and their associated MQL queries---whose databases are migrated from SQLite without data loss and scale efficiently. The strongest model, Claude Opus 4.5, achieves only 57.38\% accuracy without external knowledge evidence and 70.34\% with it. This indicates that realistic text-to-MQL generation remains challenging.

1 Introduction

Mechanical conversions can fail to migrate databases, lose rows, and produce inefficient queries as data scales. The paper proposes a human-verified, access-pattern-driven pipeline and applies it to build AptMQL-Bench.

1 Introduction

  • Mechanical conversions failed to migrate 6 of 21 BIRD databases, silently lost up to 25.9% of rows, and produced inefficient queries that worsen as data scales.Failures included infinite recursion and documents exceeding MongoDB’s 16 MB limit.Source: §1 · pp. 1–2
  • The pipeline uses coding agents with human verification to design schemas from expected access patterns, validate data migration, and translate and optimize SQL queries into MQL.Applied to BIRD, it produced 21 document databases and 3,186 requests with associated MQL queries, migrated without data loss and efficient as databases grow.Source: §1 · p. 2
  • Claude Opus 4.5 achieves 70.34% soft execution accuracy, while GPT-OSS-120B reaches 62.46%, showing that text-to-MQL generation remains challenging.The evaluation includes proprietary and open-source LLMs.Source: §1 · p. 2
  • The authors release the benchmark, prompts, and evaluation scripts to support future work.Source: §1 · p. 2

2 Related Work

Prior work includes text-to-SQL benchmarks, relational-to-NoSQL conversion, and natural-language NoSQL benchmarks. AptMQL-Bench uses coding agents and human review to convert an existing benchmark with access-pattern-driven design.

Text-to-SQL

Relational-to-NoSQL conversion

  • Relational-to-NoSQL conversion spans structure-driven mappings and workload-driven schema design, with NoSE and later query-pattern models representing the latter.Source: §2 · pp. 2–3

Text-to-NoSQL Benchmarks

  • Text-to-NoSQL benchmarks include mechanical conversions such as DocSpider and TEND-v2 [Lu et al., 2025], expert-authored TEND-v3 [Lu et al., 2026], and the medical-domain SM3-Text-to-Query [Sivasubramaniam et al., 2024].Source: §2 · p. 3
  • AptMQL-Bench occupies a middle ground by converting an existing text-to-SQL benchmark with coding agents and human verification, using access-pattern-driven design and migration without data loss.Source: §2 · p. 3

3 The Conversion Pipeline

The conversion pipeline decomposes benchmark conversion into schema design, migration and validation, and SQL-to-MQL translation. Coding agents carry out subtasks, with human review at quality gates.

3 The Conversion Pipeline

  • The pipeline uses a coding agent and lightweight human review at quality gates, dividing conversion into schema design, migration and validation, and SQL-to-MQL translation.Source: §3 · p. 3

3.1 Schema Design

  • Schema design follows the rule that data accessed together is stored together, using a guidebook of MongoDB design patterns and anti-patterns to ground modeling choices.For example, the beer retail brand page stores a precomputed rating and sample reviews with the brand, while the unbounded full review set remains in a separate collection.Source: §3.1 · p. 3, Figure 2 · p. 3
  • For conceptual design, an agent documents the source schema, application domain, data profiles, assumed access patterns, modeling choices, collections, indexes, and transformations; reviewers assess choices and retrievability of source tables.Source: §3.1 · pp. 3–4
  • Separate agent sessions define each collection’s fields, mapping SQLite columns to MongoDB fields, types, and transformation notes.Source: §3.1 · p. 4

3.2 Migration and Validation

  • Migration applies the logical schema, while collection-level, cross-collection, and SQLite-anchored checks iteratively flag mismatches, broken references, or unaccounted source values for review and correction.The source-anchored pass helps detect tables or columns omitted from the design itself.Source: §3.2 · pp. 4–5

3.3 Translation from SQL to MQL

  • SQL-to-MQL translations retain the original natural-language requests and are checked for exact result equivalence; cases requiring unnatural constructs are flagged for manual review.Source: §3.3 · p. 5
  • An optimization pass removes relational-shaped SQL legacy patterns by using embedded fields and precomputed aggregates; reviewers check semantic equivalence, and 1,444 MQLs (45.32%) were optimized.External-knowledge evidence is also rewritten to use MongoDB collection.field paths and behavior hints aligned with the ground-truth MQL.Source: §3.3 · p. 5

4 AptMQL-Bench

AptMQL-Bench converts 21 BIRD databases into document databases and pairs them with reused natural-language requests and MQL queries. The paper also estimates the human review effort involved in construction.

Databases

  • 21 databases average 7.1 collections and roughly 218,000 documents each, and all were migrated without data loss using coding agents and human review.Source: §4 · p. 5

Natural-language requests and MQL queries

  • 3,186 BIRD requests are reused verbatim, while their associated MQL queries average 4.0 aggregation stages and a nesting depth of 4.8.The requests average 82 characters and have a 3-gram diversity of 0.68.Source: §4 · p. 5

Human Effort

  • Schema design and validation took about 27 minutes per database on average, while translation review time concentrated on optimization and evidence rewriting.These human-effort estimates were reviewer-estimated rather than instrumented; optimization took 30–60 seconds per optimized MQL, and evidence rewriting took 15–30 seconds per evidence.Source: §4 · pp. 5–6

5 Evaluation Metrics

Soft-EX evaluates semantic correctness while allowing predicted answers to include extra fields. It still requires the same rows and value-matching ground-truth fields, and averages instance scores.

5 Evaluation Metrics

  • Soft-EX accepts extra predicted fields when every ground-truth column has a value-equivalent predicted column.This permits returning whole documents when they contain the requested fields.Source: §5 · p. 6
  • Predicted and ground-truth answers must contain the same number of rows, with every ground-truth column matching value-for-value.Returning an entire collection does not qualify unless the ground-truth answer already spans it.Source: §5 · p. 6
  • Soft-EX is the mean of its instance-level scores across test instances.The paper reports that accepted cases rejected by Strict-EX contained correct answers with additional fields, not extra rows or incorrect values.Source: §5 · p. 6

6 Benchmarking

The paper benchmarks eight open-source and proprietary models with and without BIRD external knowledge. Claude Opus 4.5 performs best, while the results show substantial remaining difficulty and a consistent benefit from external knowledge.

6 Benchmarking

  • Eight open-source and proprietary LLMs are evaluated with and without BIRD external knowledge on Exec-Rate, Strict-EX, and Soft-EX.The open-source group contains five models; the proprietary group contains three Claude models.Source: §6 · p. 6
  • 57.38% Soft-EX without external knowledge and 70.34% with it are Claude-Opus-4-5’s scores, and the benchmark remains challenging for every model.External knowledge adds 12.15–18.51 points across models, including +12.96 for Opus.Source: §6 · p. 6
  • Execution success is near-saturated for capable models, but Opus’s Strict-EX scores of 38.45/50.44 trail its Soft-EX scores of 57.38/70.34.The Strict-EX/Soft-EX gap reflects correct answers with additional fields, which Strict-EX penalizes.Source: §6 · p. 6

7 Schema Design Matters: One-to-One, FK-Embedding, and Access-Pattern Modelling

The comparison evaluates data integrity and query performance as databases scale across One-to-One, FK-Embedding, and Access-Pattern conversions. Access-Pattern preserves data and yields the fastest, gentlest-scaling ground-truth query execution in the reported comparisons.

Schema comparison

  • The strategies are One-to-One table-to-collection mapping, FK-Embedding along foreign keys, and Access-Pattern document modelling.The comparison considers both row-level data loss and query performance under scale.Source: §7 · pp. 6–7

Data integrity

  • Access-Pattern migrates databases without data loss, while FK-Embedding fails outright on six databases and silently drops up to 25.9% of rows on toxicology.One-to-One is lossless by construction, but mirrors the relational schema; FK-Embedding partially migrates eight additional databases.Source: §7 · p. 7

Ground-truth MQL execution under scale

  • The study compares all three strategies on three Spider databases, but only Access-Pattern and One-to-One on eleven BIRD databases.FK-Embedding ground-truth MQLs cannot be reproduced for BIRD; the study scales databases while preserving schema, keys, and referential integrity.Source: §7 · pp. 7–8
  • One-to-One is slowest throughout because relationship-spanning queries use unindexed cross-collection lookups; FK-Embedding’s advantage is uneven on many-to-many schemas.Where embedding is possible, FK-Embedding co-locates related rows and uses array unwinding instead of joins.Source: §7 · p. 8
  • Access-Pattern is fastest on every database; on Spider at 200 MB it is ∼11× faster than FK-Embedding and ∼102× faster than One-to-One.On BIRD, its speedup over One-to-One grows from around 40x on the original databases to 77x at roughly 160 MB.Source: §7 · p. 8, Figure 3 · p. 8

8 Conclusion

The paper presents a coding-agent-assisted pipeline for translating text-to-SQL benchmarks into text-to-MQL benchmarks and applies it to BIRD. The resulting benchmark preserves data, while model results show text-to-MQL remains challenging.

8 Conclusion

  • The semi-automated pipeline has three stages: document schema design, database migration and validation, and SQL-to-MQL translation.It uses coding agents to assist conversion of existing text-to-SQL benchmarks.Source: §8 · p. 8
  • AptMQL-Bench contains 21 document databases and 3,186 natural-language requests with associated MQL queries, migrated without data loss.Source: §8 · p. 8
  • Claude-Opus-4-5 achieves 57.38% Soft Execution Accuracy without external knowledge and 70.34% with it.The paper releases the benchmark, conversion pipeline, and evaluation scripts.Source: §8 · p. 8

Limitations

The pipeline’s schemas depend on assumed access patterns, reviewer-effort figures are estimates, and empirical validation is limited to MongoDB.

Limitations

  • Schema representativeness is bounded by how well the assumed application workload reflects the workload the design is meant to serve.The agent infers an application and its access patterns and operation rates, which a human reviewer checks; the authors identify practical query workloads as future grounding.Source: Limitations · p. 9
  • Reviewer-effort reporting is estimated rather than instrumented and indicates only the rough scale of human involvement.Source: Limitations · p. 9
  • The experiments validate MongoDB and MQL only, leaving generalization to other document stores empirically untested.Source: Limitations · p. 9

Ethics Statement

The paper reports its data-use and release plans, limits empirical validation to MongoDB, analyzes model errors, and describes reviewer effort and migration checks.

Ethics Statement

  • The benchmark is built from BIRD under its intended use and licensing, with no additional personal or sensitive data collected and release planned under CC BY-SA 4.0.The authors foresee no direct ethical risks from releasing the benchmark, pipeline, and evaluation scripts.Source: Ethics Statement · p. 9

A Generality beyond MongoDB.

  • The pipeline may apply to document-oriented stores with similar document and aggregation models, but cross-store transfer remains untested.Source: A Generality beyond MongoDB · p. 10

B Error Analysis

  • Among 100 manually analyzed Opus predictions with external knowledge, 37 were acceptable mismatches and 33 were calculation errors; most failures were semantic or evidence-grounding issues rather than syntax errors.Other errors involved knowledge and evidence (12), database content (8), schema linking (6), and syntax (4).Source: B Error Analysis · p. 10

C Reviewer Expertise and Effort

  • Reviewer-effort estimates put schema design and validation at roughly 27 minutes per development database, while per-sample review concentrates on query optimization and evidence rewriting.The reported manual effort is estimated, not instrumented; optimization takes 30–60 seconds per optimized query and evidence rewriting 15–30 seconds per sample.Source: C Reviewer Expertise and Effort · p. 11

E Validation Mappings

  • Validation mappings compare source-side SQLite query results with target MongoDB aggregation results, checking equality for each small, single-purpose mapping.For precomputed brand ratings and review counts, a mapping recomputes the aggregates from the source and compares them with stored values, checking the transformation.Source: E Validation Mappings · p. 12

D Example Conceptual Schema Design

  • For beer_factory, the conceptual design maps every SQLite table to a destination, and its data-loss check indicates all source data remains retrievable.Figure 4 and Table 7 present abridged conceptual and logical designs; Table 6 covers the conceptual-level check.Source: D Example Conceptual Schema Design · p. 11, Table 6 · p. 12

F Soft-EX vs. Strict-EX: Analysis of the Disagreement Set

The Soft-EX/Strict-EX disagreement set consists of predictions that return the correct answer with extra fields, which Strict-EX rejects. The authors therefore treat Strict-EX as a conservative lower bound, with true accuracy closer to Soft-EX.

F Soft-EX vs. Strict-EX: Analysis of the Disagreement Set

  • Strict-EX requires bidirectional containment, making the predicted and gold result sets equal up to field renaming and row order.Source: F Soft-EX vs. Strict-EX: Analysis of… · p. 13
  • The analysis selects Claude-Opus-4-5 with external knowledge and examines instances Soft-EX scores correct but Strict-EX scores incorrect.The disagreement set contains 634 instances; Opus has 2,241 Soft-EX-correct and 1,607 Strict-EX-correct predictions.Source: F Soft-EX vs. Strict-EX: Analysis of… · p. 13
  • All 634 Soft-EX-correct but Strict-EX-incorrect cases had the correct row count and matched ground-truth fields, differing only by extra fields.The authors describe these predictions as idiomatic for document stores and conclude that Strict-EX is a conservative lower bound, with true accuracy closer to Soft-EX.Source: F Soft-EX vs. Strict-EX: Analysis of… · p. 13

G Scaling Procedure for the Execution-Time Study

The execution-time study scales identical source data while holding relational structure fixed, then compares how ground-truth MQL latency changes across migration strategies. Synthetic rows preserve declared and undeclared relationships, and measurements average query times across databases.

G Scaling Procedure for the Execution-Time Study

  • The study compares latency growth for three Spider databases under all conversions and eleven BIRD databases under Access-Pattern and One-to-One; FK-Embedding is unavailable for BIRD.The FK-Embedding migration is reproduced, but its MQL queries and translation prompt are not released for BIRD.Source: G Scaling Procedure for the… · p. 13, Figure 7 · p. 20
  • Each source database is scaled with its relational schema, keys, and relationships fixed, then identical data is migrated under each conversion..Source: G Scaling Procedure for the… · p. 14
  • Scale factor k adds (k−1) times each table’s original row count using generated keys, valid foreign keys, unique counters, and source-distribution values.Special cases include one-to-one links, text-typed foreign keys, and many-to-many junctions.Source: G Scaling Procedure for the… · p. 14
  • Generation preserves undeclared semantic links, including bike_1 trip-to-station references and the weather(date, zip_code) composite key.New trips retain station-name consistency, while weather rows pair fresh dates with existing ZIP codes.Source: G Scaling Procedure for the… · p. 14
  • Scaled databases pass SQLite foreign-key and integrity checks; Spider targets roughly 50 and 200 MB, while BIRD uses 5× and 10× expansions.BIRD expansions span mean sizes of roughly 20, 80, and 160 MB, and documents are checked against MongoDB’s 16 MB limit.Source: G Scaling Procedure for the… · pp. 14–15
  • Per-query execution times are averaged at each size, then database averages are combined by geometric mean and plotted against scale factor.Measurements use a MacBook Air with an Apple M2 and 24 GB RAM.Source: G Scaling Procedure for the… · p. 15, Figure 7 · p. 20

Related papers

Loading 2610.02770v1…