Source-linked AI summary
Replacing Training with Memory: Listwise Selection for Text-to-SQL
Yeonseok Jeong, Soyoung Yoon, Seongjun Lee, Seung-won Hwang
TL;DR
Text-to-SQL listwise selectors can be costly to fine-tune while handling long candidate lists and positional bias. MAP-SQL replaces these training roles with structured memory retrieval and execution-aware permutation aggregation, improving selection accuracy and efficiency across benchmarks. On BIRD-dev, it exceeds R3-SQL by 2.02 execution accuracy points on average with the same candidate sets and 2.92× fewer tokens.
Problem
Fine-tuning listwise selectors is costly because candidate queries and execution results create long training contexts and high computational overhead.
Method
MAP-SQL uses structured memories as selection criteria and aggregates rankings across input permutations, optimizing inference with execution feedback and selective pointwise scoring.
Results
2.02 execution accuracy points: MAP-SQL outperforms R3-SQL on BIRD-dev, Spider-test, and EHRSQL by 2.02, 0.53, and 0.68 points, respectively, while using fewer calls and tokens.
Takeaways & Limitations
MAP-SQL provides fine-tuning-free listwise selection that is compatible with off-the-shelf language models and improves efficiency over existing selector methods.
Takeaways & Limitations
Overall execution accuracy of 70–72% lags behind proprietary-model SOTA systems at 75–76%, partly because of generator quality; results assess selectors on fixed candidate pools.
Abstract
from arXiv · showhide
Modern Text-to-SQL systems often follow generate-execute-select pipelines, generating multiple candidate queries then selecting the best one. Listwise selection, by jointly comparing multiple candidates, has been widely adopted, but fine-tuning listwise selectors is costly. We thus propose a fine-tuning-free listwise selector. We replace two major fine-tuning objectives with inference-time strategies: (1) learning selection criteria as ordering and (2) mitigating positional bias. First, we build reusable structured memories instead of learning selection behavior as model parameters. Given a question, MaP-SQL retrieves memories distilled from training data that encode how natural language maps to schema elements, SQL operations, and expected outputs. These memories serve as explicit decision criteria for evaluating candidates in a listwise manner. Second, to mitigate ordering bias of listwise selectors, we aggregate rankings across multiple input permutations, with inference cost optimized by execution results and pointwise scoring. Our approach improves selection accuracy while maintaining efficiency and compatibility with existing large language models. Across Text-to-SQL benchmarks, it produces more stable selection without fine-tuning and fewer unnecessary comparisons than existing methods. On BIRD-dev, it outperforms the previous state-of-the-art selector-based method R^3-SQL by 2.02 execution accuracy points on average using the same candidate sets, with 2.92x fewer tokens.
1 Introduction
MAP-SQL replaces costly fine-tuning for listwise SQL selection with structured memory retrieval and inference-time permutation aggregation. It improves accuracy and efficiency against R3-SQL using the same candidate pools.
- Generate–execute–select systems produce multiple SQL candidates before a selector chooses the best one.
- Fine-tuning listwise selectors is costly because training examples contain multiple queries and execution results, creating long contexts and high computational overhead.
- Structured memories encode mappings from natural language to schema elements, SQL operations, and expected outputs, providing explicit criteria for listwise comparison.
- Permutation aggregation mitigates positional bias by combining rankings across input orders while leveraging execution signals and selective pointwise scoring.
- MAP-SQL avoids updating selector parameters while retaining joint candidate comparison and using pretrained models with labeled question–SQL pairs for retrieval memories.
- 2.02 execution accuracy points: MAP-SQL improves over R3-SQL on BIRD-dev, Spider-test, and EHRSQL by 2.02, 0.53, and 0.68 points, respectively.It also requires 6.54×, 6.85×, and 7.29× fewer selector calls, and 2.92×, 2.12×, and 4.16× fewer tokens, respectively.
2 Related Work
Prior selection methods trade off independent scoring, exhaustive comparisons, and costly training with positional bias. MAP-SQL replaces fine-tuned selection criteria and bias mitigation with memory retrieval and execution-aware permutation aggregation.
- Existing selection paradigms: Pointwise selectors score candidates independently, while pairwise selectors compare all pairs and can require O(n2) comparisons.
- Training and positional bias: Existing listwise selectors require expensive training to encode selection behavior and jointly process long candidate lists exposed to positional bias.
- Inference-time distinction: MAP-SQL identifies selection criteria and positional-bias mitigation as two fine-tuning roles that inference-time strategies can replace.
- Inference-time distinction: Memory retrieval decomposes natural-language-to-SQL correspondence into encoding, translating, and decoding, while execution feedback groups candidates sharing results.Grouping reduces the permutation space from O(n!) to O(g!), where g ≪n.
- Inference-time distinction: MAP-SQL retrieves selection criteria and permutes candidates only within execution-result groups, using confidence-based comparison to improve accuracy and efficiency.
3 Problem Setup
The setup begins with a question, database schema, and retrieved memories, then generates and executes SQL candidates before selecting one. Execution results are cached, and evaluation measures execution accuracy and efficiency.
- Input: The input consists of a natural-language question x, database schema S, and retrieved memories M.
- Candidate generation and execution: A candidate generator produces SQL candidates C, and each query q_i is executed to obtain a result e_i provided to the selector.Results may be tables, empty results, or errors.
- Candidate generation and execution: Each execution result is computed once and cached for reuse across sliding windows and permutations.
- Output: The selector chooses one query q̂ ∈ C that maximizes correctness measured by execution match.
- Evaluation: Evaluation reports execution accuracy across benchmarks and efficiency through selector calls or input tokens per question.
4 Proposed Method
MAP-SQL replaces listwise-selector fine-tuning with retrieved structured memories for candidate validation and permutation-based ranking aggregation for positional-bias mitigation. Execution-result grouping, confidence estimation, and selective pointwise tie-breaking reduce unnecessary comparisons while retaining joint candidate evaluation.
- MAP-SQL uses memory retrieval and permutation-based aggregation as inference-time substitutes for listwise-selector fine-tuning.Retrieved memories provide selection criteria, while permutation aggregation addresses positional bias.
- Memory generation and retrieval: Structured memories encode natural-language grounding, SQL operations, and expected outputs for validating candidate queries.Memories are organized into encoding, translating, and decoding groups and retrieved from training questions.
- Memory generation and retrieval: Retrieved criteria help distinguish superficially similar SQL candidates by checking relevant fields, operations, ordering, limits, and required outputs.An example combines date-field grounding, ORDER BY ... DESC LIMIT 1, and location-output requirements.
- Listwise selection: Listwise reranking initializes candidates by execution-result frequency and compares windowed candidate sets against shared retrieved memories.The implementation uses window size w = 8 and stride s = 4, keeping each selector call to at most 8 candidates when n = 32.
- Permutation-based bias mitigation: Group-based permutation fixes ordering between execution-result groups and shuffles only within groups, reducing the permutation space from O(n!) to O(g!).The average number of execution-result groups is approximately 2 for n = 8 candidates.
- Permutation-based bias mitigation: Candidates are reranked by average rank across K runs, with confidence estimation comparing the top two and optional pointwise scoring resolving ties.The confidence heuristic uses paired rank differences, and the pointwise selector scores only tied candidates.
5 Experiments
Experiments evaluate MAP-SQL across three Text-to-SQL benchmarks, generators, candidate-pool sizes, and selector baselines. The method consistently improves execution accuracy while reducing selector calls and input tokens.
- Experimental Setup: Three benchmarks—BIRD-dev, Spider-test, and EHRSQL—measure execution accuracy and computational efficiency across diverse Text-to-SQL settings.Execution accuracy checks whether predicted and gold SQL produce the same result; efficiency is assessed using selector calls and input tokens.
- Experimental Setup: MAP-SQL consistently outperforms baselines in accuracy while requiring substantially fewer LLM calls and input tokens.The evaluation compares pointwise, pairwise, and multiple-selector approaches under shared candidate pools and selector protocols.
- Accuracy: 73.08% execution accuracy on BIRD-dev with Agentar-32B and n=32 exceeds R3-SQL by 1.11 points and pairwise selection by 2.02 points.Gains remain consistent across both generators and both candidate-pool sizes; a single listwise selector surpasses R3-SQL in most settings.
- Efficiency: 9.07× fewer calls and 4.16× fewer tokens than R3-SQL are achieved on BIRD-dev with n=32 using Arctic-R1-7B.In the same setting, pairwise selection averages 184.59 calls and 443,713 tokens per query, versus 5.91 calls and 27,440 tokens for MAP-SQL.
- Generalization: 87.59% accuracy on Spider-test surpasses R3-SQL by 0.77 points using 8.43× fewer calls, while EHRSQL reaches 44.71% with a 0.68-point gain and 11.09× fewer calls.These results extend the observed listwise-selection advantage beyond BIRD-dev.
6 Analysis
The analysis shows that memory retrieval and permutation-based aggregation provide complementary gains, while group-based permutation and tie-breaking improve selection consistency. MAP-SQL also reaches comparable quality to a fine-tuned selector without updating selector parameters.
- 72.62 and 72.16 are the full MAP-SQL accuracies on Agentar and Arctic generators, respectively, with eight BIRD-dev candidates.
- 0.55 points is the accuracy drop without Memory, compared with a 0.32-point drop without Permutation.The average accuracy falls from 72.39 to 72.07 without Permutation and to 71.84 without Memory.
- 0.33 and 0.39 percentage points are the memory gains with gpt-5.1-codex-mini across Agentar-32B and Arctic-R1-7B, respectively.Accuracy rises from 71.90% to 72.23% and from 70.01% to 70.40%.
- 33.17% consistency and 22.63% consistency & correct are achieved with tie-breaking after group-based permutation.Group-based permutation reaches 29.04% consistency, versus 23.04% for global permutation and 18.98% for the baseline.
- At four permutations, tie-breaking matches eight group-based permutation runs without requiring additional inference.Group-based permutation saturates faster because it avoids noisy orderings.
- 71.90% execution accuracy slightly surpasses the reported 71.84% of R3-SQL with a fine-tuned selector on the same OmniSQL-7B setting.MAP-SQL achieves this without selector fine-tuning.
7 Conclusion
The conclusion presents MAP-SQL as a fine-tuning-free listwise selector that combines retrieved memories with permutation-based aggregation. It reports stronger selection than pointwise and pairwise baselines while reducing comparison and token costs.
- MAP-SQL uses memory retrieval for selection criteria and permutation-based aggregation to mitigate positional bias without fine-tuning.
- Listwise selection outperforms pairwise and pointwise baselines by jointly contrasting SQL structures, execution outcomes, and schema usage.
- O(N)–O(N log N) listwise comparison complexity replaces O(N2) pairwise complexity, with up to 27.92× fewer input tokens per query.
Limitations
The paper’s limitations concern coordination strategies, end-to-end benchmark scope, schema-scale context management, and generalization beyond Text-to-SQL.
- Advanced listwise coordination strategies beyond sliding windows remain unexplored and are left for future work.
- Overall execution accuracy is 70–72%, below the 75–76% of proprietary-model SOTA systems, partly because of generator quality.The results characterize selector performance on fixed candidate pools rather than end-to-end BIRD SOTA performance.
- Enterprise-scale databases assume upstream schema linking to prune hundreds of tables; schema-scale context management is outside the selector’s scope.
- Whether memory-guided selection and group-based permutation generalize beyond Text-to-SQL remains future work.
Appendices
Appendix experiments test tie-breaking dependence, statistical reliability, candidate-pool robustness, and comparison with a sorted-list strategy. MAP-SQL remains competitive without a separate reward model and shows consistent gains across several evaluations.
- A.1 Effect of Pointwise Tie-Breaking: 72.23% execution accuracy is achieved without tie-breaking, exceeding R3-SQL’s 71.51% in the same BIRD-dev setting.Qwen3-Coder tie-breaking raises accuracy to 72.42%, while Contextual-RM reaches 72.62%.
- A.1 Effect of Pointwise Tie-Breaking: The pointwise reward model is optional: MAP-SQL’s core method does not require an additional reward model.
- A.2 Statistical Reliability: MAP-SQL improvements are statistically supported in both BIRD settings and Spider, but the EHRSQL improvement from one candidate pool is inconclusive.EHRSQL has a confidence interval including zero and an exact McNemar p-value of 0.125.
- A.3 Candidate-Pool Robustness: Across candidate pools generated with seeds 42, 43, and 44, MAP-SQL has higher mean execution accuracy in every dataset and generator setting.
- A.4 Comparison with MCS-SQL: MAP-SQL outperforms the reproduced MCS-SQL sorted-list strategy in all four settings using the same selector and eight-candidate pools.
A.5 Robustness on Dr.Spider
The Dr.Spider experiments test whether question-based memory retrieval remains effective under meaning-preserving wording changes. Listwise selection with memory achieves the strongest post-perturbation accuracy and the smallest accuracy drop reported in Table 12.
- Robustness evaluation: The evaluation uses nine Dr.Spider question perturbations that preserve intended meaning while changing wording.All methods use n=8 candidates generated by Arctic-Text2SQL-R1-7B; Pre, Post, and Drop measure performance before perturbation, after perturbation, and their difference.
- Results: 77.80% is the highest Post accuracy achieved by listwise selection with memory.Listwise selection without memory reaches 77.06% after perturbation.
- Results: 12.72 points is the smallest Drop reported for listwise selection with memory.The corresponding Drop without memory is 13.13 points.
- Interpretation: The Dr.Spider results indicate that retrieved memories remain useful when questions are expressed with different wording.The broader experiments also report consistent gains across three generated candidate pools and competitiveness without a separate pointwise reward model.
C Usage of AI Assistants
This section documents AI-assisted writing support and illustrates how retrieved memories guide listwise SQL selection. The qualitative example shows three executable candidates being distinguished by the requested answer form and the memory’s constraints.
- Usage of AI Assistants: ChatGPT was used to improve writing clarity through rephrasing suggestions and grammatical corrections.The stated purpose was to make the text flow more naturally.
- Memory construction: The memory-generation example derives a structured memory from a training question–SQL pair.The paper presents the generation example and its prompt template in Figures 4 and 5.
- Qualitative selection example: The retrieved memory specifies a single-row text result, a reviewer-name return column, and constraints against extra filters and non-schema-defined columns.These fields provide explicit criteria for evaluating candidate outputs.
- Qualitative selection example: Three candidates all execute successfully but disagree on whether to return a person, a post title, or a person with view count.The example concerns which post by Harvey Motulsky or Noah Snyder has higher popularity.
- Reranking output: The listwise reranker outputs (A) > (C) > (B), selecting candidate A in the illustrated example.The paper also provides full listwise, pairwise, and execution-result prompt templates for the selection procedures.