Replacing Training with Memory: Listwise Selection for Text-to-SQL

1 Introduction

• Listwise selectors jointly evaluate multiple SQL candidates, capturing subtle differences while avoiding the exhaustive O(n^2) comparisons required by pairwise selection.Pointwise selectors score candidates independently and may miss cross-candidate differences.Source: §1 · pp. 1–2
• Fine-tuning listwise selectors is costly because each training example includes multiple SQL queries and execution results, creating long contexts and high computational overhead.MAP-SQL replaces fine-tuning’s selection-criteria and positional-bias roles with inference-time strategies.Source: §1 · p. 2
• Retrieved memories distilled from training data encode mappings from natural language to schema elements, SQL operations, and expected outputs as criteria for listwise candidate comparison.Given a test question, MAP-SQL retrieves relevant memories to guide candidate ranking.Source: §1 · p. 2
• To mitigate lost-in-the-middle positional bias, MAP-SQL aggregates rankings across input permutations using execution signals and selective pointwise scoring to control inference cost.The positional bias is identified as a known issue in listwise selection (Liu et al., 2024; Tang et al., 2024).Source: §1 · p. 2
• MAP-SQL avoids additional fine-tuning while retaining joint candidate comparison; fine-tuning-free means selector parameters are not updated, though pretrained models and labeled question–SQL pairs build its memories.The method is described as compatible with off-the-shelf language models.Source: §1 · p. 2
• On BIRD-dev, Spider-test, and EHRSQL, MAP-SQL improves execution accuracy over R3-SQL by 2.02, 0.53, and 0.68 points, using 2.92×, 2.12×, and 4.16× fewer tokens, respectively.With the same candidate pools, it also requires 6.54×, 6.85×, and 7.29× fewer selector calls, respectively.Source: §1 · p. 2

2.1 Selector Paradigms in Text-to-SQL

• Text-to-SQL selection spans majority voting, pointwise, pairwise, and listwise methods, each with limitations described in prior work.Majority voting (Sheng and Shuai, 2025) can let a larger incorrect group dominate; pointwise selectors lack candidate comparisons, while pairwise selectors (Pourreza et al., 2025; Wang et al., 2025) require quadratically many comparisons. XiYan-SQL (Liu et al., 2026) compares candidates in a window; MCS-SQL (Lee et al., 2025a) sorts candidates before selection without selector fine-tuning, but does not explore listwise challenges or improvements.Source: §2.1 · p. 2

2.2 Training for Listwise Reranking and Positional Bias

• Listwise reranking requires costly training and faces positional bias, while existing mitigation approaches leave inference-time efficiency unresolved.Long candidate lists and execution results make joint training expensive, and long inputs expose models to the “lost-in-the-middle” problem (Liu et al., 2024). R3-SQL uses a pointwise tie-breaker during training; self-consistency can require up to O(n!) inference calls (Tang et al., 2024), and relevance-ranking optimizations (Lee et al., 2025b) have not addressed this target problem.Source: §2.2 · p. 2

2.3 Our Distinction: Inference-time Strategies

• MaP-SQL replaces fine-tuning selection criteria and bias mitigation with inference-time memory retrieval and permutation aggregation.It builds on Deng et al. (2022)'s encoding, translating, and decoding framework and distinguishes itself from PAS-SQL (Kong et al., 2026). Execution feedback groups candidates with identical results, reducing the permutation space from O(n!) to O(g!), where g ≪ n; candidates are permuted within result groups and compared with confidence-based selection.Source: §2.3 · pp. 2–3

• The selector receives a natural-language question, database schema, retrieved memories, and a set of generated SQL candidates.Source: §3 · p. 3
• Each candidate is executed once, and its result is cached for reuse across sliding windows and permutations.Source: §3 · p. 3
• The selector outputs the candidate that maximizes correctness, measured by execution match.Source: §3 · p. 3
• Evaluation measures execution accuracy across benchmarks, with BIRD as the primary benchmark, and efficiency by selector LLM calls or input tokens per question.Source: §3 · p. 3

4.1 Fine-tuning-free Selection with Memory

• Training-question memories encode schema grounding, SQL operations, and expected output formation through Encoding, Translating, and Decoding criteria for judging candidates.The selector retrieves question-similar memories to provide these criteria.Source: §4.1 · pp. 3–4
• Retrieved memories are prepended to the prompt, and candidates are reranked in sliding windows initially ordered by execution-result frequency.The selector includes as many relevant memories as fit within its context limit and uses the same specifications to compare each window.Source: §4.1 · p. 4

4.2 Fine-tuning-free Bias Mitigation with Permutation

• Grouping candidates by execution outcome reduces permutations to groups, preserves between-group ordering, and shuffles within groups, reducing runs while mitigating positional bias.The paper reports that preserving between-group ordering improves accuracy and reduces positional bias compared with prior methods.Source: §4.2 · pp. 4–5
• Candidates are sorted by average rank across runs, and a confidence estimate identifies possible top-candidate ties for optional pointwise scoring.The pointwise selector scores only tied candidates, keeping the dominant computational path listwise.Source: §4.2 · p. 5

5.1 Experimental Setup

• The evaluation uses BIRD-dev (1,534 queries), Spider-test (2,147), and EHRSQL (1,008), reporting execution accuracy plus average LLM calls and input tokens per query.It compares pointwise, pairwise, and multiple-selector baselines, using two generators and fixed pools of n=8 or n=32 candidates.Source: §5.1 · pp. 5–6

5.3 Experimental Results

• 73.08% execution accuracy on BIRD-dev with Agentar-32B and n=32 beats R3-SQL by 1.11 points and pairwise by 2.02 points.The full method achieves the highest execution accuracy across all settings, with gains consistent across both generators and candidate pool sizes; the single listwise selector surpasses R3-SQL in most settings.Source: Table 2 · p. 6, §5.3 · p. 7
• On BIRD-dev with Arctic-R1-7B and n=32, listwise uses 5.91 calls and 27,440 tokens versus pairwise’s 184.59 calls and 443,713 tokens per query.Against R3-SQL in the same setting, the full method uses 9.07× fewer calls and 4.16× fewer tokens; on EHRSQL with n=32, pairwise exceeds 1.3M tokens while listwise uses 48,060.Source: Table 2 · p. 6, Table 3 · p. 6, §5.3 · p. 7
• With n=32, the full method reaches 87.59% accuracy on Spider-test, 0.77 points above R3-SQL with 8.43× fewer calls, and 44.71% on EHRSQL, 0.68 points above with 11.09× fewer calls.These results extend the listwise selection advantage beyond BIRD-dev.Source: Table 3 · p. 6, §5.3 · p. 7

5.2 Implementation Details

• At inference, MaP-SQL retrieves the most relevant training-derived memories with bge-m3, adding as many as fit within the selector’s context limit.BIRD and Spider use memories from their own training splits; EHRSQL uses a combined BIRD and Spider training-split memory bank because it has no training split.Source: §5.2 · p. 7

6.1 Ablation Study

• Removing permutation lowers average accuracy from 72.39 to 72.07, while removing memory lowers it to 71.84.The full system is best on both generators, scoring 72.62 on Agentar-Scale-SQL-Generation-32B and 72.16 on Arctic-Text2SQL-R1-7B.Source: §6.1 · p. 7

6.2 Memories with a Strong Code Selector

• With a gpt-5.1-codex-mini selector, retrieved memories raise accuracy from 71.90% to 72.23% on Agentar-32B and from 70.01% to 70.40% on Arctic-R1-7B.The short structured criterion set complements the selector’s internal reasoning and is applied once to the current candidate list.Source: §6.2 · pp. 7–8, Table 5 · p. 8

6.3 Mitigating Positional Bias

• Group-based permutation raises consistency to 29.04% from 23.04% with global permutation, while tie-breaking reaches 33.17% consistency and 22.63% consistency & correct.At four permutations, tie-breaking reaches 33.17%, comparable to eight group-based permutations without it, with no additional inference; group-based permutation also saturates faster than global permutation.Source: §6.3 · p. 8, Table 6 · p. 8, Figure 3 · p. 8

6.4 Comparison with a Fine-Tuned Selector

• MAP-SQL reaches 71.90% execution accuracy on BIRD-dev with OmniSQL-7B and n=32, versus R3-SQL’s reported 71.84%, without selector fine-tuning.The paper reports that this reaches a comparable level of selection quality to a trained selector.Source: §6.4 · pp. 8–9, Table 7 · p. 9

7 Conclusion

• MAP-SQL’s listwise selector outperforms pairwise and pointwise baselines by contrasting SQL structures, execution outcomes, and schema usage across candidates in one call.Source: §7 · p. 9
• Retrieved memories further improve listwise selection, with gains consistent across multiple generators when using a strong code-focused selector.Source: §7 · p. 9
• Listwise selection reduces call complexity from O(N2) to O(N)–O(N log N) and uses up to 27.92× fewer input tokens per query than pairwise selection.These reductions are reported from the paper’s experiments.Source: §7 · p. 9

Limitations

• The authors leave advanced listwise coordination strategies beyond sliding windows, including TourRank and set-based ranking, for future work.They identify these as alternatives to explore for listwise selection.Source: Limitations · p. 9
• 70–72% overall execution accuracy trails proprietary-model SOTA systems’ 75–76%, partly due to generator quality; results measure selector performance on the same candidate pools, not end-to-end BIRD performance.The authors attribute part of the gap to generator quality rather than selection strategy.Source: Limitations · p. 9
• For enterprise databases with hundreds of tables, the approach assumes upstream schema linking prunes the schema; schema-scale context management is outside the selector’s scope.The passage cites Talaei et al. (2024) for upstream schema linking.Source: Limitations · p. 9
• Whether memory-guided selection and group-based permutation generalize beyond Text-to-SQL remains future work.The paper focuses on Text-to-SQL as a challenging listwise setting with long schemas, execution results, and multiple candidate queries.Source: Limitations · p. 9

A Additional Experiments

• The additional experiments also assess candidate-pool consistency, compare MAP-SQL with MCS-SQL’s sorted-list strategy, and test whether question wording changes affect memory-retrieval robustness.The appendix introduction identifies these as additional evaluation perspectives.Source: A Additional Experiments · p. 12

A.1 Effect of Pointwise Tie-Breaking

• The tie-breaking ablation compares MAP-SQL without tie-breaking, with Qwen3-Coder-30B-A3B-Instruct reused as a pointwise selector, and with Contextual-RM-32B, against R3-SQL.It uses BIRD-dev with n=8 candidates generated by Agentar-Scale-SQL-Generation-32B.Source: §A.1 · p. 12

A.2 Statistical Reliability of MAP-SQL improvements

• The reliability evaluation uses n=32 settings on BIRD-dev, Spider-test, and EHRSQL (Lee et al., 2022), with both methods evaluated on the same candidate pools.Pools come from Agentar and Arctic-Text2SQL-R1-7B (Yao et al., 2026); paired bootstrap estimates 95% confidence intervals and exact McNemar tests compare differing question outcomes.Source: §A.2 · p. 12

A.1 Effect of Pointwise Tie-Breaking

• 72.23% execution accuracy without tie-breaking exceeds R3-SQL’s 71.51%; Qwen3-Coder tie-breaking reaches 72.42%, and Contextual-RM-32B reaches 72.62%.These results show that a separate pointwise reward model is optional, rather than required by the core method.Source: Table 8 · p. 12, §A.1 · p. 12

A.2 Statistical Reliability of MAP-SQL improvements

• MAP-SQL’s gains over R3-SQL are statistically supported on both BIRD settings and Spider: confidence intervals exclude zero and exact McNemar test p-values are below 0.05.The appendix treats the EHRSQL improvement from one candidate pool as inconclusive.Source: §A.2 · p. 12
• For EHRSQL, the confidence interval includes zero and the exact McNemar test p-value is 0.125, so the improvement from this candidate pool is inconclusive.The appendix contrasts this with the statistically supported results on both BIRD settings and Spider.Source: §A.2 · p. 12

A.3 Robustness Across Randomly Generated Candidate Pools

• To test dependence on a particular candidate pool, both selectors were evaluated on pools generated with seeds 42, 43, and 44, each containing n=32 candidates.The preceding analysis used a fixed candidate pool.Source: §A.3 · p. 12
• The robustness comparison reports mean execution accuracy and sample standard deviation across the three generated pools.Both selectors were evaluated on every pool.Source: §A.3 · p. 12, Table 10 · p. 13
• Across three randomly generated n=32 candidate pools, MAP-SQL has higher mean execution accuracy in every dataset and generator setting.The consistent pattern includes EHRSQL, despite an inconclusive result from the single-pool comparison in Table 9.Source: §A.3 · p. 12, Table 10 · p. 13

A.4 Comparison with MCS-SQL

• The MCS-SQL experiment reproduces its multiple-choice selection strategy because its official implementation was unavailable, using Qwen3-Coder-30B-A3B-Instruct and the same n=8 candidate pools for both methods.This setup isolates the selection-strategy difference.Source: §A.4 · p. 13
• MAP-SQL achieves higher execution accuracy than the reproduced MCS-SQL sorted-list strategy in all four settings.The comparison uses the same selector and candidate pools for both strategies.Source: §A.4 · p. 13, Table 11 · p. 13
• The MCS-SQL comparison shows this improvement with both generators on BIRD-dev and also on Spider-test and EHRSQL.The reported pattern supports group-based permutation over the reproduced sorted-list strategy under the tested conditions.Source: §A.4 · p. 13, Table 11 · p. 13

A.5 Robustness on Dr.Spider

• 77.80% Post accuracy and a 12.72-point Drop make memory-based listwise selection the strongest Dr.Spider perturbation result; without memory, Post is 77.06% and Drop 13.13 points.The table reports average execution accuracy across nine Dr.Spider question perturbations, using n=8 candidates generated by Arctic-Text2SQL-R1-7B.Source: Table 12 · p. 13, §A.5 · p. 13
• The evaluation uses bge-m3 retrieval and nine meaning-preserving Dr.Spider rewordings, testing whether question-based memories remain useful under changed wording.All methods use the same eight generated candidates; Dr.Spider perturbations are described by Chang et al. (2023), and bge-m3 by Chen et al. (2024).Source: §A.5 · p. 13
• Across settings, gains over R3-SQL receive statistical support in three of four fixed-pool settings, remain consistent across three generated pools, and beat reproduced MCS-SQL in all four settings.MaP-SQL also remains competitive without a separate pointwise reward model.Source: §A.5 · p. 13

B.1 Memory Examples and Generation Prompt

• The memory design follows Deng et al. (2022)’s Encoding, Translating, and Decoding decomposition, with examples of generated memories and prompts for memory generation and listwise selection.Figures 4–6 illustrate a training question–gold SQL memory, its generation prompt, and an example of retrieved memory guiding selection.Source: §B.1 · p. 14, Figure 4 · p. 15, Figure 5 · p. 16

B.2 Selection Prompts

• The appendix supplies the full listwise selector prompt and a pairwise prompt for the CHASE-SQL-style baseline (Pourreza et al., 2025).The listwise template takes candidate SQLs, execution results, optional scores, and retrieved memories; the pairwise template compares non-identical execution-result pairs.Source: §B.2 · p. 14, Figure 7 · p. 18, Figure 8 · p. 19

C Usage of AI Assistants

• ChatGPT was used to improve writing clarity and grammatical accuracy by suggesting rephrasings and correcting grammatical errors.Source: C Usage of AI Assistants · p. 14

B.1 Memory Examples and Generation Prompt

• The memory’s output_form selects the person-name candidate over plausible post and over-specified outputs, moving it from rank 18/32 to the top.Execution alone cannot distinguish the near-tied candidates; the memory specifies the queried entity’s name column as the expected output form.Source: Figure 6 · p. 17