PaperlayerView exact revision on arXiv

Source-linked AI summary

Beyond the Vote: When Do LLMs Add Value to Retrieval-Based Tabular Classification?

Ziquan Wang, Yuxuan Nie, Jian Ma, Yingjie Li

arXiv:2610.29520v1cs.IRcs.CL

TL;DR

Retrieval-based tabular ICL leaves open whether LLM inference improves on voting over the same retrieved examples. The paper compares four general-purpose LLMs with Distance-KNN across 19 datasets and finds Distance-KNN leads in mean accuracy, while LLM gains are selective and neighborhood-dependent.

  • Problem

    It remains unclear how much predictive value LLM inference adds beyond direct voting on the same retrieved examples, since retrieval selection and prediction can otherwise be difficult to distinguish.

    Source: §1 · p. 2

  • Method

    The study compares four general-purpose LLMs with Distance-KNN across 19 datasets using identical retrieved neighbors while varying candidate-pool size.

    Source: §1 · p. 2, §3.3 · p. 4

  • Results

    Distance-KNN has the highest mean accuracy at every candidate-pool size, while some LLMs outperform it in low-consensus neighborhoods depending on model and feature proximity.

    Source: §6 · p. 8

  • Takeaways & Limitations

    The findings characterize when LLM inference adds predictive value beyond distance-weighted voting, with low-consensus gains not extending to the most distant neighborhoods.

    Source: §6 · p. 8

    The study controls retrieved neighbors rather than evaluating independently optimized end-to-end systems or task-specific adaptation, and generalization to other or highly domain-specific benchmarks remains unvalidated.

    Source: Limitations · p. 9

Abstract

from arXiv · show

Retrieval-based in-context learning (ICL) enables large language models (LLMs) to predict tabular labels from similar examples, but it remains unclear whether LLM inference improves on direct voting over those same examples. We address this question by comparing four general-purpose LLMs with distance-weighted nearest-neighbor voting (Distance-KNN) using identical retrieved neighbors across 19 tabular classification datasets. With the number of demonstrations fixed, enlarging the retrieval candidate pool improves accuracy for all predictors, yet Distance-KNN achieves the highest mean accuracy at every pool size. More detailed analyses reveal that LLM gains are concentrated where retrieved neighbors exhibit low label consensus and vary across models. These gains further depend on the average feature-space distance to the test query: under low label consensus, some LLMs outperform Distance-KNN in close and intermediate neighborhoods, but all fall behind on average in the most distant group. Complementary experiments on the number of demonstrations, prompt formulation, retrieval representation, and label-balanced demonstration selection largely preserve the overall findings. These results characterize when LLM inference adds predictive value beyond distance-weighted voting in retrieval-based tabular ICL.

1 Introduction

The paper asks whether general-purpose LLM inference adds predictive value beyond voting over the same retrieved tabular examples. It compares the approaches across 19 datasets and finds that Distance-KNN leads overall, while LLM gains depend on neighborhood structure.

1 Introduction

  • The unresolved question matters because retrieval-based ICL can differ in both which demonstrations are selected and how predictions are made from them.Tabular classification predicts labels from numerical and categorical features and has applications including healthcare, commerce, and energy.Source: §1 · pp. 1–2
  • Under low label consensus, some LLMs gain predictive value, but the gains vary by model and feature proximity.The paper frames label consensus as the majority-label fraction and feature proximity as mean feature-space distance from the query to its neighbors.Source: §1 · p. 2
  • The shared-neighbor comparison controls for demonstration selection by giving Distance-KNN and LLMs the same retrieved examples.The predictors use those examples differently: Distance-KNN uses labels and distances for voting, while LLMs infer from serialized demonstrations.Source: §1 · p. 2
  • Distance-KNN achieves the highest mean accuracy across evaluated candidate pools, while LLMs show selective gains in low-consensus neighborhoods that vary with feature proximity.The paper also examines demonstration count, prompt formulation, retrieval representation, and label-balanced selection; its reported advantage for Distance-KNN largely persists across these experiments.Source: §1 · p. 2

2 Related Work

Prior work ranges from specialized tabular foundation models and trained adaptations of language models to retrieval-based ICL with frozen LLMs. This paper distinguishes itself by comparing frozen LLM inference with Distance-KNN on identical retrieved neighbors.

Tabular Prediction and Tabular Foundation Models

  • Tabular foundation models learn predictive patterns through pretraining, while TabDPT uses a dedicated tabular architecture for in-context prediction.The passage contrasts these approaches with adapting general-purpose LLMs.Source: §2 · p. 2

Adapting Language Models for Tabular Prediction

  • Language-model approaches have used fine-tuning, synthetic-task training, cross-dataset transfer, retrieval-guided tuning, or structure-aware adaptation.This study instead evaluates frozen, general-purpose LLMs without additional task-specific training.Source: §2 · p. 2

Retrieval-Based In-Context Learning with LLMs

  • Retrieval-based ICL work has explored demonstration selection and added information, whereas this study compares LLM inference with Distance-KNN using identical neighbors.The shared-neighbor design focuses the comparison on predictive behavior given the same retrieved evidence.Source: §2 · p. 3, Figure 2 · p. 3

3 Experimental Setup

The experimental setup compares four general-purpose LLMs with Distance-KNN using shared retrieved neighbors and a fixed demonstration budget. It evaluates 19 public datasets under a common preprocessing, retrieval, prompting, and evaluation protocol.

3 Experimental Setup

  • A common retriever supplies the same eight nearest examples to both predictors while candidate-pool size varies.Distance-KNN uses labels and distances; the LLM receives serialized features and labels followed by the query.Source: §3 · p. 3

3.1 Models

  • The comparison uses Qwen3-8B, Qwen3-32B, Gemma3-12B, and Gemma3-27B as the four general-purpose LLMs.Distance-KNN serves as the reference predictor and uses the exact same retrieved neighbors.Source: §3.1 · p. 3

3.2 Datasets

  • The evaluation covers 19 public binary and multiclass datasets from UCI and OpenML, spanning several domains.Datasets use stratified 8:1:1 train, validation, and test splits; accuracy is macro-averaged across datasets.Source: §3.2 · p. 3

3.3 Protocol

  • Candidate pools are nested prefixes of a shared sequence, with class coverage ensured before remaining examples are sampled by class frequency.The evaluated pool sizes range from 16 to 512, plus each dataset’s full training split.Source: §3.3 · p. 4
  • Both predictors retrieve neighbors using mixed-feature distance, while Distance-KNN aggregates their labels with inverse-distance weighting.The LLM receives serialized demonstrations ordered nearest to farthest, but retrieval distances are omitted from its prompt.Source: §3.3 · p. 4
  • Neighborhood structure is assessed by majority-label consensus and mean query-to-neighbor distance; accuracy is averaged over seeds and then datasets.The evaluation also reports mean rank and wins, and uses bootstrap intervals and corrected sign-flip tests for paired comparisons.Source: §3.3 · p. 4

4 Results and Discussion

Across the evaluated candidate pools, Distance-KNN has the highest mean accuracy, while LLM gains are selective: they depend on retrieved-label consensus, model, and feature proximity. Under low consensus, some LLMs outperform Distance-KNN in close and intermediate neighborhoods, but all fall behind in the most distant group.

4.1 Do LLMs Outperform Distance-KNN as the Candidate Pool Expands?

  • 72.81% to 85.50%: Distance-KNN accuracy rises as the candidate pool expands and remains highest at every pool size.The strongest LLM, Gemma3-27B, rises from 72.73% to 85.26%; paired comparisons show Distance-KNN significantly outperforms three of four LLMs after Holm correction.Source: §4.1 · pp. 4–5, Figure 3 · p. 5

4.2 When Does LLM Inference Add Value Beyond Distance-KNN?

  • At 4/8 consensus, three LLMs outperform Distance-KNN, with Gemma3-27B gaining 3.60 percentage points, while Qwen3-8B falls behind.At 5/8–7/8 consensus, Distance-KNN consistently outperforms all LLMs; at 8/8, accuracy gaps approach zero.Source: §4.2 · pp. 5–6

4.3 How Do Label Consensus and Feature Proximity Jointly Shape LLM Gains?

  • At ≤4/8 consensus, three LLMs outperform Distance-KNN in the closest stratum, and all four have positive gaps at the intermediate 40–60 percentile.Gemma3-27B reaches an 8.1% gain in the closest stratum; all four models trail Distance-KNN in the most distant 80–100 percentile, with gaps from −9.8% to −1.6%.Source: Figure 5 · p. 6, §4.3 · p. 6
  • At 5/8–7/8 consensus, Distance-KNN wins in 58 of 60 model–proximity combinations, while full-consensus gaps are at most 0.7%.The only two positive gaps at 5/8–7/8 consensus occur for Gemma3-27B.Source: §4.3 · p. 6, Figure 5 · p. 6

5 Complementary Analyses

Across complementary analyses, Distance-KNN largely retains its advantage, while changing demonstration count, prompt, retrieval representation, and selection can affect LLM performance. The analyses show that prompt and retrieval changes do not consistently close the gap, and label balancing reduces LLM accuracy.

Number of demonstrations

  • Distance-KNN leads in 19 of 21 pool–demonstration-count settings; Gemma3-12B narrowly leads at the two smallest pools with k = 16.Accuracy rises with candidate-pool size for all predictors; Distance-KNN shows no further improvement beyond eight neighbors.Source: §5 · p. 7

Prompt formulation

  • Distance-KNN leads at all seven candidate-pool sizes; Qwen3-8B gains 3.53% with Rule-Augmented NL at 64 candidates, while Gemma3-12B generally performs best with KV.Prompt formulations affect the two LLMs differently, so the default prompt does not alone explain the performance gap.Source: §5 · p. 7, Figure 7 · p. 8

Retrieval representation

  • Feature-space retrieval beats text-embedding retrieval for all three predictors at every pool size; at the full pool, accuracy drops approximately 16 points for Distance-KNN and Gemma3-12B and 24.20 points for Qwen3-8B.Distance-KNN remains highest under both representations.Source: §5 · p. 8, Table 2 · p. 8

Label-balanced demonstration selection

  • Label-balanced selection reduces mean accuracy by 7.49 points for Gemma3-12B and 10.58 points for Qwen3-8B, with declines on all 19 datasets.It replaces some nearby examples with more distant ones, changing both label consensus and feature proximity; enforcing label balance does not improve LLM accuracy over nearest-neighbor selection.Source: §5 · p. 8, Table 3 · p. 8

6 Conclusion

The conclusion evaluates whether LLM inference adds value beyond distance-weighted voting on shared retrieved examples. Distance-KNN leads overall, while some LLMs gain in low-consensus neighborhoods, with results varying by model and feature proximity.

6 Conclusion

  • Across 19 datasets and expanding candidate pools, Distance-KNN has the highest mean accuracy at every pool size, while some LLMs outperform it in low-consensus neighborhoods depending on model and feature proximity.As consensus rises, Distance-KNN generally retains its advantage; no LLM gains extend to the most distant neighborhoods, and complementary analyses largely preserve these findings.Source: §6 · p. 8

Limitations

The study measures general-purpose LLMs against Distance-KNN using the same retrieved neighbors, not independently optimized end-to-end systems or task-specific adaptation.

Limitations

  • The study tests general-purpose LLMs given the same retrieved neighbors as Distance-KNN, rather than independently optimized end-to-end systems.It does not evaluate task-specific model adaptation.Source: Limitations · p. 9
  • Whether the observed patterns generalize beyond the 19 evaluated datasets, including to emerging benchmarks or highly domain-specific datasets, remains open.Source: Limitations · p. 9

Ethical Considerations

The paper reports that its benchmark experiments do not establish real-world deployment suitability. It also describes the evaluated models, inference procedures, and dataset cohorts.

Ethical Considerations

  • Experiments use public benchmark datasets and existing pretrained models, without new data collection or human annotation; they do not establish suitability for real-world deployment.Sensitive-domain applications would require additional reliability and fairness evaluation.Source: Ethical Considerations · p. 9

A.1 Models and inference

  • Generative LLMs remain frozen; predictions use first-token label log-probabilities, while embedding retrieval uses normalized 2,560-dimensional Qwen embeddings and cosine similarity.Source: §A.1 · p. 11

A.2 Datasets and analysis cohorts

  • The main comparison covers 19 datasets, five seeds, seven candidate pools, and five predictors; complementary analyses use the same datasets and three seeds.Label-balanced retrieval uses full pools; the other complementary analyses use all seven pool sizes.Source: §A.2 · p. 11

A.3 Implementation and statistics

The implementation preprocesses features using candidate-pool information and specifies how Distance-KNN handles near-zero distances and ties. The classification prompt presents task and label information, retrieved examples, and the target, requiring a one-number prediction.

A.3 Implementation and statistics

  • Identifier-like and constant columns are removed; numerical missing values use candidate-pool medians, while categorical values are stringified and remaining missing values use a pool-derived mode.Source: §A.3 · p. 11
  • Distance-KNN gives equal votes only to neighbors at distance ≤ 10^-12 and excludes other neighbors when such near-zero neighbors are present; ties favor the first retrieved neighbor in a tied class.Otherwise, class-score ties are resolved by the first retrieved neighbor whose label belongs to a tied class.Source: §A.3 · pp. 11–12
  • Percentile-bootstrap intervals use 10,000 paired dataset resamples, sign-flip tests enumerate 2^19 assignments, and Holm correction covers four LLM comparisons per metric and pool size.Bootstrap intervals are pointwise rather than simultaneous; seed standard deviations use denominator S −1.Source: §A.3 · p. 12

A.4 Classification prompt

  • The prompt supplies task description, feature names, label mapping, ordered top-k examples, and target features, and requires exactly one candidate-label number as output.Source: §A.4 · p. 12
  • The prompt places retrieved examples and the target instance in separate labeled sections and reiterates the one-number-only response requirement.Source: §A.4 · p. 12

B Supplementary Results for RQ1–RQ3

Supplementary analyses report dataset-level accuracy, prediction overrides, and proximity-conditioned comparisons, alongside experiments varying demonstration count, prompt formulation, and retrieval representation. Distance-KNN leads at full pools across tested demonstration budgets, and feature-based retrieval outperforms text-embedding retrieval in all reported predictor-by-pool comparisons.

B.2 RQ2: Complementary predictions and overrides

  • Override rates by consensus average dataset-level rates equally within each seed and then across five seeds, excluding empty groups rather than assigning them zero.The denominator is all queries at that consensus level for each dataset and seed.Source: §B.2 · pp. 12–13

B.1 RQ1: Dataset-level accuracy / B.2 RQ2: Complementary predictions and overrides

  • Table 5 reports full-pool accuracy for Distance-KNN and four LLMs across 19 datasets, while Table 6 reports beneficial and harmful override rates whose difference equals the accuracy gap before rounding.Override rates average five seeds; Figure 8 illustrates beneficial and harmful cases.Source: §B.2 · pp. 12–13, Table 5 · p. 13, Figure 8 · p. 13

B.3 RQ3: Proximity and label composition

  • Distance percentiles rank each query’s mean distance to its eight retrieved neighbors within dataset and seed; lower percentiles mean relatively closer neighborhoods, not absolute cross-dataset proximity.Figure 10 illustrates neighborhoods with the same label consensus but differing proximity.Source: §B.3 · pp. 13–14

C.1 Demonstration count

  • At full pools, Distance-KNN leads for every tested demonstration budget; its accuracy rises 0.80 points from k = 4 to 8, then falls 0.21 at k = 16.Gemma3-12B leads at the two smallest pools with k = 16.Source: §C.1 · p. 14

C.2 Prompt formulation

  • Prompt-formulation comparisons keep the same eight retrieved examples, labels, and order across conditions and use three seeds; Rule-Augmented NL adds rules distilled from candidate-pool data.The LLM makes final test predictions, while an auxiliary XGBoost model assists rule construction.Source: §C.2 · pp. 14–15

C.3 Feature-based versus embedding retrieval

  • Feature-based retrieval has higher accuracy in all 21 predictor-by-pool comparisons; at full pools it improves Distance-KNN and Qwen3-8B on all 19 datasets and Gemma3-12B on 18.Text-embedding retrieval improves Gemma3-12B on KC1 by 2.21 percentage points; Figure 11 reports dataset-level embedding results.Source: §C.3 · p. 15, Figure 11 · p. 15

C.4 Label-balanced selection

Label-balanced selection chooses eight examples from the nearest 32, making label counts as even as candidate availability permits. This selection reduces accuracy for both evaluated LLMs, while the complete-results tables document dataset-level comparisons across candidate-pool sizes.

C.4 Label-balanced selection

  • Balanced selection chooses eight label-balanced examples from the nearest 32, favoring the closest remaining row to resolve ties and ordering the final eight by distance.The comparison matches splits, prompts, and inference settings within each seed.Source: §C.4 · pp. 15–16, Table 8 · p. 16
  • Balancing reduces mean accuracy by 7.49 percentage points for Gemma3-12B and 10.58 for Qwen3-8B, with decreases on every dataset.The selection changes both label composition and the identity and proximity of retrieved rows.Source: §C.4 · p. 16, Table 8 · p. 16

D Complete Accuracy Results

Related papers

Loading 2610.29520v1…