PaperlayerView exact revision on arXiv

Source-linked AI summary

Polaris: Learning to Generate Table Descriptions from Retrieval Feedback

Ting Cai, Tuan Minh Phan, AnHai Doan

arXiv:2608.17171v1cs.CLcs.DB

TL;DR

Table retrieval is difficult when cryptic names create vocabulary mismatch, and existing description generators prioritize fluency over retrieval effectiveness. Polaris uses retrieval feedback from labeled benchmarks to train an LLM for retrieval-oriented descriptions, outperforming AutoDDG by up to a 44.6% increase in NDCG@100.

  • Problem

    Table retrieval is difficult because cryptic table and column names hinder keyword search, while prior LLM-generated descriptions have limitations for retrieval.

  • Method

    Polaris generates candidate descriptions, ranks them by retrieval performance on labeled benchmarks, fine-tunes an LLM with DPO, and expands table and column names.

  • Results

    A 44.6% increase of NDCG@100 is reported, with Polaris consistently outperforming AutoDDG across six datasets.

  • Takeaways & Limitations

    Retrieval benchmarks can provide supervision for training metadata generators, extending their role beyond evaluating retrieval systems.

  • Takeaways & Limitations

    Polaris requires query–table relevance judgments and optimizes preferences using BM25 scores, limiting applicability without retrieval supervision or to dense retrieval.

Abstract

from arXiv · show

Many table-centric NLP tasks such as NL2SQL first retrieve relevant tables from large collections using keyword search. Recent work uses LLMs to generate natural-language table descriptions to improve retrieval, but they are typically optimized for fluency rather than retrieval effectiveness. We present Polaris, a system that trains an LLM to generate table descriptions directly from retrieval feedback. Our key insight is that existing table retrieval benchmarks already contain the supervision needed for this task: given query-table relevance judgments, we generate multiple candidate descriptions for each table, rank them by their BM25 retrieval effectiveness, and use the resulting preference pairs to fine-tune the LLM with Direct Preference Optimization (DPO). Polaris further expands abbreviated table and column names before generation to reduce vocabulary mismatch. Extensive experiments show that Polaris outperforms the state-of-the-art AutoDDG solution, often by a significant margin. More broadly, our results demonstrate that retrieval benchmarks can be repurposed as supervision for training LLMs to generate retrieval-oriented metadata.

1 Introduction

Table keyword search is difficult because table and column names are often cryptic, motivating natural-language descriptions and retrieval-oriented metadata. Polaris addresses this through simpler generation, name expansion, preference fine-tuning, metadata-aware retrieval, and a six-dataset benchmark, outperforming AutoDDG.

  • Motivation: Keyword search first narrows large table collections before task-specific reasoning, including NL2SQL query generation.NL2SQL systems may retrieve relevant tables before generating SQL for a natural-language query.
  • Motivation: Cryptic, abbreviated table and column names make keyword search difficult, motivating LLM-generated natural-language table descriptions.Examples include names such as “DTPh” and “ActEmployees.”
  • Polaris: Polaris simplifies description generation to a single LLM call, expands names into full English phrases, and fine-tunes the LLM using labeled retrieval data and DPO.The labeled data ranks descriptions by their usefulness for keyword search, replacing human preference judgments.
  • Polaris: Polaris combines generated descriptions with other table metadata, including name expansions and contexts, to improve keyword-search accuracy.Generated descriptions alone are useful but imperfect.
  • Benchmark: 6 labeled datasets form the largest benchmark to date for evaluating keyword search over tables.The benchmark combines three cleaned major datasets with three newly created datasets and contains table-query-relevance-score triplets.
  • Results: 44.6% is Polaris’s reported maximum NDCG@100 increase over AutoDDG, while average description length falls by 47-59%.Extensive experiments report that Polaris often outperforms AutoDDG by a large margin.

2 Preliminaries

Polaris generates table descriptions from metadata and fine-tunes an LLM using labeled query–table relevance data, while retaining AutoDDG’s BM25-based keyword-search setting. Unlike AutoDDG, Polaris omits table tuples and also considers combining descriptions with other metadata during retrieval.

  • AutoDDG: AutoDDG generates KWS-oriented table descriptions from table names, column names, metadata, sample tuples, statistics, topics, and column summaries through at least four LLM calls.Its pipeline uses the generated description d(T) for retrieval.
  • Table Description Generation in Polaris: Polaris generates descriptions from table names, column names, and available additional context such as captions or Web page titles.This metadata-based generation parallels AutoDDG’s input structure.
  • Table Description Generation in Polaris: Polaris excludes table tuples because they are often unavailable for privacy and legal reasons, yet later outperforms AutoDDG with tuples.The comparison is reported for Section 4.
  • Table Description Generation in Polaris: Polaris fine-tunes its LLM on labeled datasets containing tables, queries, and relevance scores to improve descriptions for keyword search.The labels provide supervision for generating retrieval-oriented descriptions.
  • Keyword Search in Polaris: Polaris adopts AutoDDG’s BM25 TF/IDF keyword-search setting, ranking tables by the score between each description and the query.Embedding-vector and hybrid similarity scores are left for future work, while another setting combines descriptions with other table metadata.

3 The Polaris Solution

Polaris generates retrieval-oriented table descriptions through a two-stage pipeline: expanding cryptic names, fine-tuning an LLM from retrieval-based preferences, and applying it to new tables. It ranks candidate descriptions using labeled query–table judgments and BM25 effectiveness, then trains with DPO.

  • Two-stage pipeline: Polaris first expands table and column names, then generates multiple candidate descriptions and ranks them using labeled datasets before DPO fine-tuning.Name expansion converts abbreviations such as “eSal” into full English phrases, while the LLM also uses available table context.
  • Two-stage pipeline: After training, Polaris expands names in a target corpus and uses the trained LLM to generate one description per table.The second stage is applied to a corpus N over which keyword search is to be enabled.
  • Candidate generation: For gold tables, Polaris samples k ≥2 diverse descriptions, while each non-gold table receives a single description.Gold tables have relevance score greater than 0; sampling uses non-zero temperature and top-p sampling.
  • Retrieval-based ranking: Candidate descriptions are ranked by mean BM25 scores over queries for which the table is relevant, with the highest- and lowest-scoring candidates forming DPO preference pairs.For a gold table relevant to multiple queries, each candidate’s score is averaged across those queries.
  • Efficiency and limitations: Polaris approximates separate per-candidate corpora with one unified corpus, producing negligible BM25-score loss, but currently runs the process once per gold table because repeated reassignment is time consuming.The current procedure assigns random descriptions to non-target tables rather than repeatedly using their best descriptions until convergence.

4 Experiments

Across six labeled datasets, Polaris consistently improves table retrieval over AutoDDG, while ablations show benefits from name expansion and multi-field retrieval. Sensitivity analyses indicate that performance depends on the LLM, DPO regularization, candidate count, and training duration.

  • Overall Performance: Polaris outperforms AutoDDG on the first five datasets across nearly all K values and both NDCG@K and Recall@K measures.On WikiTables, the methods perform similarly.
  • Overall Performance: 47-59% shorter average descriptions distinguish Polaris from AutoDDG.Polaris generates substantially shorter table descriptions than AutoDDG.
  • Ablation Studies: Removing DPO and table/column name expansion still leaves Polaris ahead of AutoDDG on AW, LTER, Arctic, and ECIR.The disabled version performs similarly on WTR and WikiTables, suggesting the simpler prompt remains competitive.
  • Ablation Studies: Name expansion improves retrieval at @5 by +21.2% on LTER, +5.4% on AW, +6.6% on Arctic, and +8.0% on WTR.Its benefit is small on ECIR and WikiTables, which rely on external context rather than table names.
  • Sensitivity Analysis: Changing design choices produces dataset-dependent effects: Qwen improves Polaris on AW, Arctic, and WikiTables but hurts LTER, while β = 0.5 improves AW but hurts LTER.Polaris with Qwen remains superior to AutoDDG with Qwen, and default β = 0.1 offers the best balance across datasets.
  • Multi-Field KWS with Polaris: Multi-field Polaris outperforms single-field Polaris on 5 datasets and remains competitive on 1 dataset.It combines descriptions, name expansions, and table context with a best-field-plus-tiebreaker BM25 score; descriptions remain the most critical field.

5 Related Work

Prior work spans table-metadata generation, keyword-search retrieval, and preference optimization. Polaris complements existing retrieval models by generating table descriptions optimized with automatically derived retrieval-performance preferences rather than human or AI judgments.

  • Metadata Generation for Tables: Table-metadata research covers descriptions, name expansion, semantic column types, and spatial or temporal coverage summaries for downstream tasks.The cited work includes table, dataset, column, and database descriptions, along with table or column name expansion.
  • Keyword Search over Tables: Prior table keyword-search methods improve retrieval through semantic matching, BERT-based content selection, and modeling row and column structure.These approaches focus primarily on retrieval and ranking models rather than generating retrieval-optimized metadata.
  • Keyword Search over Tables: Polaris complements existing retrieval models by generating descriptions optimized for keyword search that can serve as input to any of them.The contribution changes the table representation while remaining compatible with prior retrieval architectures.
  • Keyword Search over Tables: Table-retrieval benchmarks provide labeled relevance data for Polaris’s DPO training, while recent systems address metadata quality through dense retrieval or pseudo-query generation.TARGET reports dense retrievers outperform BM25 but remain sensitive to metadata quality; PIPER generates pseudo-queries from table contents for dense retrieval.
  • Preference Learning and DPO: Unlike standard DPO, Polaris derives preference pairs automatically from keyword-retrieval performance rather than human or AI judgments.Related work applies retrieval signals to query generation or document rewriting, whereas Polaris optimizes table-description generation.

6 Conclusion

Polaris generates table descriptions optimized for keyword search by using retrieval performance to construct preference pairs for DPO fine-tuning, combined with table and column name expansion. Across six datasets, it consistently outperforms the state of the art and suggests that retrieval benchmarks can also train metadata generators.

  • Core approach: Polaris closes the loop between metadata generation and retrieval by using benchmark retrieval performance to construct preference pairs for DPO fine-tuning.The system combines this retrieval-feedback training with table and column name expansion to better match users’ search vocabulary.
  • Experimental conclusion: six datasets spanning enterprise, scientific, government, and web domains show that Polaris consistently outperforms the state of the art with a simpler generation pipeline.The experiments cover diverse domains and report consistent gains over the state of the art.
  • Broader implication: Retrieval benchmarks can serve not only to evaluate retrieval systems, but also to train metadata generators.The paper identifies possible extensions beyond keyword search, including dense retrieval, NL2SQL, and table question answering.

7 Limitations

Polaris depends on retrieval supervision and BM25-based preferences, while generating descriptions from schemas and metadata rather than table contents. Its effectiveness varies across corpora, and extending it to other retrieval objectives and selectively using row-level data remain open directions.

  • Retrieval supervision: Polaris requires query–table relevance judgments for DPO preference construction and is less applicable where retrieval supervision is unavailable.An untrained Polaris variant remains applicable in such settings but does not benefit from retrieval-guided optimization.
  • Retrieval objectives: BM25-derived preferences optimize descriptions for lexical retrieval and may not transfer to dense-embedding or hybrid ranking systems.The paper identifies preference construction for dense or hybrid retrieval objectives as future work.
  • System scope: Polaris optimizes metadata generation rather than the retrieval model, making it complementary to dense retrievers and learned ranking models.These retrieval architectures could potentially benefit from improved table descriptions.
  • Input information: Schema- and metadata-only generation supports settings where table contents are unavailable, but representative rows could yield richer descriptions for some datasets.Future work could selectively exploit row-level data while preserving Polaris’s simplicity and broad applicability.
  • Evaluation scope: Across six benchmark datasets, leave-one-dataset-out evaluation shows encouraging crossdomain generalization, yet retrieval-guided training benefits vary by dataset.The authors call for better understanding of when retrieval feedback is most informative.

A.1 Table/Column Name Expansion

Polaris expands abbreviated column and table names with few-shot prompts that enforce precise, token-level mappings. Table-name expansion additionally uses expanded column names as context, while inference batches columns and bounds table-name input length.

  • Column Name Expansion: The column-expansion prompt enforces eight rules, including expanding abbreviations, preserving numbers and token order, avoiding additions, and producing one short expansion per token.Non-abbreviated tokens remain unchanged.
  • Column Name Expansion: Four few-shot examples teach the model to read table context, split column names into tokens, expand each token, and return a token-level mapping.Mappings are concatenated into one expanded string per column, with 10 column names per prompt by default.
  • Table Name Expansion: The table-expansion prompt reuses the same system message, guidelines, and four examples, but supplies column names as evidence for interpreting the table name.For example, hh_id, age_grp, and edu_lvl support reading Pop_census as “Population Census.”
  • Table Name Expansion: At inference, expanded column names replace originals, and no more than the first 25 columns are included to bound prompt length.Tables without a name skip table-name expansion.

A.2 Initial Table Description Generation

The initial generation stage prompts an LLM to produce concise, schema-grounded table descriptions in strict JSON, illustrated by a sample describing quarterly language-preference data for SSDI claimants.

  • Generation prompt: The prompt instructs the assistant to use only the provided table name, columns, and context, mark non-obvious information as “unknown,” and return valid JSON without surrounding text.The required output is wrapped in triple backticks with the json language tag.
  • Sample output: The sample description identifies quarterly language-preference data for Social Security Disability Insurance claimants from fiscal year 2016 onward and describes related analyses.It includes themes, use cases, keywords, synonyms, and additional context about fiscal-year and quarter organization.

A.3 Ranking the Table Descriptions

Polaris ranks each table’s candidate descriptions by their BM25 retrieval effectiveness on queries for which the table is relevant. It creates a unified corpus in which the target table appears once per candidate description, enabling retrieval-oriented comparison.

  • For each gold table T, Polaris evaluates its K candidate descriptions on the relevant-query set Q(T) = {q : R(q, T) > 0}.
  • The unified corpus contains K copies of T, each with a different candidate description, plus every other table represented once.Other gold tables use one randomly selected candidate description, so the target copies differ only in their descriptions.
  • Polaris averages retrieval effectiveness across relevant queries, giving each query equal weight and one retrieval-oriented score per description.
  • Descriptions are ordered by descending score, with higher-ranked candidates making the correct table score more highly for relevant queries.

A.4 DPO Training with the Rankings

Polaris trains descriptions with DPO using preference pairs selected from retrieval rankings, increasing the winning description’s likelihood relative to the losing one without a separate reward model. Training uses quantized Llama 3.1 8B Instruct with LoRA and 167M trainable parameters.

  • The system ranks descriptions as dπ1 ≻ dπ2 ≻ dπ3 and retains the most separated pair as one training example.
  • DPO increases the winning description’s likelihood relative to the losing description without fitting a separate reward model.
  • DPO uses x = prompt(T), πθ as the trained model, πref as the frozen base model, and β as the regularization control.
  • Training uses 4-bit-quantized Llama 3.1 8B Instruct with LoRA, targeting all attention and feed-forward projections at rank r = 64 and α = 64.
  • LoRA training updates 167M parameters out of 8.2B total (2.05%) for one epoch with β = 0.1, learning rate 5 × 10−6, and effective batch size 8.

A.5 Datasets and Experimental Settings … A.12 Multi-Field Keyword Search

The paper evaluates Polaris across six datasets and multiple retrieval settings, using ablations and sensitivity analyses to examine DPO, name expansion, model choice, training configuration, and multi-field search. Results include comparisons with AutoDDG and variants that remove key components.

  • A.6 Overall Performance: Figure 6 compares the average lengths of descriptions generated by AutoDDG and Polaris on each dataset.The supplied passage identifies the comparison but does not provide the plotted values.
  • A.7 Ablation Studies: Ablation tables evaluate prompt quality and separately measure the effects of DPO training and table/column name expansion using NDCG@K and Recall@K.The compared variants include AutoDDG, P−(exp+DPO), Polaris, P−DPO, and P−exp.
  • A.9 Sensitivity Analysis with Different DPO β: The sensitivity analysis compares default Polaris, using DPO β = 0.1, against P(β=0.5) with β increased to 0.5.Both NDCG@K and Recall@K are reported for this regularization comparison.
  • A.10 Sensitivity Analysis with Different Numbers of Descriptions: The number-of-descriptions study compares Polaris’s default three candidate descriptions per gold table with P(5-desc), which uses five.The comparison reports both NDCG@K and Recall@K.
  • A.11 Sensitivity Analysis with Different Numbers of Training Epochs: The epoch sensitivity analysis compares Polaris trained for one epoch with P(3-epoch), trained for three epochs, using NDCG@K and Recall@K.The supplied passages identify the two training durations and both metric tables.
  • A.12 Multi-Field Keyword Search: Multi-field Polaris is compared with single-field Polaris and AutoDDG, while field ablations remove descriptions, name expansions, or table context.The multi-field evaluation reports NDCG@K and Recall@K, and context ablation applies to ECIR, WTR, and WikiTables.
  • A.8 Sensitivity Analysis with Different LLMs: With Qwen2.5-7B-Instruct, NDCG@5 is .73 for P(Qwen) versus .66 for Polaris on AW, while Arctic reports .29 for both methods.The broader analysis also compares Polaris and AutoDDG under Qwen2.5-7B-Instruct.
Loading 2608.17171v1…