Source-linked AI summary
Bridging Search Intent and Tabular Data in Dataset Search
Wenjing Lin, Madelon Hulsebos
TL;DR
Dataset search often depends on keywords users may not know, despite analysts having a clear task in mind. The paper proposes HySE for task-based schema retrieval and FlexPrune for natural-language metadata filtering. On KAGGLEDS, HySE improves recall over semantic-task retrieval and a fine-tuned dense table retriever.
Problem
Dataset search relies largely on matching keywords to metadata, although users may know their analytical objective without knowing the relevant dataset or matching indexed terms.
Source: §1 · p. 1Method
HySE generates task-aligned hypothetical schemas for dense retrieval, while FlexPrune translates natural-language metadata constraints into SQL predicates.
Source: §1 · p. 2, §3.1 · p. 3Results
On KAGGLEDS, HySE consistently outperforms semantic-task retrieval and leads fine-tuned DTR at every cutoff, including +3.9 pp at R@10.
Source: §6.1 · p. 7Takeaways & Limitations
The approach offers a lightweight, generalizable mechanism to bridge user intent to schemas and support metadata filtering through natural language.
Source: §1 · p. 2, §7 · p. 8KAGGLEDS assigns one gold table per query although real tasks may have several valid datasets, so Recall@k may understate methods’ utility unevenly.
Source: Limitations · p. 9
Abstract
from arXiv · showhide
Dataset search remains a bottleneck in data analysis workflows, as existing systems largely rely on keyword matching or shallow retrieval that fail to capture user intent. To facilitate intent-driven search, we present two retrieval components to equip multi-faceted dataset search interfaces through natural language. First, we propose Hypothetical Schema Embeddings (HySE) that uses a language model to generate task-aligned hypothetical table schemas which are used for embedding-based retrieval, bridging the gap between intent and table content. We introduce KaggleDS, a corpus comprising 6,503 tables and diverse task-queries derived from Kaggle, and show that HySE consistently improves recall over strong keyword and embedding-based baselines, as well as over a dense table retriever fine-tuned on KaggleDS. We also show that HySE generalizes to end-to-end question answering on, for example, the NQ-Tables dataset, a corpus of 169,898 Wikipedia tables with human-written queries, where HySE improves Recall@10 by +6.0 pp over dense retrieval baselines. Second, we introduce FlexPrune, which translates dataset constraints expressed in natural-language into SQL predicates over a customizable metadata index, enabling filtering across arbitrary metadata attributes. Combined with HySE, FlexPrune increases Recall@10 by +6.0 pp over dense retrieval followed by tag-based filtering. Our results show that natural-language-based retrieval supporting task and metadata queries offers a viable alternative to schema-level keyword search in dataset search.
1 Introduction
Dataset search systems largely match keywords to metadata, but analysts may know their goal without knowing the relevant dataset or indexed terms. The paper presents natural-language task retrieval and metadata filtering, evaluates them on KAGGLEDS, and reports improved recall and reduced result-set size.
1 Introduction
- Keyword-based retrieval works when users know dataset names or metadata terms, but analysts may have an objective without knowing which indexed keywords match it.Prior work often assumes a known reference table or schema, whereas analysts may begin with an analytical goal.Source: §1 · p. 1
- HySE turns an analytical task into hypothetical schemas for dense retrieval, while FlexPrune translates natural-language constraints into SQL over metadata.The two components support natural-language refinement of dataset search.Source: §1 · p. 2
- HySE improves top-k recall over keyword and dense retrieval baselines and a dense table retriever trained on KAGGLEDS; adding FlexPrune reduces result-set size.The corpus contains 6,503 tables from 3,141 real-world Kaggle datasets.Source: §1 · p. 2
- KAGGLEDS provides a task-oriented corpus of 6,503 tables from 3,141 Kaggle datasets for training and evaluation.The paper evaluates retrieval paradigms and reports that HySE improves recall over keyword, dense, and corpus-trained retrieval baselines.Source: §1 · p. 2
2 Related Work
Prior dataset discovery spans metadata keyword search, semantic and embedding-based retrieval, and benchmarks for table search. The paper identifies a missing resource for training and evaluating task-based dataset search.
2 Related Work
- Metadata keyword systems support dataset lookup, while semantic approaches use language models or learned representations to retrieve tables by similarity.Examples include Google Dataset Search, Starmie, TAPAS, DTR, and SOLO.Source: §2 · p. 2
- Existing benchmarks cover web tables, data lakes, union search, and domain-specific collections, but none supports training and evaluation for task-based dataset search.KAGGLEDS is introduced to address this gap with task-oriented queries over real-world Kaggle datasets.Source: §2 · pp. 2–3
3 Components for Bridging Search Intent and Tabular Datasets
The paper combines task-based semantic retrieval with flexible natural-language metadata refinement. HySE aligns task intent with table schemas, and FlexPrune applies metadata constraints to narrow candidate results.
3.1 HySE: Task-based Semantic Search
- HySE addresses the mismatch between task queries and target schemas by generating schemas rather than expanding queries into free-form text.Its design draws on query expansion and hypothetical document embeddings.Source: §3.1 · p. 3
- HySE uses an LLM to generate task-aligned hypothetical schemas, then embeds and fuses them with the task query for ANN retrieval over corpus table representations.The retrieval compares hypothetical-schema and corpus-schema representations rather than relying on the task query alone.Source: §3.1 · p. 3, Figure 1 · p. 3
3.2 FlexPrune: Flexible Metadata Filtering
- FlexPrune translates natural-language constraints into SQL predicates over a customizable metadata index, avoiding dependence on predefined filter attributes.It filters within semantically aligned candidates to narrow results and improve precision and comprehensibility.Source: §3.2 · pp. 3–4, Figure 2 · p. 4
4 KAGGLEDS for Task-based Search
KAGGLEDS is a Kaggle-derived corpus designed for task-driven dataset search, with metadata and query types supporting retrieval training and evaluation.
4 KAGGLEDS for Task-based Search
- KAGGLEDS was curated because public benchmarks for task-to-table retrieval were missing, using Kaggle datasets suited to analytics tasks and a uniform metadata schema.The resulting corpus is publicly released.Source: §4 · p. 4
4.1 Dataset Construction Pipeline
- The construction pipeline filtered Kaggle datasets for CSV files, permissive licenses, and non-empty descriptions, then split 3,141 datasets and 6,503 tables by dataset.The held-out test set contains 1,257 datasets and 2,337 tables; the split prevents test-set schemas from appearing in training.Source: §4.1 · p. 4
4.2 Metadata Enhancement
- The corpus includes dataset- and table-level metadata, with GPT-4o used to infer temporal and geographic granularity from column headers and example rows.The metadata also includes fields such as dataset descriptions, tags, file size, column headers, and example rows.Source: §4.2 · pp. 4–5
4.3 Task Query Construction
- For each table, KAGGLEDS provides three natural-language task queries, four keyword queries, and metadata-filtering queries for each task-query and metadata-field combination.Task queries are based on dataset descriptions.Source: §4.3 · p. 5
5 Experimental Setup
The setup builds a hybrid index and evaluates HySE on KaggleDS and NQ-Tables against retrieval baselines, using held-out test data and Recall@k. The experiments also examine table structure and dataset cardinality.
5.1 Corpus Index Construction
- The hybrid metadata index combines dense table representations with symbolic attributes for semantic retrieval and SQL filtering.Tables are serialized as Markdown snippets with headers and sample rows; the index also stores structural, descriptive, and auxiliary metadata.Source: §5.1 · pp. 5–6
5.2 Datasets
- The KaggleDS validation split is used only for model selection to choose the HySE configuration; all reported results are computed on the held-out test set of 7,011 task queries and 2,337 tables.The validation portion contains 1,451 tables.Source: §5.2 · p. 6
- HySE is also evaluated on NQ-Tables, a benchmark of 169,898 Wikipedia tables and 11,628 factual human-written questions.Its train, development, and test splits contain 9,594, 1,068, and 966 questions, respectively.Source: §5.2 · p. 6
5.3 Metrics
- Retrieval effectiveness is measured with Recall@k, the fraction of queries whose ground-truth table appears in the top k results..Source: §5.3 · p. 6
5.4 HySE Configuration
- HySE is configured to generate two hypothetical schemas per task query and fuse their mean embedding with the task-query embedding at λ=0.5.The schemas include a table name, column names, and two example rows.Source: §5.4 · p. 6
5.5 Baselines
- The evaluation compares HySE with keyword, embedding-based, and fine-tuned dense table retrieval baselines on held-out KaggleDS queries.The test split contains 7,011 queries over 2,337 tables.Source: §5.5 · p. 6, §5.6 · p. 6
5.6 Evaluation Procedure
- KaggleDS results are partitioned by normalized versus denormalized tables and single-table versus multi-table datasets.These segments assess how schema formulation and dataset cardinality relate to retrieval behavior.Source: §5.6 · pp. 6–7
6 Evaluation Results
The evaluation tests HySE on Kaggle-derived dataset search, human-query tabular QA, and metadata filtering with FlexPrune. HySE improves retrieval over semantic and trained dense baselines, and its combination with FlexPrune improves Recall@10 over query-embedding retrieval with tag filtering.
6.1 Retrieval with Hypothetical Schema Embeddings (HySE)
- HySE leads the trained DTR baseline at every cut-off, including by +3.9 pp at R@10, without training data.It also exceeds semantic-task retrieval by +2.0 pp at R@10 and +2.3 pp at R@50.Source: §6.1 · p. 7, Table 1 · p. 7
- HySE’s Kaggle retrieval is strongest on denormalized and single-table datasets, with single-table R@10 of 0.777 versus DTR’s 0.687.Normalized relational-style tables and one-shot multi-table retrieval remain more challenging.Source: §6.1 · p. 7
6.2 Generalization to Retrieval in Tabular QA
- HySE exceeds DTR-large by up to +6.0 pp at R@30 on NQ-Tables, despite using no NQ-Tables supervision.The comparison is conservative because DTR is fine-tuned on the training split; the unchanged HySE setup also transfers to factual queries.Source: §6.2 · pp. 7–8, Table 2 · p. 8
6.3 HySE with FlexPrune Metadata Filtering
- FlexPrune converts natural-language metadata constraints to SQL and filters HySE’s top-50 candidates while preserving their order.The filtering step intersects metadata-query results with the HySE candidate set.Source: §6.3 · p. 8
- Recall@10 rises from 0.644 to 0.704, a +6.0 pp gain over query-embedding retrieval followed by tag filtering.The authors report the gain with one additional refinement interaction; the combined approach avoids requiring users to enumerate columns or understand schemas upfront.Source: §6.3 · p. 8, Table 3 · p. 8
7 Conclusion
The paper proposes HySE to bridge task intent and table schemas, and FlexPrune to translate metadata queries into SQL. On KaggleDS, NQ-Tables, and a chained retrieval-and-filtering evaluation, the authors report improved recall over the respective baselines.
7 Conclusion
- HySE hypothesizes relevant schemas for a task to bridge user intent and target schemas, while FlexPrune parses metadata queries into SQL for an available metadata index.The paper introduces KAGGLEDS, a task-based dataset-search corpus.Source: §7 · p. 8
Limitations
The authors identify limitations in relevance judgments, corpus diversity, and scaling evidence.
Limitations
- Each KAGGLEDS query has one gold table, so Recall@k may understate method utility when several datasets satisfy a task.The authors propose graded relevance judgments over pooled candidates as future work, especially for ranking quality at small k.Source: Limitations · p. 9
- Kaggle’s 6,503-table corpus has rich author metadata, while noisy, sparsely documented enterprise and web-table settings are not evaluated.Although NQ-Tables has 169,898 tables, the authors report no scaling curve; metadata-dependent FlexPrune may face materially harder conditions in sparse-metadata settings.Source: Limitations · p. 9
Use of Generative AI
The paper discloses generative AI use in its methods and corpus construction, then documents prompts for inferring metadata, generating queries and schemas, supporting iterative search, and translating constraints into SQL.
Use of Generative AI
- LLMs generate HySE schemas, translate FlexPrune constraints into SQL, and enhance KaggleDS metadata and queries; AI tools also assisted with code and manuscript editing.The authors state that AI did not inform research ideas, system design, evaluation protocol, or results analysis, and that they reviewed and verified AI-assisted code and text.Source: Use of Generative AI · p. 9
A LLM Prompts
- The appendix supplies the complete system prompts, including prompts for content-granularity inference and task-query generation.The granularity prompt selects from temporal and geographic categories or leaves the result empty when it cannot be inferred from the table.Source: A LLM Prompts · p. 11, §A.1.1 · p. 11
A.2.1 Task Query Generation: Single-Table Databases; A.2.2 Task Query Generation: Multi-Table Databases
- Single-table and multi-table prompts request four content keywords and three analytics-task queries for each dataset or CSV file.The prompts distinguish expected data content from specific analytical tasks.Source: §A.2.1 · p. 11, §A.2.2 · p. 11
A.2.3 Metadata Query Generation Prompt
- Metadata-query prompts express task requirements against available attributes, with numeric examples constrained to plausible values for the metadata.For example, a dataset with 4,836 rows can yield a request for at least 2,000 rows, but not 10,000.Source: §A.2.3 · p. 11
A.3.1 Action Inference Prompt; A.3.2 Field Inference Prompt
- The action prompt distinguishes a reset, which changes analytical focus, from a refine, which builds on or slightly alters the previous query.The field-inference prompt identifies referenced semantic or metadata fields, using separate candidate lists for each group.Source: §A.3.1 · p. 11, §A.3.2 · p. 12
A.4.1 Normalized Hypothetical Schema Prompt; A.4.2 Denormalized Hypothetical Schema Prompt; A.4.3 Normalized Hypothetical Schema Prompt (with Example Rows); A.4.4 Denormalized Hypothetical Schema Prompt (with Example Rows)
- Schema prompts request task-specific normalized or denormalized tables, with the latter favoring descriptive columns and directly included attributes.The example-row variants request realistic examples and specify either relational design with IDs or a single comprehensive table without foreign-key relationships.Source: §A.4.1 · p. 12, §A.4.2 · p. 12, §A.4.3 · p. 12
A.5.1 NL-to-SQL Metadata Predicate Prompt
- The NL-to-SQL prompt converts identified fields and a user query into structured SQL clauses, with separate clauses for multiple tags.It specifies allowed temporal and geographic values and numeric comparison operators, and requires parseable JSON output.Source: §A.5.1 · pp. 12–13
B Kaggle Dataset Sample
The example presents a KaggleDS instance with dataset metadata and generated queries alongside the keyword queries used by baselines.
B Kaggle Dataset Sample
- A KaggleDS sample juxtaposes dataset metadata, task and metadata queries, and baseline keyword queries.The figure caption identifies these as the three components of the example.Source: B Kaggle Dataset Sample · p. 13
C.1 Ablation Experiment
The ablations examine schema formulation, schema count, candidate-table representation, and query fusion. Joint fusion performs best overall in the reported validation comparison, while the selected configuration's strongest test-set performance is on denormalized tables.
Ablations
- Normalized schemas yield the highest validation recall across k, while N ∈{1, 2} performs strongly with minimal compute overhead and table name plus header plus two rows is the most effective candidate representation.The ablation table varies one design dimension at a time while holding the others at the selected configuration.Source: §C.1 · p. 13, Table 4 · p. 13
- HySE's strongest test-set performance is on denormalized tables, which the authors say dominate open-data corpora.They hypothesize that wider headers and example rows expose richer semantic signals that align with hypothetical schemas, while joint fusion preserves task phrasing.Source: §C.1 · pp. 13–14
HySE Variation Experiment
- +1.4 pp at R@10 and +1.2 pp at R@50: joint fusion improves recall over schema-only embeddings.Table 5 compares the variants with N=2 normalized schemas and a candidate corpus encoded as table name, header, and two example rows.Source: Table 5 · p. 14, §C.1 · p. 14
- Schema-only HySE exceeds the Semantic–Task baseline by +0.6–1.5 pp across R@10–50.The authors interpret this as evidence that generated schema structure alone provides strong retrieval alignment.Source: §C.1 · p. 14
- Fusion's largest reported gain is at R@10, where retaining task phrasing alongside structural cues may sharpen top-ranked disambiguation.The authors suggest this is useful when users rely heavily on the first page, with negligible added cost.Source: §C.1 · p. 14