Source-linked AI summary
Ad Hoc Table Retrieval using Semantic Similarity
Shuo Zhang, Krisztian Balog
TL;DR
Ad hoc table retrieval matters as a standalone problem and as a building block for other table-related tasks, but prior approaches relied on lexical matching. The paper introduces semantic matching using concepts and embeddings, and reports substantial, significant improvements over a strong baseline.
Problem
Prior table retrieval relied on lexical matching, and public test collections and proper evaluation methodology were lacking despite the task’s importance to other table-related work.
Source: §1 · p. 1Method
The paper represents queries and tables with concepts and continuous vectors, then matches them using early- or late-fusion similarity measures.
Source: §1 · p. 1, §3.2 · p. 4Results
7.6% to 15.3% relative improvements over the LTR baseline, depending on rank cut-off, were reported for semantic features.
Source: §1 · p. 2, §5.4 · p. 6Takeaways & Limitations
Ad hoc table retrieval is an important standalone task and a core building block in other table-related tasks.
Source: §1 · p. 1The setup assumes specified information is available for every table in the corpus.
Source: §2.2 · p. 2
Abstract
from arXiv · showhide
We introduce and address the problem of ad hoc table retrieval: answering a keyword query with a ranked list of tables. This task is not only interesting on its own account, but is also being used as a core component in many other table-based information access scenarios, such as table completion or table mining. The main novel contribution of this work is a method for performing semantic matching between queries and tables. Specifically, we (i) represent queries and tables in multiple semantic spaces (both discrete sparse and continuous dense vector representations) and (ii) introduce various similarity measures for matching those semantic representations. We consider all possible combinations of semantic representations and similarity measures and use these as features in a supervised learning model. Using a purpose-built test collection based on Wikipedia tables, we demonstrate significant and substantial improvements over a state-of-the-art baseline.
1 INTRODUCTION
The paper frames ad hoc table retrieval as important in its own right and as a building block for other table-related tasks, while identifying gaps in evaluation and lexical-only matching. It introduces semantic query–table matching and reports improvements over a strong baseline using a purpose-built collection.
1 INTRODUCTION
- Ad hoc table retrieval ranks tables for keyword queries and matters both as a standalone problem and as a core building block for other table-related tasks.Related table work includes searching, extracting knowledge, and augmenting tables.Source: §1 · p. 1
- Prior approaches relied on lexical matching, while public test collections and proper evaluation methodology were lacking, motivating semantic matching and a purpose-built evaluation collection.The paper uses graded relevance judgments and establishes a learning-to-rank baseline.Source: §1 · pp. 1–2
- The framework represents queries and tables using concepts and continuous word or entity vectors, with early- and late-fusion strategies yielding four similarity measures.These representations include sparse, discrete concepts and dense, continuous embeddings.Source: §1 · p. 1
- Semantic matching substantially and significantly improves retrieval over a strong learning-to-rank baseline on a purpose-built collection of 1.6M Wikipedia tables with graded judgments.The collection includes graded relevance judgments.Source: §1 · p. 2
2 AD HOC TABLE RETRIEVAL
The section defines ad hoc table retrieval and outlines table representations and ranking methods, from unsupervised document ranking to supervised feature-based ranking.
2.1 Problem Statement
- Given a keyword query, ad hoc table retrieval returns a ranked list of tables, with each table scored independently and results sorted by score.The section also frames the task as assigning a score to each table in the corpus.Source: §2.1 · p. 2
2.2 The Anatomy of a Table
- The assumed table information includes its page and section titles, caption, headings, and body.These fields provide the table and its surrounding page context.Source: §2.2 · p. 2
2.3 Unsupervised Ranking
- A simple unsupervised baseline represents a table as a document and applies standard retrieval methods such as BM25 or language models.Text from all fields can be combined in one representation.Source: §2.3 · p. 2, §2.3.1 · p. 2
2.3.2 Multi-field Document Representation.
- Multi-field ranking scores each table field against the query separately, then combines the field scores using a weighted sum.The field weights sum to 1, and the field-level scores can use standard retrieval methods; the experiments use language models.Source: §2.3.2 · p. 2
2.4 Supervised Ranking
- Supervised ranking organizes retrieval signals as table, query, and query-table features, drawing on prior table-search features and additional features from other retrieval tasks.The paper does not regard the additional document- and entity-ranking features as novel contributions.Source: §2.4 · p. 2
2.4.1 Query Features.
- Query features include query length and field-specific IDF, while table features capture table structure and page context.Examples include row, column, and empty-cell counts, heading PMI, page connectivity and popularity, and a table’s importance within its page.Source: §2.4.1 · p. 2, Query-table features · p. 3, §2.4.2 · p. 3
2.4.3 Query-Table Features.
- Query-table features match query terms against page titles, captions, and table regions, and include parent-page search rank and Mixture of Language Models similarity.The section notes that these features are based on lexical matching and introduces semantic matching as the next topic.Source: §2.4.3 · p. 3
3 SEMANTIC MATCHING
The paper matches queries and tables beyond lexical overlap by representing their content in semantic spaces and comparing those representations. It combines content extraction, semantic representations, and similarity measures in a learning-to-rank framework.
3 SEMANTIC MATCHING
- The framework combines semantic representations and similarity measures into features for learning to rank query–table pairs.It considers all possible combinations of the representations and measures.Source: §3 · p. 3
3.1 Content Extraction
- Raw query and table content is represented as sets of words or entities before terms are mapped to semantic vectors and compared.For word-based content, tables use unique words from titles, captions, and headings; queries use their unique words.Source: §3.1 · p. 3, §3 · p. 3, §3.1.1 · p. 4
3.1.1 Word-based.
- Table entity sets combine core-column entities with entities retrieved from the page title and table caption, while query entities are retrieved from the knowledge base.The table set is Ecc ∪ Rk(Tpt) ∪ Rk(Ttc); the query set is Rk(q).Source: §3.1.2 · p. 4
3.1.2 Entity-based.
- The core column is selected as the column with the highest entity rate, defined as the fraction of its cells containing entities.Entity ranking uses five fields with uniform weights and returns the top 10 entities.Source: §3.1.3 · p. 4, §3.1.4 · p. 4
3.2 Semantic Representations
- Each query or table term receives its own semantic vector, allowing sparse, discrete concept spaces and dense, continuous embedding spaces to share one matching framework.The concept spaces use entities or categories; embeddings cover words and entities.Source: §3.2 · p. 4, §3.2.1 · p. 5
3.2.1 Bag-of-concepts.
- Bag-of-entities vectors mark links between entities, while bag-of-categories vectors mark an entity’s Wikipedia categories.The entity space has millions of dimensions and the category space hundreds of thousands.Source: §3.2.1 · p. 5
3.2.2 Embeddings.
- The embedding alternatives use word vectors for words and graph vectors for entities, based on pre-trained word2vec and RDF2vec representations.The word vectors have 300 dimensions and the graph vectors 200 dimensions.Source: §3.2.2 · p. 5
3.3 Similarity Measures
- Early fusion compares query and table centroids, whereas late fusion aggregates pairwise vector similarities using max, sum, or average.For word embeddings, centroid construction incorporates TF-IDF term weights.Source: §3.3.2 · p. 5, §3.3 · p. 5, §3.3.1 · p. 5
4 TEST COLLECTION
The test collection pairs a large Wikipedia table corpus with queries and graded relevance judgments. Its assessments use multiple trained judges and report moderate inter-annotator agreement.
4.1 Table Corpus
- The collection uses 1.6 million Wikipedia tables, with captions, headings, body, page and section titles, and table statistics.Linked table-body cells are represented by DBpedia identifiers when available, otherwise by anchor text.Source: §4.1 · p. 5
4.2 Queries
- The query set contains 60 queries, split evenly between crowdsourced topics or URLs and class–property queries from Google Squared logs.The class and property fields are concatenated into a single query string.Source: §4.2 · pp. 5–6
4.3 Relevance Assessments
- Three trained judges assess pooled top-20 results on a three-point relevance scale, judging usefulness for creating a table on the query topic.The pool combines results from five baseline methods.Source: §4.3 · p. 6
- The 3,120 test judgments include 377 highly relevant, 474 relevant, and 2,269 non-relevant pairs; inter-annotator agreement is κ = 0.47, described as moderate.Majority vote determines labels, with average scores used when there is no majority.Source: §4.3 · p. 6
5 EVALUATION
The evaluation asks whether semantic matching improves table retrieval and which representations and similarity measures work best. These questions structure the paper’s comparison of semantic-matching choices.
5.1 Research Questions
- The evaluation asks whether semantic matching improves retrieval performance.Source: §5.1 · p. 6
- It also compares the effectiveness of semantic representations and similarity measures.Source: §5.1 · p. 6
5.2 Experimental Setup
The setup evaluates table retrieval with NDCG at four rank cut-offs and uses paired tests for significance. Implementations rely on Nordlys and external resources for selected features and representations.
5.2 Experimental Setup
- Retrieval is evaluated using NDCG at cut-offs 5, 10, 15, and 20, with two-tailed paired t-tests for significance.† and ‡ denote significance at the 0.05 and 0.005 levels, respectively.Source: §5.2 · p. 6
- The implementations are based on Nordlys, and feature computation draws on DBpedia, Wikipedia’s MediaWiki API, and ACSDb.Source: §5.2 · p. 6
- Distributed-representation features use pre-trained embedding vectors, as specified in the semantic-representation section.Source: §5.2 · p. 6
5.3 Baselines
The baselines span single- and multi-field document ranking, prior table-search methods, and a learning-to-rank model. The LTR baseline outperforms the existing literature methods and becomes the comparator for the semantic approach.
5.3 Baselines
- The authors implement four baseline methods from prior work and add an LTR baseline using pointwise Random Forest regression.The LTR model uses 1,000 trees, at most three features per tree, and five-fold cross-validation for NDCG@20; results average five runs.Source: §5.3 · p. 6
- Single-field ranking treats table text as a document, while Pimplikar and Sarawagi score five table fields separately and combine their scores with weighted late fusion.The single-field approaches use language models with Dirichlet smoothing; multi-field ranking uses a Mixture of Language Models with learned field weights and field-specific smoothing.Source: §5.3 · p. 6
- The WebTable method trains linear regression on its features, whereas the WikiTable method trains a Lasso model with coordinate ascent.Both use five-fold cross-validation.Source: §5.3 · p. 6
- The LTR baseline significantly and substantially outperforms all implemented methods from the literature, so later comparisons use it with the same Random Forest settings.The paper emphasizes semantic matching features rather than the supervised learning algorithm.Source: §5.3 · p. 6
5.4 Experimental Results
Adding semantic matching features to the LTR baseline substantially and significantly improves retrieval, while comparisons across representations and similarity measures identify effective combinations. The experiments favor bag-of-entities among representations and late fusion with sum or average aggregation when using all four.
5.4 Experimental Results
- Gradient Boosting and Support Vector Regression show the same general feature-importance patterns as Random Forests, but have lower overall performance.Source: §5.4 · p. 6
- 7.6%–15.3% relative improvement over the LTR baseline, depending on rank cut-off, comes from STR's 16 semantic matching features added to baseline features.The improvement is substantial and statistically significant.Source: §5.4 · p. 6, Table 5 · p. 7
- Bag-of-entities and word embeddings significantly improve over the LTR baseline, and bag-of-entities is the most effective representation overall.All individual semantic features improve over baseline, though not all improvements are statistically significant; Table 6 reports NDCG@20 and the same trends hold at other cut-offs.Source: Table 6 · p. 7, §5.4 · p. 7
- With all four semantic representations, Late-sum and Late-avg achieve the highest overall improvement, while combining all similarity measures adds no further improvement.Each of the four similarity measures significantly and substantially beats the baseline in this setting; differences between measures for a single representation are generally below 5%.Source: §5.4 · p. 7, Table 6 · p. 7
5.5 Analysis
The analysis finds semantic features prominent in the ranker and examines retrieval differences across representations, query subsets, and individual queries. STR improves both subsets consistently, although individual queries show gains and losses.
5.5.1 Features.
- Bag-of-entities and word embeddings have similar query-level difference distributions, but bag-of-entities has fewer queries substantially helped or hurt and a larger overall improvement.The analysis measures NDCG@20 differences; Figure 5's middle bar denotes little change (∆<0.05), and its outer bars denote substantial changes (∆>0.25).Source: §5.5.2 · p. 7, Figure 5 · p. 8
5.5 Analysis
- 8 of the top 10 table-retrieval features are novel semantic features, according to Gini feature importance.Figure 4 distinguishes novel features by color.Source: §5.5.1 · p. 7, Figure 4 · p. 8
5.5.2 Semantic Representations.
- Both LTR and STR perform better on QS-1 than QS-2, while STR consistently improves over LTR on both subsets.QS-2 queries target a specific instance type with a required property and are considered more difficult.Source: §5.5.3 · p. 7, Figure 6 · p. 8
6 RELATED WORK
Prior work spans table search, extension and completion, and table mining. Table search is both a fundamental task and a component used by other table-based tasks.
Table Search
- Table search retrieves tables from web-search results or uses column keywords, page-derived semantics, or result-diversity measures; prior work also searches with a seed table [1, 12, 22].The authors state that they know of no prior work performing semantic matching of tables against queries.Source: §6 · pp. 8–9
Table Extension/Completion
- Table search supports table extension, which adds elements such as columns, and table completion, which fills empty cells.Related methods use table search as an initial step for extension; completion methods extract values from related tables or predict them with machine learning.Source: §6 · p. 9
Table Mining
- Table mining includes extracting table semantics and relations, answering queries with tables, augmenting knowledge bases, and annotating or classifying tables.The paper identifies table search as an essential component in table mining.Source: §6 · p. 9
7 CONCLUSION
The paper reports substantial and significant improvements over a strong baseline for ad hoc table retrieval, using a semantic-matching framework evaluated on a purpose-built Wikipedia-table collection.
7 CONCLUSION
- The proposed methods achieve substantial and significant improvements over a strong baseline on a purpose-built Wikipedia-table test collection.Source: §7 · p. 9
- Queries and tables are represented with semantic concepts and dense word and graph embeddings in a uniform matching framework.The framework uses multiple similarity measures to match those representations.Source: §7 · p. 9
- Ad hoc table retrieval returns a ranked list of tables in response to a keyword query.The authors identify future work in extending the methods beyond Wikipedia tables to scientific and Web tables.Source: §7 · p. 9