PaperlayerView exact revision on arXiv

Source-linked AI summary

STaRK: Benchmarking LLM Retrieval on Textual and Relational Knowledge Bases

Shirley Wu, Shiyu Zhao, Michihiro Yasunaga, Kexin Huang, Kaidi Cao, Qian Huang, Vassilis N. Ioannidis, Karthik Subbian, James Zou, Jure Leskovec

arXiv:2404.13207v3cs.IRcs.LG

TL;DR

Real-world retrieval often requires answering queries that combine textual and relational information, while prior work has generally studied these separately. STARK builds a benchmark of realistic queries and ground-truth answers across three domains; experiments find that current retrieval models still face substantial challenges handling both information types.

  • Problem

    Prior work focused on textual retrieval or structured SQL and knowledge-graph queries, leaving open how well LLM retrieval handles queries requiring both forms of information in semi-structured knowledge bases.

    Source: §1 · p. 2

  • Method

    STARK builds three semi-structured knowledge bases and uses a pipeline to synthesize realistic queries with precise answers, validating query quality and adding 274 human-generated queries.

    Source: §1 · p. 2

  • Results

    Experiments show that current retrieval models face significant challenges handling textual and relational information on STARK.

    Source: §5 · p. 10

  • Takeaways & Limitations

    STARK provides a testbed for retrieval research focused on reducing latency and enhancing reasoning for complex retrieval systems.

    Source: §5 · p. 10

    The current knowledge bases cover only textual and relational information, and privacy remains a concern when extending to other domains with real-user data.

    Source: §5 · p. 10

Abstract

from arXiv · show

Answering real-world complex queries, such as complex product search, often requires accurate retrieval from semi-structured knowledge bases that involve blend of unstructured (e.g., textual descriptions of products) and structured (e.g., entity relations of products) information. However, many previous works studied textual and relational retrieval tasks as separate topics. To address the gap, we develop STARK, a large-scale Semi-structure retrieval benchmark on Textual and Relational Knowledge Bases. Our benchmark covers three domains: product search, academic paper search, and queries in precision medicine. We design a novel pipeline to synthesize realistic user queries that integrate diverse relational information and complex textual properties, together with their ground-truth answers (items). We conduct rigorous human evaluation to validate the quality of our synthesized queries. We further enhance the benchmark with high-quality human-generated queries to provide an authentic reference. STARK serves as a comprehensive testbed for evaluating the performance of retrieval systems driven by large language models (LLMs). Our experiments suggest that STARK presents significant challenges to the current retrieval and LLM systems, highlighting the need for more capable semi-structured retrieval systems. The benchmark data and code are available on https://github.com/snap-stanford/STaRK.

1 Introduction

Real-world questions combine free-form needs with structured constraints, but prior retrieval work has largely treated textual and relational information separately. STARK addresses this gap with realistic queries and ground-truth answers across diverse domains.

1 Introduction

  • Complex information needs combine free-form descriptions and constraints, and answering them supports user experience, informed decisions, and hallucination prevention.Examples include product searches and medical questions about gene-associated disease and symptoms.Source: §1 · p. 2
  • Prior work studied textual retrieval and structured SQL or knowledge-graph queries separately, leaving LLM retrieval on semi-structured knowledge bases an open question.These knowledge bases combine natural-language descriptions with entity relationships, and queries can require both.Source: §1 · p. 2
  • STARK uses a novel pipeline to simulate user queries and construct precise ground-truth answers across three knowledge bases.The queries are validated for naturalness, diversity, and practicality, and the benchmark includes 274 human-generated queries.Source: §1 · p. 2
  • STARK spans product recommendation, academic paper search, and precision-medicine inquiries, with queries requiring context-specific reasoning.The benchmark highlights challenges in handling textual and relational data and latency on knowledge bases with millions of entities or relations.Source: §1 · p. 3

2 Benchmarking Retrieval Tasks over Textual and Relational Knowledge

STARK defines retrieval over knowledge bases that combine graph relations with node-associated text, then builds three domain-specific datasets of queries requiring both kinds of information. Its construction pipeline, query analyses, and human evaluation characterize the benchmark’s scope and quality.

2.1 Problem Definition

  • Each SKB pairs a knowledge graph with free-text documents associated with nodes, and retrieval answers must satisfy both query-specified relational and textual requirements.Documents can include product descriptions and reviews; the three constructed SKBs cover product, academic-paper, and biomedical entities.Source: §2.1 · p. 3, §2.2 · p. 4

2.2 Semi-structured Knowledge Bases (SKBs)

  • Amazon, MAG, and Prime provide three constructed SKBs with different entity and relation types and textual sources.Prime is denser and has more relation types than Amazon and MAG.Source: §2.2 · pp. 3–4

2.3 Retrieval Tasks on Semi-structured Knowledge Bases

  • STARK comprises Amazon product recommendation, MAG academic-paper search, and Prime biomedicine datasets, with relational and textual requirements tailored to each domain.Amazon includes single-hop relations and 68% of synthesized queries have multiple ground-truth answers; MAG includes multi-hop paths, while Prime uses 28 multi-hop templates and has relational-only queries for some entity types.Source: §2.3 · p. 4

2.4 Benchmark Construction: Synthesized Queries

  • The synthesis pipeline samples expert-informed relation templates, extracts role-relevant text properties from a gold entity, generates contextual queries with two LLMs, then verifies candidate answers.Gold answers passed verification at average rates of 86.6%, 98.9%, and 92.3% across the three datasets.Source: §2.4 · p. 5

2.5 Synthesized Data Distribution Analysis and Human Evaluation

  • Most queries are around 16 words, while about 22% of STARK-AMAZON queries have answers with more than 30 entities.Longer queries, up to 50 words, often mention other entities or provide detailed context.Source: §2.5 · p. 5
  • Across datasets, query word-distribution entropy is high and TTR is steady; relational-to-textual word ratios vary, but do not directly measure answer importance.Table 4 reports the diversity measures, and Figure 5 depicts average relative composition of relational versus textual information.Source: §2.5 · p. 6, Table 4 · p. 6, Figure 5 · p. 6
  • 94.1%, 85.3%, and 89.4% of participants rated synthesized queries neutral or above for naturalness, diversity, and practicality, respectively.The evaluation involved 63 participants.Source: §2.5 · p. 6

2.6 Benchmark Construction: Human-Generated Queries

  • Human-generated queries were manually checked for answer correctness; their answer counts were comparable to synthesized queries, but their expressions were often more unique.Thirty-one participants generated 263 queries across the three SKBs.Source: §2.6 · p. 6

3 Experiments

The experiments compare sparse, dense, multivector, and LLM-reranking systems on synthesized and human-generated queries. Results show that retrieval quality remains challenging, with performance varying across models, query sets, and latency requirements.

3.1 Baseline Retrieval Models and Evaluation Metrics

  • The evaluated systems span sparse BM25, compact and LLM-based dense retrievers, multivector retrievers, and LLM rerankers.The rerankers score candidate entities from 0 to 1 using textual and relational information; evaluation uses a random 10% sample because of their high computational cost.Source: §3.1 · p. 7
  • Evaluation uses Hit@1 and Hit@5 for top-result accuracy and Recall@20 for the proportion of relevant items retrieved on synthesized queries.The study also reports MRR, which emphasizes the rank of the first relevant result.Source: §3.1 · p. 7

3.2 Results and Analysis

  • BM25 outperforms dense models such as ANCE on synthesized queries, while finetuned DPR and QAGNN show insufficient performance.The authors suggest small model size and overfitting may contribute to the difficulty of training on SKBs, where entities can be hard to distinguish without detailed information.Source: §3.2 · p. 8
  • Across larger and multivector models, GritLM-7b surpasses ada-002 overall, multi-ada-002 generally beats ada-002, and ColBERTv2 improves retrieval accuracy.The paper links ColBERTv2’s gains to its fine-grained representations and multi-ada-002’s gains to using multiple vectors per document.Source: §3.2 · p. 8
  • On the random split, LLM rerankers generally outperform GritLM-7b and ColBERTv2, especially on Hit@k metrics.The paper attributes rerankers’ advantage to re-evaluating candidates using deeper understanding of query and document content.Source: §3.2 · p. 8
  • Despite improving top-rank precision, rerankers remain suboptimal: GPT-4 reaches only about 18% Hit@1 on STARK-PRIME and 41% on STARK-MAG.GPT-4 Recall@20 is 55% on STARK-AMAZON, 49% on STARK-MAG, and 34% on STARK-PRIME; its STARK-PRIME MRR is around 27%.Source: §3.2 · p. 9
  • On human-generated queries, rerankers consistently outperform other models, and most models perform better than on synthesized queries, though overall trends remain consistent.The authors characterize the synthesized queries as more challenging.Source: §3.2 · p. 9
  • On STARK, measured using a single NVIDIA A100-SXM4-80GB GPU, DPR and QAGNN exhibit lower average latency, while ada-002 and multi-ada-002 have moderate latency due to multiple API calls.Source: §3.2 · p. 9

4 Related Work

Prior QA datasets separately emphasize unstructured text, structured graphs or tables, and semi-structured table-text combinations. STARK is positioned as retrieval over richer entity relations and textual content together.

Unstructured QA Datasets

  • Unstructured QA datasets cover single- and multi-document text, but often lack the relational reasoning needed for complex user queries.Examples include SQuAD [38], HotpotQA [54], and TriviaQA [24].Source: §4 · p. 10

Structured QA Datasets

  • Structured QA datasets use graphs or tables, but predefined entities and relationships limit query scope.ComplexWebQuestions [47] and GraphQA [16] address graph QA, while WikiSQL [60] and Spider focus on table querying.Source: §4 · p. 10

Semi-Structured QA Datasets

  • Semi-structured QA datasets combine table structures and text, but often lack rich entity relations or rely on cumbersome external links.STARK integrates textual content within structured frameworks for retrieval across complex semi-structured knowledge bases.Source: §4 · p. 10

5 Conclusion and Future Work

STARK introduces a benchmark for LLM-driven retrieval over semi-structured knowledge bases, with three retrieval datasets, human-generated queries, and an automated query-simulation pipeline. Its current knowledge bases cover textual and relational information, while future work points to additional modalities and privacy concerns when using real user data.

5 Conclusion and Future Work

  • STARK is presented as the first benchmark to thoroughly evaluate LLM-driven retrieval systems for semi-structured knowledge bases.It includes three large-scale retrieval datasets with human-generated queries and an automated pipeline for simulating realistic queries.Source: §5 · p. 10
  • Experiments highlight challenges for current models in handling textual and relational information.The conclusion identifies reducing retrieval latency and enhancing reasoning abilities as future research directions.Source: §5 · p. 10
  • The current SKBs are limited to textual and relational information; extending the work to other domains with real user data raises privacy concerns despite anonymization efforts.The authors propose incorporating images, videos, and speech in future work and say privacy should be protected to comply with regulations.Source: §5 · p. 10

Checklist

The checklist reports that the authors describe the work’s scope, limitations, and potential negative impacts, and provide experimental code, data, and pipelines. It also records asset attribution and licensing information.

Checklist

  • The authors state that the paper identifies its scope and contribution, discusses limitations, and addresses potential negative societal impacts.The checklist directs readers to the conclusion and future work for limitations and societal impacts.Source: Checklist · p. 15
  • The authors report that their website and GitHub repository provide code, data, and experimental pipelines.They also report making data splits public and describing training details in the experiment setup.Source: Checklist · p. 15
  • The checklist reports that existing assets are cited, their licenses are mentioned, and the resources come from publicly accessible data.The Amazon semi-structured dataset is identified as the only resource with potential personally identifiable information or offensive content, and the public resources are anonymized.Source: Checklist · p. 15

A.1 Semi-structured Knowledge Bases (SKBs)

The three SKBs draw on public sources, and the resource licenses are respected. Their entity and relation counts vary substantially across Amazon, MAG, and Prime.

A.1 Semi-structured Knowledge Bases (SKBs)

  • The SKBs are constructed from public sources, whose licenses the authors report respecting.The cited table identifies the sources of relational structure and textual information.Source: §A.1 · p. 16, Table 9 · p. 16
  • The Amazon SKB contains around 1.0M entities and 9.4M relations.Its entities include products, brands, categories, and colors; relations include also_bought, also_viewed, and product attributes.Source: §A.1 · p. 16
  • The MAG SKB contains around 1.9M entities and 39.8M relations, while Prime has approximately 129.3K entities and 8.1M relations.MAG covers authors, papers, institutions, and fields of study; the passage does not enumerate Prime’s entity types.Source: §A.1 · p. 16

A.2 STARK-AMAZON

STARK-AMAZON’s final relational templates can combine multiple basic templates.

A.2 STARK-AMAZON

  • STARK-AMAZON relational query templates can compose multiple basic templates, such as color → product ← brand.The example combines two basic relational templates.Source: §A.2 · p. 16

A.3 STARK-MAG

STARK-MAG and STARK-PRIME use relational templates to construct queries, while the human-query instructions ask participants to consider both graph structure and text. The appendix also defines entropy and type-token ratio as measures of query diversity.

A.3 STARK-MAG

  • STARK-MAG provides seven relational templates, including multi-hop paths, with example queries for papers, authors, institutions, and fields of study.Some templates require multiple starting entities, such as an institution and a field of study.Source: §A.3 · pp. 16–17

A.4 STARK-PRIME

  • STARK-PRIME uses 28 expert-designed multi-hop relational templates intended to cover varied relation types and practical queries.Examples include queries linking anatomy, genes or proteins, and drugs, and queries connecting diseases with contraindicated drugs.Source: §A.4 · p. 17

B Mathematical Definitions of Shannon Entropy and Type-Token Ratio

  • Shannon entropy measures outcome uncertainty, while TTR is the ratio of unique words to total words in a text.The passages state that higher entropy indicates greater outcome diversity and higher TTR indicates greater lexical diversity.Source: B Mathematical Definitions of Shannon… · p. 18

C Instructions for Generating Queries

  • Human query writers examine an entity’s documents and connected entities or multi-hop paths, then draw on relational and textual aspects to write queries it satisfies.They may skip entity IDs they find difficult to understand; collected queries undergo human validation of ground-truth answers.Source: C Instructions for Generating Queries · pp. 18–19

D.1 More Experimental Results

The appendix presents human-evaluation and LLM-version tables alongside prompts for extracting requirements, fusing them into queries, and filtering additional answers. The supplied passages describe these procedures but do not report the tables’ results.

D.1 More Experimental Results

  • Table 10 is identified as reporting positive and non-negative rates from human evaluation.The supplied passage gives the table’s scope but no rates.Source: Table 10 · p. 19

E Prompts and LLM versions for Query Synthesization

Prompt for STARK-AMAZON: Textual requirement extraction

  • The Amazon extraction prompt requests a generic product category, negative perspectives, and up to five hard and five soft customer requirements with sources.Negative perspectives are drawn from reviews or Q&A, while requirements focus on product advantages.Source: Prompt for STARK-AMAZON: Textual… · p. 20

Prompt for STARK-MAG: Textual requirement extraction

  • The MAG extraction prompt asks for one general, short paper requirement that researchers might find interesting.It allows requirements about paper content, publication date, or venue.Source: Prompt for STARK-MAG: Textual… · p. 21

Prompt for STARK-PRIME: Textual requirement extraction

  • The PRIME extraction prompt requests role-relevant properties and their sources in a specified list format.The supplied prompt excerpt limits properties to no more than 10 words and asks them to begin with the target name.Source: Prompt for STARK-PRIME: Textual… · p. 21

Prompt for STARK-AMAZON: Fuse relational and textual requirements

  • The Amazon fusion prompt combines non-conflicting textual requirements and relational information into a natural customer query leading to the item.It says to exclude requirements conflicting with negative reviews and to express product relations through scenarios such as compatibility or style matching.Source: Prompt for STARK-AMAZON: Fuse… · p. 22

Prompt for STARK-MAG: Fuse relational and textual requirements

Prompt for STARK-PRIME: Fuse relational and textual requirements

Related papers

Loading 2404.13207v3…