PaperlayerView exact revision on arXiv

Source-linked AI summary

A Comparative Survey of Recent Natural Language Interfaces for Databases

Katrin Affolter, Kurt Stockinger, Abraham Bernstein

arXiv:1906.08990v1cs.DBcs.CLcs.LG

TL;DR

Existing database NLIs had not been systematically compared for functionality and expressive power. The paper surveys 24 systems, evaluates them with ten sample questions, and derives lessons about their capabilities and design. It finds that simple questions suit keyword systems, while complex questions require parsing and grammar-based systems are most powerful but depend on manually designed rules.

  • Problem

    Database NLIs lacked a systematic comparison using common benchmark questions to assess their functionality and expressive power.

  • Method

    The paper analyzes 24 recent NLIs using ten sample questions of increasing complexity and categorizes them by methodology.

  • Results

    Simple filter questions can be answered by all systems, whereas the most difficult questions are answered completely by only a few systems.

  • Takeaways & Limitations

    Grammar-based NLIs can guide users while they formulate questions, improving interaction between systems and users.

  • Takeaways & Limitations

    The evaluation is theoretical and based on published papers because several systems, especially older ones, are no longer available.

Abstract

from arXiv · show

Over the last few years natural language interfaces (NLI) for databases have gained significant traction both in academia and industry. These systems use very different approaches as described in recent survey papers. However, these systems have not been systematically compared against a set of benchmark questions in order to rigorously evaluate their functionalities and expressive power. In this paper, we give an overview over 24 recently developed NLIs for databases. Each of the systems is evaluated using a curated list of ten sample questions to show their strengths and weaknesses. We categorize the NLIs into four groups based on the methodology they are using: keyword-, pattern-, parsing-, and grammar-based NLI. Overall, we learned that keyword-based systems are enough to answer simple questions. To solve more complex questions involving subqueries, the system needs to apply some sort of parsing to identify structural dependencies. Grammar-based systems are overall the most powerful ones, but are highly dependent on their manually designed rules. In addition to providing a systematic analysis of the major systems, we derive lessons learned that are vital for designing NLIs that can answer a wide range of user questions.

1 Introduction

Natural language interfaces make database access easier for non-experts, but their expressive power and handling of ambiguity require systematic comparison. This paper addresses that need by analyzing 24 systems with ten increasingly complex sample questions and deriving design lessons.

  • Structured query languages are difficult for non-experts because they require schema knowledge, entity roles, and precise join paths.
  • NLIs let users query databases in natural language without knowing the underlying schema or formal query language.
  • Natural-language ambiguity can occur at both the word and sentence levels, requiring interpretation checks, dataset validation, and user clarification.
  • The paper compares 24 recent NLI systems using ten representative questions of increasing expressive complexity and a shared sample world.
  • The evaluation omits machine-learning approaches because their capabilities and domain dependence vary with training data, making comparison difficult.

2 Foundation: A Sample World

The survey builds a movie-database sample world and ten increasingly complex questions to compare how NLIs handle SQL/SPARQL operators and linguistic variation. Corpus analyses indicate that these questions cover many common question types, while some open-ended or unsupported types remain outside the mapping.

  • Sample world: The sample world is a movie database with movies, persons, relationships, hierarchical roles, attributes, genres, country-specific profits, and rating-based concepts.It is inspired by IMDB and includes both base data and ontology metadata.
  • Question set: The ten questions span joins, string and range filters, date filters, aggregations, ordering, unions, subqueries, negation, and ontology concepts.They were designed from SQL and SPARQL operators and cover technical and linguistic challenges.
  • Question set: The questions increase in complexity from identifying filters and joins to handling aggregations, negation, concepts, and nested subqueries.Examples include translating higher than into a comparison, recognizing great movie as rating >= 8, and splitting a subquery into two steps.
  • Question analysis: More than 40% of Yahoo! questions correspond to filtering-only Q1, while about 30% are off-topic and none require a numerical range.The analysis maps 100 Yahoo! movie questions to the ten sample questions.
  • Question analysis: About 88% of GeoData250 questions map to Q1 or Q4, while concepts occur in roughly 8% and several operator types do not occur.The absent types include numerical ranges, dates, unions, and multiple subqueries.
  • Question analysis: The ten sample questions represent a large range of questions typically posed in question-answering systems, excluding Yahoo! open-ended questions.The mapping assigns each question to the label representing its principal answering challenge.

3 Background: Natural Language Processing Technologies

The survey reviews preprocessing and parsing technologies used by database NLIs, from tokenization and lexical normalization to syntactic analysis. These technologies preserve or expose linguistic structure needed to translate natural-language questions into formal queries.

  • Overview: Relevant NLI preprocessing includes stop-word handling, synonym resolution, tokenization, part-of-speech tagging, stemming, lemmatization, and parsing.The section presents these as common NLP technologies for database interfaces.
  • Lexical processing: Stop words can encode relationships needed for computation, so NLIs may need to identify them without removing them.In “What was the best movie of each genre?”, the phrase “of each” signals an aggregation structure.
  • Lexical processing: Synonym handling is necessary because different natural-language expressions can refer to the same database relation.“Starring Brad Pitt” and “playing Brad Pitt” should produce the same answer even when only one wording appears in the sample world.
  • Lexical processing: Tokenization separates input into tokens and must handle punctuation that simple whitespace splitting may leave attached to words.The sample questions end with punctuation, making boundary detection relevant to later processing.
  • Syntactic processing: Part-of-speech tagging assigns grammatical categories using tokens and context, supporting later lemmatization and dependency parsing.Stanford CoreNLP’s dependency-tree processing requires part-of-speech tags.
  • Normalization: Stemming heuristically removes word endings, whereas lemmatization uses vocabulary and morphology to return dictionary forms while considering part of speech.For example, lemmatization distinguishes the verb “direct” from the adjective “direct.”
  • Syntactic processing: Parsing analyzes grammatical structure through dependency relations or constituency structures represented as syntax trees.NLIs can use the relations and constituents identified by these two parsing directions.

4 Limitations

The evaluation is bounded by a theoretical, paper-based comparison of ten SQL/SPARQL-derived questions. It excludes computational performance and accuracy because the surveyed studies use incompatible settings and metrics.

  • Scope: The evaluation focuses on ten sample questions based on SQL and SPARQL operators, limiting its coverage to those question forms.The questions define the evaluation scope rather than exhaustively representing all database queries.
  • Evaluation basis: The comparison is theoretical and based entirely on system papers because some systems are unavailable and only a few provide online demonstrations.This choice was made to evaluate all systems under the same access conditions.
  • Evaluation basis: Computational performance is completely excluded because the systems have different goals, including speed, efficiency, and interaction optimization.The evaluation therefore does not compare runtime or the number of user interactions.
  • Evaluation basis: Accuracy is excluded because papers differ in whether they report it, how they define metrics, dataset sizes, and preprocessing.These differences make direct comparison of reported accuracy metrics impossible.

5 Recently Developed NLIs

The survey systematically analyzes 24 recently developed database NLIs and groups them by technical approach. It uses a common data model and question set to compare the strengths and weaknesses described in the original papers.

  • Scope: The survey analyzes 24 recently developed NLIs, focusing on systems from 2005 onward rather than older historically cited approaches.The systems are examined in Sections 5.1 to 5.4 using the sample world.
  • Comparative analysis: The systems are compared using one data model and particular questions so their approaches can be assessed directly from their published descriptions.The analysis aims to highlight strengths and weaknesses and provides an overall interpretation against the sample questions.
  • Classification: The survey divides NLIs into four main groups according to the technical approach they use.The classification provides the organizing framework for the later system analysis.

1. Keyword-based systems

Keyword-based systems match input keywords against inverted indexes of database base and metadata. They are simple and adaptable but cannot answer aggregation queries such as Q7.

  • Keyword-based systems match input keywords against inverted indexes of base and metadata.
  • These systems cannot answer aggregation queries such as “What was the best movie of each genre?”
  • Their main advantage is simplicity and adaptability.

2. Pattern-based systems

Pattern-based systems extend keyword approaches with NLP technologies and natural-language patterns. Patterns may be domain-independent or domain-dependent.

  • Pattern-based systems extend keyword-based systems with NLP technologies to handle more than keywords.
  • They use domain-independent patterns, such as “by” or “how many” for aggregations.
  • They can also use domain-dependent concepts such as “great movie.”

3. Parsing-based systems

Parsing-based systems analyze the grammatical structure of input questions through parse trees. Their semantic information can be mapped to production rules for query generation.

  • Parsing-based systems parse input questions to identify their grammatical structure.
  • Parse trees represent individual tokens and how they group into phrases.
  • Their semantic meaning can be mapped to production rules for query generation.

4. Grammar-based systems

Grammar-based systems define allowable questions with explicit rules, while the survey compares NLI categories and documents their different capabilities, mechanisms, and limitations. The reviewed systems illustrate trade-offs among natural-language coverage, ambiguity handling, adaptability, and expressiveness.

  • Grammar-based systems: Grammar-based systems use rules defining the questions users can ask, enabling natural-language suggestions during typing.
  • Comparative evaluation: Table 2 categorizes representative NLIs by methodology, query-language support, NLP technologies, and evaluation datasets.
  • Comparative evaluation: Direct comparison is difficult because systems use different datasets and evaluation practices, including studies without precision or recall.
  • Comparative evaluation: The survey’s sample-world evaluation uses increasingly complex queries to enable better comparison of approaches.
  • SODA: SODA’s metadata patterns and domain ontologies let users apply concepts such as “great movie” without specifying their underlying definition.
  • SODA: SODA translates keyword questions into SQL through five steps, beginning with lookup against database indexes and metadata-graph nodes.
  • SODA: SODA’s external sources, including DBpedia, support finding homonyms and synonyms.
  • SODA: SODA requires strict non-natural-language syntax for comparisons and aggregations and lacks lemmatization or stemming.

5.3 Parsing-based systems

Parsing-based NLIs analyze grammatical structure to capture dependencies that keyword and pattern methods miss. The surveyed systems include ontology-driven, interactive, and parse-tree-based approaches with distinct strengths and constraints.

  • Parsing-based approach: Parsing-based NLIs use dependency or constituency structure to identify grammatical relationships and long-range dependencies.This structural analysis helps separate parts of complex questions, including subqueries.
  • ATHENA: ATHENA maps full English sentences to ontology elements through metadata, translation-index, time, numeric, and dependency annotations before generating a query.Its ontology-driven pipeline uses an intermediate query language and ultimately produces SQL.
  • ATHENA: ATHENA supports natural-language explanations, semantic value variants, and single-level nesting, but cannot solve multiple subqueries and relies on trigger words for aggregations.These capabilities and constraints are reported as defining strengths and weaknesses of the system.
  • Other parsing-based systems: USI Answers queries multiple database resources and accepts natural-language, formal, keyword, or mixed input forms.Its supported resources include ontologies, relational databases, SPARQL endpoints, and external knowledge.
  • Other parsing-based systems: NaLIX can answer difficult subquery questions by adjusting parse trees, while NaLIR improves user interaction but remains highly dependent on parseability.Parse-tree reliance is presented as both an enabler of complex questions and a weakness.

5.4 Grammar-based systems

Grammar-based NLIs use explicit rules to constrain and guide question formulation, enabling translation into formal query languages. They can support complex queries, but their coverage depends on handcrafted or domain-specific grammar design.

  • Grammar-based approach: Grammar-based NLIs apply rules that define understandable questions and can provide suggestions during typing.This guidance helps users formulate questions that remain translatable into formal languages.
  • TR Discover: TR Discover translates English fragments into SQL or SPARQL through a First Order Logic intermediate representation and provides auto-suggestions.Its suggestions use dataset relationships and linguistic constraints encoded in a feature-based context-free grammar.
  • TR Discover: TR Discover parses input into FOL, converts the FOL representation into a parse tree, and then continues query translation.The example uses lexical entries and grammar rules to derive the intermediate representation and tree.
  • Other grammar-based systems: SQUALL supports aggregations, negations, and subqueries by separating natural-language translation from logical-language translation.Users must nevertheless know the RDF vocabulary to formulate questions in the required terms.
  • Other grammar-based systems: MEANS handles multiple answer types and foci, but handcrafted-rule restrictions remain; comparison and negation are not supported.Adding machine learning reduces the rule problem but requires a large training corpus.
  • Other grammar-based systems: AskNow permits unrestricted formulation and complex subqueries through normalized query structures, but depends on correct part-of-speech tags and restricts question types.SPARKLIS likewise guarantees syntactically correct queries through guidance while reducing user freedom and potentially leaving queries unreachable.

6 Evaluation

The evaluation compares 24 NLIs using ten increasingly difficult sample questions, then examines three commercial systems with the same questions. Results show a progression from simple keyword handling to structurally guided support for complex queries.

  • Evaluation design: The study evaluates 24 NLIs on ten sample questions of increasing complexity to make systems previously assessed on different datasets and measures more directly comparable.The evaluation labels answers as complete, strict-syntax or partial, unable, or undocumented.
  • General findings: Keyword-based NLIs handle simple string filters, while pattern-based systems extend this to features such as aggregations but not subqueries.Trigger words do not reliably identify which portions of a question belong to separate subqueries.
  • General findings: Parsing-based NLIs identify and group subquery components using dependency or constituency trees, although trigger-word handling remains a limitation for some systems.The survey specifically notes this issue for aggregation handling.
  • System comparison: SQUALL, SPARKLIS, NaLIR, and ATHENA perform best overall, but each trades off vocabulary requirements, interaction constraints, unsupported concepts, trigger-word dependence, or subquery coverage.SQUALL requires RDF vocabulary, SPARKLIS restricts free writing, NaLIR lacks concepts, and ATHENA cannot solve multiple subqueries.
  • Commercial systems: Google performs best among the three commercial systems, while Siri handles simple select and filter questions and IMDb provides keyword and form-based search without precise answers to some select questions.Google’s evaluation treats featured snippets or knowledge-panel answers as correct.

7 Machine Learning Approaches for NLIs

Machine-learning approaches broaden linguistic flexibility and increasingly use neural translation, but the surveyed evidence remains limited by training-data demands and restricted evaluation settings. The paper therefore treats their practical use in realistic databases as unresolved.

  • Approaches: Machine-learning NLI research includes neural translation, reinforcement learning, template learning, multilingual meaning prediction, and neural semantic parsing.These approaches target translation from natural language to SQL or SPARQL and richer question representations.
  • Advantages: ML-based approaches support greater linguistic variability, allowing users to formulate queries with more flexibility than traditional systems.This is identified as a principal advantage of machine-learning methods.
  • Limitations: Supervised ML approaches require large training datasets to achieve good translation accuracy.The paper identifies training-data requirements as a major challenge.
  • Evidence scope: Seq2SQL was demonstrated on simple single-table queries without joins, while related systems focus on issues such as WHERE-clause generation and synthetic training data.The examples illustrate that current ML systems often rely on constrained query settings or substantial generated data.
  • Evidence scope: Recent ML approaches have either been demonstrated only on single-table datasets or require large amounts of training data, leaving realistic database usage to be shown.The paper also excludes a detailed discussion of conversational systems from its scope.

8 Conclusions

The evaluation of 24 NLIs against ten sample questions yields lessons about matching mechanisms to question complexity, resolving ambiguity, guiding users, and combining traditional methods with neural translation.

  • The survey evaluates 24 natural language interfaces using ten sample questions to expose their strengths and weaknesses.
  • Simple and complex questions require distinct handling: keyword-like inputs suit simple queries, whereas complex questions are often phrased as grammatical sentences.Pattern-based systems partly address this distinction, but often remain limited on subqueries.
  • Subquery identification remains a significant NLI challenge, and parse trees are the most common technology used to address it.
  • Ambiguity resolution should minimize user interactions by clarifying first the ambiguity that most affects the others.
  • Grammar-based systems can guide users while they type, helping systems understand questions and users learn which formulations produce useful results.Question histories and templates further show users what the system can understand and how questions should be asked.
  • A hybrid of traditional NLI methods and neural machine translation is proposed as a future direction combining accuracy with robustness to language variability.The practical use of neural machine translation in realistic database settings still needs to be demonstrated.
  • The evaluation shows significant progress over the last decade, while further research is needed to increase NLI expressive power.The paper presents its lessons as guidance for researchers and practitioners designing systems with industrial impact.

A SQL Representation

The SQL representations translate sample questions into relational queries using joins, filters, aggregations, grouping, and nested subqueries. The examples include both straightforward retrieval conditions and more complex comparative queries.

  • The SQL examples use SELECT DISTINCT queries to retrieve movies or people across related tables connected by join conditions.Examples connect movie, person, directing, starring, genre, and gross relations.
  • Another nested query compares movie gross with the maximum gross returned by an aggregate subquery.
  • The best-movie-per-genre query computes maximum ratings grouped by genre, then joins those results back to movies and genres.The outer query matches each movie's rating and genre to the grouped maximum-rating results.
  • A scalar subquery compares each movie's rating with the rating of a specified movie.
Loading 1906.08990v1…