PaperlayerView exact revision on arXiv

Source-linked AI summary

FAQ Retrieval using Query-Question Similarity and BERT-Based Query-Answer Relevance

Wataru Sakata, Tomohide Shibata, Ribeka Tanaka, Sadao Kurohashi

arXiv:1905.02851v2cs.IRcs.CL

TL;DR

FAQ retrieval must identify a relevant QA pair without costly labeled relevance data, while FAQ sets may be too small to train query-answer models. This paper combines unsupervised query-question similarity with BERT-based query-answer relevance, and reports the combined approach effective on localgovFAQ and StackExchange. Its examples also indicate that deeper understanding of QA texts may be needed for some queries.

  • Problem

    Labeled query-to-QA relevance data is costly to construct, and FAQ pages may have too few QA pairs to train a relevance model.

    Source: §1 · p. 1

  • Method

    The method combines unsupervised query-question similarity with BERT-based query-answer relevance, using similar FAQ sets to increase available QA training data.

    Source: §1 · p. 1, §2.2 · p. 2

  • Results

    The combined use of query-question similarity and query-answer relevance was effective in experiments on localgovFAQ and StackExchange.

    Source: §1 · p. 1, §4 · p. 4

  • Takeaways & Limitations

    The approach uses the two relevance signals to support robust FAQ retrieval, including cases with lexical gaps between query and question wording.

    Source: §1 · p. 1

    In one example, BERT matched query and answer terms but retrieved the wrong QA pair; retrieving the correct pair required deeper understanding of QA texts.

    Source: §3.3 · pp. 3–4

Abstract

from arXiv · show

Frequently Asked Question (FAQ) retrieval is an important task where the objective is to retrieve an appropriate Question-Answer (QA) pair from a database based on a user's query. We propose a FAQ retrieval system that considers the similarity between a user's query and a question as well as the relevance between the query and an answer. Although a common approach to FAQ retrieval is to construct labeled data for training, it takes annotation costs. Therefore, we use a traditional unsupervised information retrieval system to calculate the similarity between the query and question. On the other hand, the relevance between the query and answer can be learned by using QA pairs in a FAQ database. The recently-proposed BERT model is used for the relevance calculation. Since the number of QA pairs in FAQ page is not enough to train a model, we cope with this issue by leveraging FAQ sets that are similar to the one in question. We evaluate our approach on two datasets. The first one is localgovFAQ, a dataset we construct in a Japanese administrative municipality domain. The second is StackExchange dataset, which is the public dataset in English. We demonstrate that our proposed method outperforms baseline methods on these datasets.

1 INTRODUCTION

FAQ retrieval selects a relevant question-answer pair for a user's query, but labeled training data is costly and individual FAQ sets may be too small to train query-answer relevance models. The paper combines unsupervised query-question similarity with BERT-based query-answer relevance and evaluates the approach on localgovFAQ and StackExchange.

1 INTRODUCTION

  • Labeled query-to-QA relevance data is costly to construct, motivating unsupervised query-question similarity.The paper also notes that FAQ pages may not contain enough QA entries to train relevance models, and addresses this by collecting similar FAQ sets.Source: §1 · p. 1
  • The method combines unsupervised query-question similarity with BERT-learned query-answer relevance to retrieve FAQ pairs despite lexical gaps between queries and questions.The approach is evaluated on localgovFAQ and StackExchange.Source: §1 · p. 1
  • The evaluation uses localgovFAQ, constructed from Japanese local-government FAQs, and the public StackExchange dataset.The localgovFAQ evaluation includes queries collected via crowdsourcing.Source: §1 · pp. 1–2

2 PROPOSED METHOD

The method retrieves a QA pair from a target FAQ set for a user query. It combines TSUBAKI-based query-question similarity with BERT-based query-answer relevance, using different ranking signals to accommodate lexical gaps.

2.1 Task Description

  • The task is to select an appropriate QA pair from a target local-government FAQ set for a user query.The target FAQ set is one of multiple FAQ sets used in training data.Source: §2.1 · p. 2

2.2 q-Q similarity by TSUBAKI

  • TSUBAKI computes query-question similarity using an unsupervised retrieval engine based on OKAPI BM25.It uses sentence dependency structure and automatically extracted synonyms for flexible matching.Source: §2.2 · p. 2

2.3 q-A relevance by BERT

  • BERT classifies question-answer pairs to estimate their relevance, training on FAQ QA pairs with randomly selected answers as negatives.At search time, the model scores the user's query against each answer in the target FAQ set.Source: §2.3 · p. 2

2.4 Combining TSUBAKI and BERT

  • The combined ranker first takes BERT's ten highest-ranked results, then uses TSUBAKI alone above threshold α and a weighted sum of query-question similarity and query-answer relevance otherwise.The design uses TSUBAKI for strong lexical overlap and BERT to handle gaps between query and question wording; TSUBAKI scores are normalized before summing.Source: §2.4 · p. 2

3 EXPERIMENTS AND EVALUATION

The evaluation covers localgovFAQ, constructed from Japanese local-government FAQs, and the public StackExchange dataset. The proposed method performed best on both, while examples and score comparisons illustrate complementary system behavior and a remaining need for deeper QA-text understanding.

3 EXPERIMENTS AND EVALUATION

  • StackExchange contains 719 QA pairs and 1,250 paraphrase queries, evaluated with five-fold cross-validation and 60/20/20 training, development, and test splits.The task is to estimate an appropriate QA pair among the 719 entries for each query.Source: §3 · p. 2

3.1 LocalgovFAQ Evaluation Set Construction

  • localgovFAQ uses 1,786 Amagasaki-city QA pairs and 990 crowdsourced queries; after searching for correct pairs and excluding queries without one, 784 remained, with 627 used for evaluation.The collection process addressed pooling bias by searching for correct QA pairs when neither TSUBAKI nor BERT outputs included a relevant pair.Source: §3.1 · pp. 2–3

3.2 Experimental Settings

  • The experiments compare Bi-LSTM and target-only BERT on localgovFAQ, and CNN-rank and BERT without query paraphrases on StackExchange; TSUBAKI is evaluated only in q-Q.BERT was fine-tuned on Japanese FAQs from 21 local governments; the StackExchange training data were divided into (q,A) and (Q,A) pairs.Source: §3.2 · p. 3

3.3 Evaluation Results and Discussions

  • The proposed method performed best on localgovFAQ and on all measures for StackExchange.The localgovFAQ comparison reports it performed best; the StackExchange comparison reports it performed best across all measures.Source: Table 1 · p. 3, Table 2 · p. 3, §3.3 · p. 3
  • TSUBAKI’s high scores aligned with high accuracy, while BERT scores had a looser accuracy correlation, supporting their combination.BERT outperformed TSUBAKI on localgovFAQ SR@5, while their SR@1 performances were almost the same.Source: §3.3 · p. 3
  • In the examples, the proposed method recovered a correct first-ranked QA pair when TSUBAKI and BERT each ranked an incorrect pair first.The first example also shows BERT and the proposed method retrieving the correct pair where TSUBAKI did not.Source: §3.3 · p. 3, Table 3 · p. 4
  • A third example defeated every method; the correct pair required deeper understanding linking a youth-center facilities question to an answer mentioning a music room.BERT captured the relation between “instruments” in the query and “music” in the answer but still retrieved the wrong QA pair.Source: §3.3 · pp. 3–4
  • Retrieval takes about 2 seconds per localgovFAQ query using seven TITAN X Pascal GPUs; the authors suggest telescoping for larger FAQ sets [3].The paper describes the model as practical enough in this reported setting.Source: §3.3 · p. 4

4 CONCLUSION

The paper combines query-question similarity with BERT-based query-answer relevance, increasing available QA data by collecting similar FAQ sets. The authors report that the combined approach was effective in experiments.

4 CONCLUSION

  • The proposed method combines query-question similarity with BERT-based query-answer relevance, using similar FAQ sets to increase available QA data.The authors describe it as a robust, high-performance FAQ retrieval method.Source: §4 · p. 4
  • Experimental results demonstrated that combining query-question similarity and query-answer relevance was effective.Source: §4 · p. 4
  • The authors planned to make the code and constructed localgovFAQ dataset publicly available.Source: §4 · p. 4

Related papers

Loading 1905.02851v2…