PaperlayerView exact revision on arXiv

Source-linked AI summary

Putting Question-Answering Systems into Practice: Transfer Learning for Efficient Domain Customization

Bernhard Kratzwald, Stefan Feuerriegel

arXiv:1804.07097v2cs.CL

TL;DR

Domain-specific Q&A performance is often limited, while costly customization hinders practical use. The paper combines metadata filtering with fuse-and-oversample transfer learning for customization, and reports higher answer correctness than conventional systems. Its finance and film-industry cases use only a few hundred labeled question-answer pairs.

  • Problem

    Real-world Q&A performance is often limited because each domain requires costly customization.

    Source: §1 · p. 2, §6.4 · p. 14

  • Method

    The paper uses metadata filtering for document retrieval and fuse-and-oversample transfer learning to customize answer extraction with a small labeled domain dataset.

    Source: §1 · p. 2, §6.4 · p. 14

  • Results

    1 in 3.4 questions was answered correctly by conventional systems, compared with 1 in 2.0 for the proposed system.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The finance and film-industry cases show significant performance gains with a few hundred annotated question-answer pairs.

    Source: §1 · p. 2

    Fine-tuning network parameters on a domain-specific dataset of very small size is challenging.

    Source: §5.2 · p. 10

Abstract

from arXiv · show

Traditional information retrieval (such as that offered by web search engines) impedes users with information overload from extensive result pages and the need to manually locate the desired information therein. Conversely, question-answering systems change how humans interact with information systems: users can now ask specific questions and obtain a tailored answer - both conveniently in natural language. Despite obvious benefits, their use is often limited to an academic context, largely because of expensive domain customizations, which means that the performance in domain-specific applications often fails to meet expectations. This paper proposes cost-efficient remedies: (i) we leverage metadata through a filtering mechanism, which increases the precision of document retrieval, and (ii) we develop a novel fuse-and-oversample approach for transfer learning in order to improve the performance of answer extraction. Here knowledge is inductively transferred from a related, yet different, tasks to the domain-specific application, while accounting for potential differences in the sample sizes across both tasks. The resulting performance is demonstrated with actual use cases from a finance company and the film industry, where fewer than 400 question-answer pairs had to be annotated in order to yield significant performance gains. As a direct implication to management, this presents a promising path to better leveraging of knowledge stored in information systems.

1 INTRODUCTION

Question-answering systems offer tailored natural-language answers, but domain-specific performance is often limited because customization is costly. The paper proposes metadata filtering and transfer learning, reporting improved answer correctness in finance and film-industry use cases.

1 INTRODUCTION

  • Domain-specific customization is costly and often leaves real-world Q&A performance limited, despite highly specialized systems for individual use cases.Customization may require domain-specific knowledge, terminology, and slang, with prior approaches using manual linguistic rules or extensive hand-labeled datasets.Source: §1 · p. 2
  • The paper proposes inductively transferring knowledge from a general, open-domain application to a domain-specific use case, using a few hundred labeled question-answer pairs for fine-tuning.The approach is evaluated in finance and the film industry.Source: §1 · p. 2
  • 1 in 3.4 questions was answered correctly by conventional systems, compared with 1 in 2.0 for the proposed system.Correctness required an exact match to the desired word sequence.Source: §1 · p. 2
  • Metadata filtering incorporates document metadata into information retrieval, while fuse-and-oversample targets answer extraction as a transfer-learning variant.The paper presents metadata filtering as a retrieval mechanism and the transfer-learning variant as a way to improve answer extraction.Source: §1 · p. 2
  • The fuse-and-oversample transfer-learning variant increased accuracy by 8.1% to 17.0%.The paper reports that metadata filtering and this approach also benefit from being implemented together.Source: §1 · p. 2

2 BACKGROUND

The background distinguishes ontology-based Q&A from content-based systems that retrieve documents and extract answers from raw text, motivating content-based Q&A and transfer learning for practical domain customization. It frames transfer learning as using a large source dataset to support a target dataset limited to a few hundred observations.

2.1 Ontology-Based Q&A Systems

  • Ontology-based Q&A transforms questions and documents into semantic representations for reasoning, but information loss and limited domain coverage can make unseen-entity questions costly to support.The conventional remedy is to encode extensive domain knowledge manually, imposing high upfront costs.Source: §2.1 · p. 3

2.2 Content-Based Q&A Systems

  • Content-based Q&A operates on raw text through retrieval followed by answer extraction, retaining document content and avoiding manually engineered rules.Its similarity matching can handle unseen entities or relations, making it a practical choice that the paper later studies for domain customization.Source: §2.2 · p. 3

2.2.1 Information Retrieval Module.

  • Information retrieval commonly represents documents and queries as vectors and uses tf-idf weighting to emphasize terms that occur in fewer documents.State-of-the-art systems also use n-gram tf-idf to capture local word order, with feature hashing trading off performance and memory use.Source: §2.2.1 · p. 4

2.2.2 Answer Extraction Module.

  • Answer extraction selects and ranks candidate answers from retrieved documents; deep-learning models can predict answer start and end positions end-to-end.The paper uses DrQA and also plans a broader architecture comparison; the classifiers used in extraction can require extensive datasets, motivating transfer learning.Source: §2.2.2 · pp. 4–5

2.3 Transfer Learning

  • Deep neural networks require thousands of training samples to avoid overfitting and achieve satisfactory performance, but expert-labeled datasets can be costly.Fine-tuning a source-task model on a small target dataset requires extensive tuning to avoid over- or underfitting; multi-task learning is unsuitable for very small datasets.Source: §2.3 · p. 5
  • The paper proposes fuse-and-oversample transfer learning as a robust approach for inductive transfer to small datasets.Implicit oversampling is a prerequisite for the approach.Source: §2.3 · p. 5
  • The source dataset DS may contain thousands or millions of observations, while the target dataset DT is limited to a few hundred.Source: §2.3 · p. 5

3 METHODS AND MATERIALS

The paper’s generic Q&A pipeline combines document retrieval with answer extraction, customizing retrieval through metadata filtering and answer extraction through transfer learning. Its fuse-and-oversample approach targets small domain datasets while avoiding the need to change original hyperparameters.

3 METHODS AND MATERIALS

  • The generic Q&A architecture separates retrieval from answer extraction and allows different modules for each stage to test the robustness of results.The metadata filter operates upstream of information retrieval.Source: §3 · p. 5, Figure 1 · p. 6

3.1 Metadata Filter: Domain Customization for Information Retrieval

  • Metadata filtering lets users restrict documents before question-document scoring, offering cost-efficient domain customization without further development costs.The system displays input fields for corpus metadata, with selection controls suited to categorical and numeric fields.Source: §3.1 · p. 6

3.2 Domain Customization through Transfer Learning

  • The alternative initialization-and-fine-tuning approach depends strongly on hyperparameter choices and introduces more training instability.The paper notes that small datasets also limit extensive hyperparameter tuning.Source: §3.2 · p. 7
  • Fuse-and-oversample merges domain-specific and large-scale datasets, then oversamples domain-specific pairs at a 1:3 ratio to address their small share.The approach retrains the model from scratch and avoids a separate fine-tuning phase or changes to the original hyperparameters.Source: §3.2 · p. 7
  • The authors report that combining dataset fusion and oversampling is key to successful knowledge transfer across datasets.The approach is intended to support inductive transfer when the domain-specific dataset contains only a few hundred documents.Source: §3.2 · p. 7

4 DATASET

The dataset section describes a financial-news Q&A application using firm announcements and firm-name filtering. Its split includes 393 training question-answer pairs, 257 test pairs, and 13,272 unannotated news items.

4 DATASET

  • The application uses English-language financial news announcements published by firms under regulatory reporting rules and disseminated through standardized channels.The authors describe the setting as relevant to practitioners including media and investors, and challenging because of complex language and domain-specific terminology.Source: §4 · pp. 7–8
  • 393 question-answer pairs from 63 documents form the training set, while 257 pairs from another 63 documents form the test set.The test set also includes 13,272 financial news items without annotations for evaluating retrieval across a larger corpus.Source: §4 · p. 8
  • Filtering by firm name addresses questions that cannot be answered uniquely without specifying a company and can save users from typing identifiers.The paper gives adjusted net sales growth as an example of a question needing a company context.Source: §4 · p. 8
  • The transfer-learning source dataset, SQuAD, contains 107,785 question-answer pairs, leaving the financial dataset at 0.6% of the merged samples.The authors cite this imbalance as the reason for oversampling domain-specific pairs.Source: §4 · p. 8

5 RESULTS

The results examine how metadata filtering and fuse-and-oversample transfer learning affect document retrieval and answer extraction, including end-to-end question answering. They also assess the approach’s robustness across movie-industry data, architectures, and datasets.

5.1 Metadata Filtering

  • Metadata filtering improved recall@1 from 0.48 to 0.88, an absolute increase of 0.4; relative improvement was 83.3% for recall@1 and 45.5% for recall@5.Source: Table 2 · p. 9, §5.1 · p. 9
  • Hashed-bigram retrieval generally outperformed the vector-space and Okapi BM25 models, while changing the retrieval model alone produced only marginal performance changes.The paper attributes hashed-bigram performance to question-like inputs containing more information than keyword-based queries.Source: §5.1 · p. 9

5.2 Answer Extraction Module

  • Fuse-and-oversample transfer learning improved performance across all neural-network architectures, with relative gains up to 17.0% and 3.2–13.0 percentage points higher than strategy (ii).The approach trains simultaneously on open-domain and domain-specific question-answer pairs, oversampling the latter to address imbalance.Source: §5.2 · p. 10

5.3 Domain Customization

  • With metadata filtering, returning multiple documents no longer helped: extra documents could add noise, making answer extraction especially critical when retrieval was precise.DrQA using the top five documents performed best in the plain and transfer-learning experiments, but the additional selection mechanism changed that pattern [Kratzwald & Feuerriegel 2018].Source: §5.3 · p. 12

5.4 Robustness Check: Transfer Learning with Additional Domain Customization

  • The robustness check sampled 393 training and 257 test WikiMovies question-answer pairs, annotated them with supporting Wikipedia passages, and evaluated transfer learning only.Metadata filtering was excluded because the authors considered it straightforward to transfer.Source: §5.4 · p. 12
  • Answer-extraction domain customization is described as the more challenging and performance-fragile element.Source: §5.4 · p. 12
  • On a second dataset of movie questions, fuse-and-oversample consistently outperforms the literature’s naïve method; evaluation across architectures and datasets supports generalizability.Source: §5.4 · p. 12

6 DISCUSSION

The discussion presents metadata filtering and transfer learning as practical levers for domain customization, reporting gains while emphasizing design challenges and scope boundaries. It also identifies research directions and argues that transfer learning can reduce annotation demands.

6.1 Domain Customization

  • Metadata filtering raises exact matches with the shortest correct answer by up to 81.3%, while transfer learning yields gains of up to 18.7%.The authors present both as domain-customization strategies.Source: §6.1 · p. 13

6.2 Design Challenges

  • As few as 400 annotated question-answer pairs can support significant performance improvements, addressing the burden of labeling thousands for each domain-specific application.The authors note this may reduce the domain-expert input needed in settings such as medicine or law.Source: §6.2 · p. 13
  • Document retrieval upper-bounds end-to-end performance; retrieval-model choice matters less in these experiments, while metadata incorporation yields strong improvement.Practitioners are advised to consider this trade-off when choosing an implementation.Source: §6.2 · p. 13
  • Fuse-and-oversample improves performance while making hyperparameter tuning unnecessary, addressing the volatility of tuning transfer learning with limited data.The discussion contrasts it with traditional initialize-and-fine-tune transfer learning.Source: §6.2 · p. 13
  • A metadata filter can improve performance, but practitioners must adapt its design to the domain-specific use case.The authors identify advanced filtering for open-domain settings and additional semantic information as future research directions.Source: §6.2 · pp. 13–14
  • The approach is scoped to English, answers that are sub-spans of one document, questions answerable from that document, and meaningful metadata filters whose benefit can vary.The authors describe these as assumptions or boundaries of the artifact.Source: §6.2 · p. 14

6.3 Implications for Research

  • The discussion identifies advanced neural architectures as another source of performance improvement and points to extending metadata filtering to open-domain settings.It also calls for work on answers spanning multiple documents and handling semi-structured information such as tables or linked data.Source: §6.3 · p. 14

6.4 Conclusion

  • Transfer learning shifts most learning to inductive transfer, so comparatively small labeled sets can reduce the need to hand-label thousands of pairs and let small businesses use deep learning.The conclusion describes this as requiring comparatively little effort.Source: §6.4 · p. 15

A.1 Information Retrieval Module

The information retrieval module locates documents relevant to a question using three approaches: cosine-similarity vector search, Okapi BM25, and tf-idf over hashed bigrams. Each approach scores documents for retrieval, with the bigram model incorporating local word order.

A.1 Information Retrieval Module

  • The module implements cosine-similarity vector search, probabilistic Okapi BM25, and tf-idf over hashed bigram counts.The tf-idf model scores documents by the dot product of query and document vectors.Source: §A.1 · p. 15
  • Cosine similarity measures the relevance of each document vector to a query vector, and retrieval selects the document with greatest similarity.Document and query representations are tf-idf weighted vectors.Source: §A.1 · p. 15
  • Okapi BM25 uses a probabilistic approach to estimate the probability of a document given a query.The paper uses the default implementation described in.Source: §A.1 · p. 15
  • The bigram model incorporates local word ordering that unigram tf-idf ignores and uses feature hashing to manage the resulting dimensionality.Its document score is the dot product between query and document vectors.Source: §A.1 · p. 15

A.2 Answer Extraction Module

The answer extraction module uses a selected document to infer an answer within it. The paper evaluates traditional and deep-learning baselines and proposes BiDAF and R-Net as additional architectures beyond DrQA.

A.2 Answer Extraction Module

  • Answer extraction infers the response from the previously selected document, with the answer constrained to that document.The answer is represented as a span a ∈ dq.Source: §A.2 · p. 15
  • The evaluation includes two traditional machine-learning benchmarks and the DrQA network, alongside the additional BiDAF and R-Net architectures.BiDAF and R-Net extend the evaluated deep-learning architectures beyond DrQA.Source: §A.2 · p. 15
  • The proposed networks combine character-level embeddings and multiple attention mechanisms to handle unseen words and use question context.The paper links character embeddings to adapting to unseen words and attention to incorporating both question and answer context.Source: §A.2 · p. 16

A.2.1 Baseline Methods.

The section describes two answer-extraction baselines that narrow candidate answers to linguistic constituents, then rank or select those candidates using different procedures.

A.2.1 Baseline Methods.

  • Both baselines extract linguistic constituents from the source document to reduce the number of candidate answers.A constituent is one or more words that can stand on their own.Source: §A.2.1 · p. 16
  • The sliding-window approach processes the text passage and chooses a sub-span of words as an answer; the cited passage does not establish which sub-span it selects.Source: §A.2.1 · p. 16
  • The second baseline uses logistic regression to rank candidates from hand-crafted lexical features, including tf-idf weights extended with lexical information.The classifier is calibrated on documents and their correct responses.Source: §A.2.1 · p. 16

A.2.2 Deep Learning Methods.

The paper adapts machine-comprehension networks for answer extraction, using embeddings, recurrent encodings, attention, and answer-span prediction. The approaches differ in their embedding and attention designs, recurrent units, and how they predict span boundaries.

A.2.2 Deep Learning Methods.

  • DrQA, BiDAF, and R-Net are adapted from machine comprehension for answer extraction, but machine-comprehension models require the answer-containing passage to be known up front.Unlike machine comprehension, question answering also includes information retrieval.Source: §A.2.2 · p. 16
  • R-Net and BiDAF supplement word embeddings with character-level embeddings, which support representations of unseen and infrequent words.All networks use GloVe word embeddings; R-Net concatenates the word- and character-level vectors, while BiDAF combines them with a two-layer highway network.Source: §A.2.2 · p. 16
  • All three architectures use bidirectional recurrent layers; DrQA and BiDAF use LSTMs, while R-Net uses computationally cheaper but less flexible GRUs.The recurrent layers encode sequential structure in question and context sequences.Source: §A.2.2 · pp. 16–17
  • The networks fuse question and context through attention, with DrQA using a simpler mechanism and BiDAF and R-Net combining multiple mechanisms.Attention weights text segments according to their relevance to the question and context.Source: §A.2.2 · p. 17
  • DrQA predicts answer boundaries independently, whereas BiDAF and R-Net condition the end-position prediction on the beginning position.R-Net uses an additional pointer network for prediction.Source: §A.2.2 · pp. 17–18
Loading 1804.07097v2…