PaperlayerView exact revision on arXiv

Source-linked AI summary

Semi-supervised Question Retrieval with Gated Convolutions

Tao Lei, Hrishikesh Joshi, Regina Barzilay, Tommi Jaakkola, Katerina Tymoshenko, Alessandro Moschitti, Lluis Marquez

arXiv:1512.05726v2cs.CLcs.NE

TL;DR

Question retrieval could help forums reuse existing answers, but long, detail-heavy questions and sparse annotations make matching difficult. The paper uses a gated recurrent-convolutional model, pre-trained from question bodies to titles on the raw corpus and fine-tuned with annotations; it reports gains over standard IR and neural baselines.

  • Problem

    Finding semantically similar questions is difficult because key information can be buried in extraneous body details and annotations of similar pairs are scarce and fragmented.

    Source: §1 · p. 1

  • Method

    A gated recurrent-convolutional encoder maps questions to representations, pre-trained to reproduce titles from bodies on the raw corpus and then discriminatively fine-tuned on limited annotations.

    Source: §1 · p. 2, §8 · p. 9

  • Results

    75.6% MRR and 62.0% P@1, with an 8% absolute improvement over a standard IR baseline and 4% over standard neural network architectures.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The architecture can glean key information from lengthy, detail-riddled questions, and body-to-title pre-training on the raw corpus is integral to its success.

    Source: §8 · p. 9

    Early experiments did not remove user-annotated “possible duplicate” sections embedded in the AskUbuntu XML, which should be removed for a more realistic evaluation.

    Source: Note on Data Pre-processing Issue · p. 12

Abstract

from arXiv · show

Question answering forums are rapidly growing in size with no effective automated ability to refer to and reuse answers already available for previous posted questions. In this paper, we develop a methodology for finding semantically related questions. The task is difficult since 1) key pieces of information are often buried in extraneous details in the question body and 2) available annotations on similar questions are scarce and fragmented. We design a recurrent and convolutional model (gated convolution) to effectively map questions to their semantic representations. The models are pre-trained within an encoder-decoder framework (from body to title) on the basis of the entire raw corpus, and fine-tuned discriminatively from limited annotations. Our evaluation demonstrates that our model yields substantial gains over a standard IR baseline and various neural network architectures (including CNNs, LSTMs and GRUs).

1 Introduction

Question retrieval could help reuse existing answers, but long, detail-heavy bodies and sparse, incomplete annotations make finding similar questions difficult. The paper combines a gated recurrent-convolutional model with raw-corpus pre-training and discriminative fine-tuning, reporting gains over IR and neural baselines.

1 Introduction

  • Identifying similar questions could let forums reuse existing answers, reducing response times and unnecessary repeated work.Forum participants otherwise identify and refer to similar questions manually, with limited, scattered success.Source: §1 · p. 1
  • Retrieval is difficult because question bodies contain extraneous user-specific details, while titles can omit crucial information and similarity annotations are sparse and incomplete.A manual AskUbuntu sample found annotations for only 5% of similar pairs, with around 79% precision.Source: §1 · pp. 1–2
  • The model encodes questions as vectors compared by cosine similarity, using adaptive gating and body-to-title encoder-decoder pre-training on the raw corpus before fine-tuning on limited annotations.The discriminative fine-tuning yields an additional performance boost.Source: §1 · p. 2
  • The full model achieves 75.6% MRR and 62.0% P@1, improving 8% absolutely over a standard IR baseline and 4% over standard neural architectures.The neural architectures include CNNs, LSTMs, and GRUs.Source: §1 · p. 2

2 Related Work

Question retrieval research spans traditional retrieval methods and newer representation-learning approaches. This paper distinguishes its approach through sequence-based recurrent convolution and semi-supervised use of unannotated questions.

2 Related Work

  • Earlier question-retrieval approaches used machine translation, topic modeling, and knowledge graphs.Source: §2 · p. 2
  • Representation-learning work includes category-informed embeddings with a Fisher kernel and a CNN combined with bag-of-words features (dos Santos et al., 2015).Source: §2 · p. 2
  • Unlike those approaches, this paper represents questions as word sequences with recurrent convolution and uses the unannotated question corpus for semi-supervised training.Related answer-selection work also explores similarity measures beyond cosine and neural attention over RNNs (Feng et al., 2015; Tan et al., 2015).Source: §2 · pp. 2–3

3 Question Retrieval Setup

The retrieval setup first narrows the search to an IR-generated candidate set, then ranks candidates by similarity to the query. The model learns from user-marked pairs and corpus-sampled negatives, with encoder pre-training used to address limited annotation coverage.

3 Question Retrieval Setup

  • At test time, an IR engine retrieves a smaller candidate set, and the model ranks its questions so similar candidates appear above dissimilar ones.Evaluation uses explicit manual annotations.Source: §3 · p. 3
  • Training positives come from user-marked similar pairs, while negative questions are sampled randomly from the entire corpus.Source: §3 · p. 3
  • The discriminative setting learns the scoring parameters with a max-margin framework.Source: §3 · p. 3
  • The scoring function is cosine similarity between question representations, and the encoder is pre-trained on the larger unannotated corpus before discriminative fine-tuning.Source: §3 · p. 3

4 Model

The model uses context-dependent gates to selectively aggregate non-consecutive patterns from question text. It is pre-trained on the raw corpus to predict titles from context, then fine-tuned with limited annotations.

4.1 Non-consecutive Convolution

  • The encoder reads title or body tokens into a state sequence, then pools the states or takes the last state; title and body vectors are averaged.The state sequence is condensed to a single vector for question retrieval.Source: §4.1 · p. 4
  • Context-dependent neural gates control how signals are averaged, integrating recurrent updates with non-consecutive convolution to emphasize informative tokens.The gates can down-weight uninformative tokens and give larger weights to semantic content words.Source: §4.1 · p. 4
  • The gate makes accumulated states represent weighted combinations of n-gram patterns, with a zero gate recovering a traditional CNN with filter width n.The learned gate responds to the previous state and current token.Source: §4.1 · p. 4

4.2 Pre-training Using the Entire Corpus

  • An encoder-decoder is pre-trained on raw-corpus title-context pairs to predict titles from titles, bodies, or similar questions, then fine-tuned discriminatively.Titles serve as distilled summaries of noisy bodies, and their short length makes title decoding faster than body decoding.Source: §4.2 · p. 5

5 Alternative models

The alternative encoders are LSTMs, GRUs, and CNNs. LSTMs and GRUs can be pre-trained and fine-tuned, whereas CNNs are trained only discriminatively and use pooled feature maps.

Alternative encoder training

  • LSTM and GRU encoders can be pre-trained analogously to RCNNs and fine-tuned discriminatively, while CNN encoders are trained only discriminatively.The paper reports that none of these alternatives reaches quite the same performance level as its model.Source: §5 · p. 5

Recurrent alternatives

  • LSTM and GRU units use neural gates to control information flow; LSTMs read the current token and prior internal and visible states to generate new states.The LSTM’s input, forget, and output gates are named in the supplied passage.Source: §5 · p. 5

CNN alternatives

  • CNN filters map local windows of consecutive words to feature representations, which are aggregated with max-pooling or average-pooling.The filter width n determines the size of each word window.Source: §5 · p. 6

6 Experimental Setup

The experiments use AskUbuntu questions, manually annotated evaluation pairs, and noisy user-marked training pairs. Neural encoders are compared with retrieval baselines using standard IR metrics and tuned across model and training configurations.

Dataset

  • 167,765 unique AskUbuntu questions form the dataset, with question bodies truncated at 100 words.Each question has a title and body, and the corpus includes user-marked similar-question pairs.Source: §6 · p. 6

Gold Standard for Evaluation

  • Only 5% of similar questions in the sampled candidates were user-marked, with 79% precision, motivating expert-annotated evaluation.The evaluation sample paired questions with 20 BM25-retrieved candidates.Source: §6 · p. 6

Training Set

  • Training uses user-marked pairs as positives and 20 randomly sampled corpus questions per query as negatives each epoch.The positive annotations are used for their high precision and to avoid additional manual training annotation.Source: §6 · p. 6

Development and Test Sets

  • The dev and test sets use the first 200 questions from prior splits, with the top 20 BM25 candidates manually labeled across 8K pairs.Source: §6 · p. 6

Baselines and Evaluation Metrics

  • Neural encoders are compared with BM25, TF-IDF, and an SVM reranker using MAP, MRR, P@1, and P@5.The SVM uses a linear kernel and similarity measures from DKPro; TF-IDF ranks by cosine similarity.Source: §6 · p. 6

Hyper-parameters

  • Hyperparameter search varies n-gram settings for TF-IDF and learning rate, dropout, CNN width, and pooling for neural models.Neural models use Adam; word2vec embeddings are 200-dimensional and fixed, and model configurations report hidden dimension, parameter count, and filter width.Source: §6 · p. 7, Table 3 · p. 7

7 Results

The pre-trained RCNN performs best across the reported retrieval metrics, and adding question bodies improves its performance. Results also show consistent gains from pre-training and explore pooling choices and the learned gate’s attention to question details.

Overall Performance

  • 62.0% P@1 on the test set puts the full pre-trained RCNN over 8 percentage points ahead of BM25 and ahead of other neural models across metrics.The authors report the best performance across all metrics on both dev and test sets.Source: §7 · p. 7
  • Pre-training the RCNN on the entire corpus consistently improves results across all reported metrics.Source: §7 · p. 7

Pooling Strategy

  • Using the last hidden state as the RCNN representation performs better than averaging its states, while RCNN outperforms other neural models under both pooling strategies.Source: Table 4 · p. 8, §7 · p. 8

Using Question Body

  • Adding question bodies improves RCNN performance by 1% to 3%, while TF-IDF changes little and shows mixed metric movement.The paper links the RCNN gain to selecting components most relevant to the question from long, descriptive bodies.Source: Table 5 · p. 8, §7 · p. 8

Pre-training

  • 25 perplexity and over 68% MRR are achieved by RCNN without fine-tuning; GRU has similar perplexity but only 63% MRR, 5% below RCNN.The figure plots heldout-corpus perplexity and dev-set MRR during pre-training.Source: Figure 3 · p. 8, §7 · p. 8

Case Study on Neural Gate

  • The scalar-decay gate visualization assigns higher weights to application names and quoted error messages, though its simplified model is about 4% worse.Source: Figure 4 · p. 9, §7 · p. 9

8 Conclusion

The paper uses gated non-consecutive convolutions to represent questions and reports their effectiveness for community QA question retrieval. It identifies body-to-title pre-training on the raw corpus as integral to the model’s success.

8 Conclusion

  • Gated non-consecutive convolutions map questions to semantic representations and are effective for question retrieval in community QA forums.The architecture is described as helping the model glean key information from lengthy, detail-riddled questions.Source: §8 · p. 9
  • Figure 5 plots the maximum and mean values of the 400-dimensional weight vector 1−λt at each word position.Source: Figure 5 · p. 9
  • Pre-training in an encoder-decoder framework from body to title on the entire raw corpus is integral to the model’s success.Source: §8 · p. 9

Note on Data Pre-processing Issue:

The authors identified user-annotated information in AskUbuntu questions that should be removed for a more realistic evaluation. They reprocessed the dataset and updated the results, while reporting that the relative differences between models did not change.

Note on Data Pre-processing Issue:

Related papers

Loading 1512.05726v2…