Source-linked AI summary
TANDA: Transfer and Adapt Pre-Trained Transformer Models for Answer Sentence Selection
Siddhant Garg, Thuy Vu, Alessandro Moschitti
TL;DR
Answer sentence selection suffers from scarce target data and unstable Transformer fine-tuning. TANDA transfers a pre-trained model using large general AS2 data, then adapts it to the target domain, improving benchmark performance and robustness to noise.
Problem
Small target datasets make Transformer fine-tuning for answer sentence selection unstable, with substantial variance across attempts and occasional single-label predictions.
Method
TANDA sequentially fine-tunes a pre-trained Transformer on a large general AS2 dataset, such as ASNQ derived from Natural Questions, then on target-domain data.
Results
TANDA achieves higher accuracy than traditional fine-tuning and reduces the performance drop to 2.5% when 20% noise is injected into WikiQA and TREC-QA.
Takeaways & Limitations
The transferred model is more stable, easier to adapt, robust to target-domain noise, and supports modular adaptation across domains.
Takeaways & Limitations
Claims that TANDA generalizes to other NLP tasks, especially textual inference, remain conjectural and require empirical confirmation.
Abstract
from arXiv · showhide
We propose TANDA, an effective technique for fine-tuning pre-trained Transformer models for natural language tasks. Specifically, we first transfer a pre-trained model into a model for a general task by fine-tuning it with a large and high-quality dataset. We then perform a second fine-tuning step to adapt the transferred model to the target domain. We demonstrate the benefits of our approach for answer sentence selection, which is a well-known inference task in Question Answering. We built a large scale dataset to enable the transfer step, exploiting the Natural Questions dataset. Our approach establishes the state of the art on two well-known benchmarks, WikiQA and TREC-QA, achieving MAP scores of 92% and 94.3%, respectively, which largely outperform the previous highest scores of 83.4% and 87.5%, obtained in very recent work. We empirically show that TANDA generates more stable and robust models reducing the effort required for selecting optimal hyper-parameters. Additionally, we show that the transfer step of TANDA makes the adaptation step more robust to noise. This enables a more effective use of noisy datasets for fine-tuning. Finally, we also confirm the positive impact of TANDA in an industrial setting, using domain specific datasets subject to different types of noise.
1 Introduction
The paper addresses data scarcity and unstable fine-tuning in answer sentence selection by introducing TANDA, a two-step transfer-and-adapt approach for Transformer models. It reports improved benchmark performance, stability, noise robustness, and validation on industrial data.
- TANDA transfers a pre-trained Transformer to answer sentence selection before adapting it to a target domain.The transfer step uses a large, high-quality AS2 dataset, while the adapt step uses target-domain data.
- ASNQ is a large AS2 dataset created by transforming the Natural Questions corpus from machine reading into answer sentence selection.The dataset supports TANDA’s transfer step, which requires large and accurate supervised data.
- Experiments cover WikiQA, TREC-QA, and three industrial datasets whose questions derive from Alexa traffic and whose candidates come from web data.The industrial experiments confirm the reported robustness to noise.
- TANDA produces more stable fine-tuning behavior, with smooth and convex accuracy across adaptation epochs for estimating optimal parameters.The approach reduces the need for selecting parameters by trial and error.
- With 20% injected noise, TANDA’s performance drop is as low as 2.5% on WikiQA and TREC-QA, up to one order of magnitude lower than traditional fine-tuning.The reported robustness extends to real-world datasets built from Alexa traffic data, where noise comes from real sources.
2 Related Work
Prior AS2 systems use neural architectures with attention and comparison mechanisms, while Transformer models offer transferable representations but can be unstable when fine-tuned on small target datasets. TANDA addresses this instability through separate transfer and adaptation steps.
- AS2 approaches: Recent AS2 models use distributed representations, intrapair and cross-pair similarities, attention, and compare-aggregate architectures.These approaches model relationships within and across question–answer pairs.
- Transformer models: Pre-trained Transformer models capture word dependencies from large-scale data and can be adapted to downstream tasks through fine-tuning.BERT and related models pre-train on surrogate objectives such as masked language modeling and next sentence prediction.
- AS2 challenges: Small target datasets make transfer from a language model to AS2 unstable, producing single-label predictions and high variance across fine-tuning attempts.The paper attributes this difficulty to the large number of Transformer parameters and limited examples.
- TANDA: TANDA first fine-tunes a Transformer on a large general-purpose AS2 dataset, then adapts the resulting model to the target domain.The transfer step specializes the model to AS2, while adaptation handles domain-specific questions and answers.
- TANDA: The transfer step is reported to reduce adaptation data requirements, stabilize the model, and improve robustness to noisy data.These claims are evaluated empirically in the paper.
4 Answer-Sentence Natural Questions
ASNQ is a large AS2 corpus derived from Natural Questions by converting paragraph-level machine-reading annotations into sentence-level answer candidates. It uses graded negative examples to represent different levels of answer confusion.
- Dataset construction: ASNQ was built from Natural Questions because existing AS2 datasets are small and a large, accurate corpus was needed for TANDA’s transfer step.Natural Questions was originally designed for machine reading and is not directly usable for AS2 because long answers contain multiple sentences.
- Candidate labeling: Positive ASNQ candidates are sentences in NQ long-answer paragraphs that contain annotated short answers; remaining document sentences are labeled negative.Negative examples are grouped by their relationship to the long answer and short-answer string.
- Candidate labeling: ASNQ contains four labels describing different levels of candidate confusion, and negative-example generation is intended to improve robustness against similar incorrect answers.The same processing is applied to the training and development sets.
- Dataset statistics: ASNQ has 57,242 distinct training questions and 2,672 distinct development questions; TANDA uses labels 1, 2, and 3 as negatives and label 4 as positive.Its training scale is an order of magnitude larger than most public AS2 datasets.
5 Experiments on Standard Benchmarks
Experiments on WikiQA and TREC-QA show that TANDA improves answer sentence selection over standard fine-tuning, while also providing greater stability and robustness to noisy data.
- WikiQA: RoBERTa-Large TANDA using ASNQ →WikiQA achieves 0.920 MAP and 0.933 MRR, establishing a new state of the art.
- TREC-QA: RoBERTa-Large TANDA with ASNQ →TREC-QA achieves 0.943 MAP and 0.974 MRR, outperforming the previous state of the art.
- TREC-QA: 0.898 MAP and 0.929 MRR from combined ASNQ ∪TREC-QA fine-tuning are lower than TANDA's 0.912 MAP and 0.951 MRR.This comparison shows sequential fine-tuning outperforming simply combining the datasets for BERT-Base.
- Stability: TANDA produces lower variance across adaptation epochs than standard fine-tuning, increasing the probability of selecting an optimal epoch on the development set.
- Transfer data: ASNQ-based transfer performs significantly better than QNLI-based transfer on WikiQA and enables better TREC-QA adaptation under TANDA.These results support ASNQ's suitability as a general and accurate AS2 transfer dataset and the importance of the transfer step.
6 Experiments on data from Alexa
Alexa-derived experiments evaluate TANDA across question sources and retrieval systems, including a larger noisy training set, and find consistent gains over standard fine-tuning.
- Datasets: Alexa test sets vary question sources and retrieval systems to provide more general results across industrial conditions.Samples include Natural Questions and Alexa-user questions, with candidates retrieved by elastic search or a commercial search engine.
- Results: ASNQ transfer always improves over standard fine-tuning across the three Alexa dataset samples.
- Results: BERT-Large TANDA improves over BERT-Base TANDA on all three Alexa dataset samples.
- Results: RoBERTa produces higher improvement with TANDA than BERT on the Alexa datasets.
- Results: Training on noisy Alexa data while testing on accurate datasets shows TANDA is robust to real-world noise and yields significantly large gains over standard fine-tuning.
7 Conclusions
TANDA uses sequential transfer and domain-adaptation fine-tuning for answer sentence selection, producing strong benchmark and industrial results. Its intermediate model improves stability, noise robustness, and scalability, while broader task generalization remains an open question.
- TANDA performs two sequential fine-tuning steps: transfer on a large, high-quality dataset, followed by target-domain adaptation.The approach was tested on answer sentence selection.
- TANDA improves answer sentence selection over the state of the art on WikiQA and TREC-QA and produces similar results in an industrial setting.The conclusions report the same results and conclusions across academic benchmarks and industrial data.
- The transferred model is more stable, easier to adapt, and robust to noise in target-domain data.The first TANDA step also produces an intermediate model that can be reused for other target applications.
- TANDA factorizes the main computation into an initial transfer step, improving scalability when adapting to multiple smaller target domains.After adapting a Transformer to a general task such as AS2, only the adaptation step is needed for each domain.
- Sequential fine-tuning over ASNQ followed by TREC-QA is better than simple union training, although union training may achieve similar accuracy improvements.The union approach requires greater weighting for target data and does not retain TANDA’s modularity.
- Whether TANDA generalizes to other NLP tasks remains an open question, especially for tasks such as paraphrasing and textual entailment.These tasks may involve different relations between paired texts than question answering.