Source-linked AI summary
Learning and Evaluating General Linguistic Intelligence
Dani Yogatama, Cyprien de Masson d'Autume, Jerome Connor, Tomas Kocisky, Mike Chrzanowski, Lingpeng Kong, Angeliki Lazaridou, Wang Ling, Lei Yu, Chris Dyer, Phil Blunsom
TL;DR
The paper asks whether modern NLP models can reuse linguistic knowledge to adapt rapidly across tasks rather than merely achieve high final performance. It analyzes pretrained models using multitask, transfer, continual-learning, and online-codelength evaluations, finding substantial fine-tuning requirements, poor same-task cross-dataset generalization, and catastrophic forgetting.
Problem
Existing final held-out performance metrics do not measure how rapidly models adapt to new tasks using previously acquired knowledge.
Method
The paper analyzes pretrained NLP models across transfer and continual-learning experiments and adds online prequential codelength to measure sample-efficient adaptation.
Results
Existing models require considerable task-specific fine tuning, generalize poorly across datasets from the same task without fine tuning, and suffer catastrophic forgetting.
Takeaways & Limitations
Progress toward general linguistic intelligence requires more sophisticated transfer and continual-learning methods and reusable memory that adapts to domain shifts.
Takeaways & Limitations
The study focuses on passive learning from static corpora and does not evaluate adversarial robustness or multilingual understanding.
Abstract
from arXiv · showhide
We define general linguistic intelligence as the ability to reuse previously acquired knowledge about a language's lexicon, syntax, semantics, and pragmatic conventions to adapt to new tasks quickly. Using this definition, we analyze state-of-the-art natural language understanding models and conduct an extensive empirical investigation to evaluate them against these criteria through a series of experiments that assess the task-independence of the knowledge being acquired by the learning process. In addition to task performance, we propose a new evaluation metric based on an online encoding of the test data that quantifies how quickly an existing agent (model) learns a new task. Our results show that while the field has made impressive progress in terms of model architectures that generalize to many tasks, these models still require a lot of in-domain training examples (e.g., for fine tuning, training task-specific modules), and are prone to catastrophic forgetting. Moreover, we find that far from solving general tasks (e.g., document question answering), our models are overfitting to the quirks of particular datasets (e.g., SQuAD). We discuss missing components and conjecture on how to make progress toward general linguistic intelligence.
1 Introduction
The paper argues that strong task performance does not establish general linguistic intelligence: models must reuse linguistic knowledge, adapt efficiently, generalize across tasks and domains, and avoid forgetting.
- Motivation: Recent deep-learning advances have substantially improved NLP performance on important tasks, including machine translation and question answering.The passage notes claims of surpassing human performance on Chinese–English translation and SQuAD document question answering.
- Motivation: Many recent datasets may be easy because crowd-sourced construction does not adequately approximate the natural distributions they intend to represent.
- Related approaches: Multitask models provide prototypes for general linguistic intelligence, although their overall results remain below task-specific models.
- Evaluation goals: The proposed evaluation paradigm rewards broad language coverage, reusable representations, resistance to catastrophic forgetting, and rapid adaptation under domain shifts.
- Contributions and findings: Experiments find that existing models need substantial task-specific fine tuning, generalize poorly across same-task datasets without fine tuning, and suffer catastrophic forgetting.
2 Tasks
The experiments span reading comprehension, semantic role labeling, relation extraction, and natural language inference, using datasets that vary in format, genre, and task formulation.
- Task coverage: The main evaluation tasks are SQuAD reading comprehension and MultiNLI natural language inference, supplemented by other training and out-of-domain datasets.
- Reading Comprehension: SQuAD, TriviaQA, and QuAC all test reading comprehension but differ in source, question style, evidence collection, and dialogue format.SQuAD uses Wikipedia articles; TriviaQA uses trivia questions with web evidence; QuAC is information-seeking dialogue.
- Semantic Role Labeling: Semantic role labeling is formulated as span prediction from a sentence and a predicate-driven wh-question.
- Relation Extraction: Relation extraction is formulated as question answering by associating questions with relation slots.
- Natural Language Inference: Natural language inference predicts entailment, contradiction, or neutrality from sentence pairs using MNLI and SNLI.
3 Models
The study compares pretrained self-attention and recurrent models, adding task-specific prediction components to evaluate transfer across diverse language tasks.
- Model classes: The models comprise self-attention architectures based on Transformer and recurrent neural networks.
- Self-attention model: The self-attention model starts from pretrained BERT, trained with masked language modeling and next sentence prediction.
- Self-attention model: Each BERT task uses an additional task-specific final prediction layer, while retaining the default vocabulary and leaving new-word learning for future work.
- Recurrent model: The recurrent setup augments pretrained ELMo with a 300-dimensional bidirectional LSTM and BiDAF to aggregate context and question representations.
- Training: Hyperparameters include batch size, learning rate, dropout rate, and the ℓ2 regularization constant.
4 Evaluating Transfer
The paper supplements final held-out performance with online codelength, an information-theoretic measure intended to quantify how rapidly models learn new tasks from sequential examples.
- Motivation: Standard held-out metrics measure final task performance but do not directly capture rapid adaptation using previously acquired knowledge.
- Online codelength: The online prequential code evaluates learning behavior alongside traditional performance metrics.
- Online codelength: The procedure orders examples, trains on increasing prefixes, and uses each learned model to encode subsequent labels.
- Online codelength: The number of possible labels determines the uniform-code term in the codelength calculation.
- Approximation: A practical approximation evaluates the model only M times on increasing subsets S1 through SM, with SM equal to the full dataset.
- Properties and limitations: Online codelength can incorporate model complexity and training cost, but the experiments require a predefined dataset ordering and do not fully account for those factors by default.
5 Experiments
Across experiments, pretrained models adapt better than models trained from scratch, but still need substantial task-specific data, generalize poorly across datasets, and forget earlier tasks during continual learning. Online codelength reveals transfer benefits that final performance metrics may understate, while random curricula reduce forgetting at the cost of requiring all tasks upfront or repeated retraining.
- 5.1 Unsupervised Pretraining: Approximately 40,000 training examples are needed for both SQuAD and MNLI models to approach their asymptotic errors, despite pretrained modules.BERT outperforms ELMo and approaches its asymptotic error faster; BERTscratch reaches only 14.9 F1 on SQuAD and 66.3 accuracy on MNLI after 500,000 iterations.
- 5.1 Unsupervised Pretraining: 14.9 F1 on SQuAD and 66.3 accuracy on MNLI are achieved by BERTscratch after 500,000 training iterations, far below a similar pretrained model.The comparison uses a Transformer with BERT-like architecture trained from scratch on the full training sets.
- 5.1 Unsupervised Pretraining: 102.42 kbits for BERT versus 112.96 kbits for ELMo on SQuAD, and 89.25 versus 132.17 kbits on MNLI, show lower online codelength for BERT.Codelengths generally correlate with F1, exact match, and accuracy, with lower codelength indicating better performance in these experiments.
- 5.2 Beyond Unsupervised Pretraining: Pretraining on supervised tasks slightly improves final exact match and F1, but significantly improves codelength by enabling prediction before SQuAD-specific examples.BERT and ELMo reach 62.9 and 43.3 F1, respectively, after other-task training and before seeing SQuAD examples.
- 5.3 Generalization: High-performing SQuAD models do not perform well on TriviaQA, QuAC, QA-SRL, and QA-ZRE without examples from those datasets, exposing a gap between learning a task and learning a dataset.The authors note that the datasets have different distributions and that SQuAD-trained models still require other-dataset examples for general-purpose question answering.
- 5.4 Curriculum and Catastrophic Forgetting: Continual training on TriviaQA or MNLI rapidly degrades SQuAD performance, whereas a random curriculum preserves prior knowledge and yields reasonable performance across tasks after about 50,000 iterations.The random curriculum requires all tasks at the beginning or retraining on all tasks after a new task appears, leaving effective continual transfer open.
6 Discussion
The experiments indicate that current models remain short of general linguistic intelligence: they need substantial task-specific adaptation, forget earlier knowledge, and generalize poorly across datasets. The paper identifies more sophisticated transfer, continual learning, memory, and curriculum methods as promising directions.
- Existing models still require many task-specific training examples and are highly prone to catastrophic forgetting.Fine-tuning task-specific components remains data-intensive, while learning new tasks can degrade prior performance.
- More sophisticated transfer and continual learning methods are needed to improve robustness to forgetting.The paper cites elastic weight consolidation and progress and compress as potentially promising approaches.
- Memory modules that rapidly adapt to domain shifts and generalize across tasks remain an underexplored component.The paper points to neural cache and dynamic evaluation as evidence for memory-based approaches in language modeling.
- Meta-learning offers a route to rapid adaptation, but natural-language meta-learning remains relatively underexplored and standard methods assume a known task distribution.In meta-learning, model parameters are trained so that a few gradient steps on a new task yield good performance.
- When multiple tasks are available beforehand, curriculum design can substantially affect how well and how quickly models learn.The paper suggests methods such as Population Based Training for designing curricula.
- Task-specific components are a key reason models generalize poorly beyond the datasets on which they were trained.The paper contrasts solving a dataset with learning reusable task-level representations and modules.