Source-linked AI summary
Injecting Numerical Reasoning Skills into Language Models
Mor Geva, Ankit Gupta, Jonathan Berant
TL;DR
Large pre-trained language models struggle to acquire numerical reasoning from language modeling alone, while existing solutions rely on inflexible specialized architectures. The paper injects numerical skills into BERT through automatically generated synthetic data and additional multi-task pre-training. GENBERT reaches comparable state-of-the-art performance on DROP, generalizes to math word problems, and maintains high performance on SQuAD.
Problem
Pre-trained language models struggle with high-level numerical reasoning, while existing numerical-reasoning models use specialized architectures with limited flexibility.
Method
GENBERT adds pre-training on automatically generated numerical and textual data using a general-purpose encoder-decoder architecture and a multi-task language-modeling objective.
Results
GENBERT reaches comparable state-of-the-art performance on DROP, generalizes to math word-problem datasets, and maintains high performance on SQuAD.
Takeaways & Limitations
The approach provides a method for injecting numerical reasoning skills into pre-trained language models when automatic data generation is possible.
Takeaways & Limitations
Existing numerical-reasoning architectures have constrained output spaces and face difficult search problems when expanded to arbitrary computations.
Abstract
from arXiv · showhide
Large pre-trained language models (LMs) are known to encode substantial amounts of linguistic information. However, high-level reasoning skills, such as numerical reasoning, are difficult to learn from a language-modeling objective only. Consequently, existing models for numerical reasoning have used specialized architectures with limited flexibility. In this work, we show that numerical reasoning is amenable to automatic data generation, and thus one can inject this skill into pre-trained LMs, by generating large amounts of data, and training in a multi-task setup. We show that pre-training our model, GenBERT, on this data, dramatically improves performance on DROP (49.3 $\rightarrow$ 72.3 F1), reaching performance that matches state-of-the-art models of comparable size, while using a simple and general-purpose encoder-decoder architecture. Moreover, GenBERT generalizes well to math word problem datasets, while maintaining high performance on standard RC tasks. Our approach provides a general recipe for injecting skills into large pre-trained LMs, whenever the skill is amenable to automatic data augmentation.
1 Introduction
The paper addresses the difficulty of learning numerical reasoning from language-modeling objectives by injecting the skill through automatically generated data and additional pre-training. GENBERT uses a general-purpose architecture and retains language understanding while supporting numerical reasoning.
- Motivation: Pre-trained language models encode linguistic information but struggle with high-level numerical reasoning from a language-modeling objective alone.Numerical questions may require understanding number values, computing differences, and generating answers absent from the input.
- Motivation: Existing numerical-reasoning models use specialized modules with restricted computation and limited flexibility.Their supported operations include counting only up to 9 or addition and subtraction over only a few numbers.
- Approach: The proposed approach injects reasoning skills through additional pre-training on automatically generated numerical and textual data.Numerical data teaches arithmetic operations, while textual data teaches computations expressed in pseudo-natural language.
- Approach: GENBERT generates output numbers token-by-token with a standard encoder-decoder architecture, allowing answers to be extracted or generated.Pre-training uses a multi-task language-modeling setup to help avoid catastrophic forgetting of the original model’s linguistic information.
- Results: Pre-training equips GENBERT to solve its numerical pre-training tasks and reach comparable state-of-the-art performance on DROP.The experiments also examine fine-tuning on numerical reasoning and reading comprehension datasets.
- Conclusion: The method is presented as a general approach for injecting skills into pre-trained language models when automatic data generation is possible.GENBERT combines generative and extractive abilities with a framework for generating numerical and textual synthetic data.
2 Numerical Reasoning Over Text
Numerical reasoning over text requires models to combine language understanding with computations whose results may be absent from the input. Existing symbolic, distributed, and hybrid approaches each impose important constraints on flexibility, differentiability, or data efficiency.
- Task formulation: Numerical reasoning over text maps a question and context to either extracted spans or computed numbers.The task is formulated as reading comprehension, but answers may require internal numerical computation over entities and numbers in the context.
- Existing approaches: Symbolic systems generate numerical expressions for external evaluation, but expression search is discrete and combinatorial, and final answers may be text spans.This creates optimization and answer-type limitations for DROP-style questions.
- Existing approaches: Distributed systems directly generate answers, simplifying training but requiring the model to learn numerical computation from relatively small target datasets.The paper reports that this approach leads to low performance in general.
- Existing approaches: Hybrid models combine extractive QA with specialized heads for spans, counting, and arithmetic, while marginalizing over answer-producing heads and expressions.Their specialized modules support only limited numerical computations, including counting to 9 and combinations of a few numbers.
- Limitations: These hybrid architectures are difficult to expand because the expression space grows exponentially and external calculators cannot handle questions requiring computation followed by span extraction.The latter limitation arises when the computed value determines which text span answers the question.
- Proposed direction: The proposed direction is to generate large amounts of data automatically and use additional pre-training to endow a pre-trained LM with numerical skills.The stated goal is an end-to-end differentiable model with an output space controlled through data generation.
3 GENBERT: A BERT-based Model for Generating Arbitrary Outputs
GENBERT extends a BERT-initialized Transformer with a decoder so it can extract spans or generate arbitrary answers. Two synthetic-data pre-training steps teach numerical operations and pseudo-natural-language computations while preserving a standard multi-task LM setup.
- Architecture: GENBERT combines a BERT-initialized Transformer encoder with a generative decoder that performs numerical computations internally.The model is designed to generate arbitrary outputs rather than relying on specialized arithmetic modules.
- Architecture: The encoder and decoder share tied weights, with feed-forward transformations and distinct boundary tokens separating their representations and sequences.These choices fully initialize the Transformer model with BERT weights while distinguishing encoding from decoding.
- Architecture: GENBERT uses a type head to select among context-span, question-span, and decoder heads, allowing answers to be extracted or generated.The decoder supplies the generative path, while span heads retain extractive answering abilities.
- Pre-training modifications: Digit-by-digit tokenization improves the model’s ability to learn numerical operations, while random positional shifts reduce overfitting to numbers at fixed input positions.Random shifts are applied to short inputs by offsetting position IDs within the 512-token range.
- Training: Training marginalizes over all span-based ways to predict the gold answer, while generated answers receive token-by-token decoder probabilities.The architecture supports both extractive and generative answer pathways.
- Initial evaluation: Without synthetic numerical pre-training, BERT-initialized GENBERT achieves 46.1 EM and 49.3 F1 on DROP, roughly 20 points below prior models.This motivates generating training data to provide numerical skills before fine-tuning on DROP.
4 Pre-training Tasks for Numerical Skills
The paper pre-trains GENBERT on automatically generated numerical and textual data to teach numerical operations and their expression in pseudo-natural language, while retaining standard language modeling.
- Numerical Data: Six numerical templates generate expressions and solutions for operations chosen based on DROP, with extensions to other domains described as straightforward.The templates use automatically sampled numeric values and operations such as combinations, min/max/average, and related numerical reasoning skills.
- Textual Data: A simple grammar generates textual passages in which entities and containers change through events, producing contexts that require numerical reasoning.Templates abstract numbers, entities, containers, attributes, and verbs into world-state updates, then instantiate passages from a small vocabulary.
- Textual Data: Questions instantiated from each generated world state teach the model to track events and perform numeric and discrete operations.The framework creates 13 question templates for seven skills, with answers computed from the maintained world state.
- Training GENBERT: The multi-task pre-training objective combines numerical-data and textual-data losses with a weighted masked-language-modeling loss.The training procedure samples mini-batches from ND, TD, and MLM datasets and optimizes Lmodel(XND) + Lmodel(XTD) + λ·Lmlm(XMLM).
5 Experimental Evaluation
The experiments test whether automatically generated numerical and textual pre-training data teaches GENBERT numerical reasoning while preserving language performance and transferring across architectures and tasks.
- 5.1 Pre-training Performance: More than 96% accuracy across ND, TD, and ND+TD settings shows GENBERT learns the intended numerical skills from generated data.ND and TD are numerical-data and textual-data pre-training tasks.
- 5.2 Numerical Reasoning Performance: 68.8 EM on DROP after ND+TD pre-training rises from 46.1 EM without pre-training and matches MTMSN’s 68.2 EM.The two pre-training skills are reported as complementary.
- 5.2 Numerical Reasoning Performance: Answer-type analysis finds gains concentrated on number answers, an advantage on span answers, and a substantial MTMSN advantage on non-contiguous span lists.The span advantage is attributed as a probable consequence of span questions requiring internal numerical computation, while MTMSN uses a specialized multi-span procedure.
- 5.3 Generalization to MWP: GENBERT+ND+TD improves zero-shot performance on addition-and-subtraction math word problems compared with GENBERT, although MTMSN outperforms it.ND performs much better than TD when the context is short.
- 5.3 Generalization to MWP: All models struggle with expressions containing more terms, but combined ND+TD has a smaller two-to-three-term performance drop than either single-task variant.The passage reports complete failure beyond three terms and interprets the smaller drop as evidence that both tasks improve robustness.
- 5.5 Error Analysis: Error analysis finds 43% of sampled failures require skills absent from pre-training or non-numerical reasoning, while 23% involve inaccurate predictions.The authors suggest extending pre-training to additional numerical skills and larger number ranges.
- 5.4 GENBERT With Other Architectures: GENBERT+ND+TD retains almost identical SQuAD performance to BERT and improves DROP initialization results by about 2 EM points for NABERT+ and MS-TAG.Reported improvements are 63.0 → 65.1 EM for NABERT+ and 67.3 → 69.3 EM for MS-TAG.
6 Related Work
Related work studies specialized numerical-reasoning architectures, symbolic approaches, neural mathematical reasoning, and skill-oriented pre-training tasks.
- Numerical reasoning models: Most DROP numerical-reasoning models augment extractive QA with specialized modules, while some symbolic models output programs with text operations.GENBERT instead performs numerical computations latently inside the model.
- Neural mathematical reasoning: Prior research analyzes mathematical reasoning in neural models over text and arithmetic problems.The cited work spans reasoning over text and standalone arithmetic tasks.
- Skill-oriented pre-training: Other work designs pre-training tasks to teach cross-lingual mappings or improve information retrieval.These studies provide related examples of injecting additional skills through pre-training objectives.
7 Conclusions
The paper concludes that numerical reasoning can be injected into pre-trained language models through automatic data generation and a general-purpose model.
- 7 Conclusions: GENBERT uses automatically generated synthetic examples to learn numerical reasoning while retaining a general-purpose architecture.The conclusion frames automatic data generation as an assumption underlying the general method.
- 7 Conclusions: GENBERT performs on par with state-of-the-art numerical-reasoning-over-text models of the same size.The conclusion reports successful learning of the numerical skills alongside this performance comparison.
A.1 Synthetic Numerical Data Generation
Synthetic numerical data is generated from sampled integers and floats using templates for arithmetic and aggregation operations.
- Synthetic numerical data: Integers are sampled from 0 through 20K and split into disjoint training and development sets to test generalization.The templates include signed combinations of up to four two-decimal-place floats.
A.2.1 Sentence template extraction
The appendix describes how synthetic numerical-reasoning data is constructed from abstracted sentence and question templates, controlled vocabularies, and generated Wikipedia-based masked-LM training data. It also specifies experimental settings and categorizes model errors on DROP.
- A.2.1 Sentence template extraction: Math word-problem sentences are split and abstracted into NUM, VERB, ENT, CONT, and ATTR categories for template extraction.The extraction process operates independently on each sentence.
- A.2.1 Sentence template extraction: The generation framework adds agent and environment containers, distinguishing active collectors or droppers from passive places or time periods.Agents include people or organizations, whereas environments include places or time periods.
- A.2.1 Sentence template extraction: Two-level containers encode inclusion relations, so entities anchoring near Devonport also anchor near England.The example represents a relation between a city and its containing country.
- A.2.1 Sentence template extraction: The 12 most common extracted sentence templates are used to generate synthetic data.These templates are listed in Table 8.
- A.2.2 Template instantiation: Templates are instantiated with small domain-specific vocabularies covering history and the National Football League.The vocabularies map abstract categories to words and are distributed in JSON format.
- A.2.3 Question templates: Question templates test seven numerical-reasoning skills, with 13 templates documented in Table 9.Their placeholders are filled using values from the world state after passage generation.
- A.3 Data for Masked LM task: Masked-LM training data combines 156K Wikipedia pages selected by title keywords with remaining pages filtered for paragraphs containing at least 15 numbers.Pages are tokenized using DT and chunked into 512-token sequences.
- A.4 Experimental Setup: Experiments use an older Hugging Face Transformers version, with hyperparameters reported in Table 10, while Table 11 analyzes GENBERT+ND+TD errors on DROP.The error analysis covers 100 development examples excluding multi-span answers, and Table 11 is based on 85 manually analyzed examples.