Source-linked AI summary
Representing Numbers in NLP: a Survey and a Vision
Avijit Thawani, Jay Pujara, Pedro A. Szekely, Filip Ilievski
TL;DR
NLP systems commonly represent numbers as words despite evidence that numeric understanding requires specialized treatment. This survey organizes numeracy into seven tasks and analyzes 18 encoder and decoder designs, finding that default subword representations are clearly suboptimal while holistic numeracy remains unresolved.
Problem
NLP systems rarely give special consideration to numbers, despite numbers’ distinct representation in neuroscience and their ubiquity in text.
Method
The paper develops taxonomies of seven numeracy tasks and number representations, analyzes 18 encoders and decoders, and synthesizes design guidance and a unified vision.
Results
Default subword segmentation with lookup embeddings is clearly suboptimal for numbers, while character-tokenized models and scientific notation show performance benefits in reported studies.
Takeaways & Limitations
Holistic numeracy requires further research on the specificity, coverage, and inductive bias needed to represent numbers effectively across tasks.
Takeaways & Limitations
The work focuses on the Hindu-Arabic numeral system and English number words, excluding other number systems still in use.
Abstract
from arXiv · showhide
NLP systems rarely give special consideration to numbers found in text. This starkly contrasts with the consensus in neuroscience that, in the brain, numbers are represented differently from words. We arrange recent NLP work on numeracy into a comprehensive taxonomy of tasks and methods. We break down the subjective notion of numeracy into 7 subtasks, arranged along two dimensions: granularity (exact vs approximate) and units (abstract vs grounded). We analyze the myriad representational choices made by 18 previously published number encoders and decoders. We synthesize best practices for representing numbers in text and articulate a vision for holistic numeracy in NLP, comprised of design trade-offs and a unified evaluation.
1 Introduction
Numbers are ubiquitous and require context-sensitive numerical understanding, yet NLP systems often represent them like ordinary words. The survey responds by organizing numeracy tasks and number representations while motivating better numeric processing.
- Motivation: Numbers require decoding quantities, interpreting units and context, and reasoning about approximate variation in ordinary text.The same numeral can denote different units and value expectations depending on context.
- Motivation: 150 million numbers occur across English Wikipedia’s 6 million pages, underscoring numeracy’s centrality to language understanding.
- The problem: NLP systems filter numbers, collapse them into UNK, or split them into arbitrary subword tokens.For example, 1234 may be segmented as 12-34, 123-4, or 1-234.
- Evidence: On DROP, BERT performs five times worse for numeric answers than text spans, while character tokenization and scientific notation can improve performance.These findings motivate deeper study of number representations.
- Survey scope: The paper surveys numeracy tasks and representations, proposes NLP’s first taxonomies, synthesizes takeaways, and presents a unifying future vision.
2 Tasks
The survey separates numeracy into distinct tasks organized by exactness and whether quantities are abstract or grounded in units. It covers decoding, comparison, arithmetic, estimation, facts, language modeling, and downstream applications.
- Task landscape: Numeracy tasks span retrieval, counting, arithmetic, decoding strings to values, magnitude comparison, and estimating real-world quantities.
- Taxonomy: The taxonomy organizes tasks along granularity—exact versus approximate—and units—abstract versus grounded.Grounded quantities require understanding numbers in the context of words.
- Exact tasks: Simple arithmetic operates over numbers alone, whereas arithmetic word problems ground operations in textual scenarios.
- Grounded tasks: Exact facts encode commonsense assertions, while measurement estimation asks models to approximate object properties and physical attributes.Examples include dice having six faces and estimating a telephone’s weight.
- Language modeling: Numerical language modeling predicts numbers in naturally occurring text and is evaluated with regression metrics such as mean absolute or root mean squared error.
- Applications: Numeracy tasks also support applications including sarcasm detection, financial claim identification, and question answering.
3 Methods
The survey classifies number representations as string-based or real-based, with design choices involving notation, tokenization, pooling, scale, direction, and discretization. It reviews representative encoders and decoders, including digit, value, distributional, and handcrafted approaches.
- Taxonomy: String-based representations treat numbers as surface forms, while real-based methods compute over numerical values using encoders f: R → R^d or decoders g: R^d → R.
- String-based methods: String-based design varies notation, tokenization, and pooling; character-level inputs and pooled digit embeddings are alternatives to arbitrary subword segmentation.
- Real-based methods: Real-based methods vary by encoding direction, linear or logarithmic scale, and continuous or binned discretization.Continuous value functions over broad numeric ranges can be practically infeasible, motivating bins.
- Coverage: The review compares methods across seven numeracy tasks using a taxonomy table whose cells identify representative, not exhaustive, prior experiments.
- Representative methods: Prior systems include digit-pooling RNNs or CNNs, scientific-notation NumBERT, digit-tokenized GenBERT, exponent embeddings, DICE, and value-based neural encoders.
- Real-based methods: Log-Laplace decoding samples a latent value from a context-parameterized distribution and exponentiates it to produce the numeric output.
4 Results
Published experiments show that number representations affect numeracy unevenly across abstract probes, arithmetic, language modelling, measurement estimation, and exact numerical facts.
- Abstract Probes: Word embeddings vastly outperform random baselines on abstract numeration, magnitude-comparison, and sorting probes.DICE, Value, and Log Value embeddings perform especially well, while DigitCNNs are the best encoders on these probes.
- Abstract Probes: Character-tokenized models generally outperform subword-tokenized models on abstract numerical probes.ELMo is given as an example of a character-tokenized model, while BERT represents the subword-tokenized comparison.
- Arithmetic: GPT-3 performs extremely well at zero-shot simple arithmetic when operand digit counts are low, while digit-level tokenization improves arithmetic performance.Arithmetic word-problem solvers often predict an equation and then fill it with question-specific values, bypassing numeric embeddings.
- Masked Language Modelling: NumBERT matches BERT’s masked-language-modelling loss and nearly matches its GLUE performance, while exponent embeddings can outperform mantissa modelling for numeric language modelling.Gaussian Mixture Models are reported as the best decoders for causal numeric language modelling.
- Measurement Estimation: NumBERT is a better text encoder than BERT for measurement estimation, with decoder performance depending on the dataset’s ground-truth distribution.MCC performs better than RGR on Distribution over Quantities, but not on VerbPhysics.
- Exact Facts: BERT and RoBERTa capture limited numerical commonsense, although Wikipedia-based finetuning improves results on exact numerical facts.Commonsense question answering is among the hardest Numbergame tasks, and these evaluations use exact-match metrics.
5 Recommendations
The paper recommends representation choices according to notation, scale, task granularity, and encoding or decoding role, while noting that several alternatives remain insufficiently compared.
- Rule of thumb for string-based methods?: Scientific notation is recommended over decimal notation, and character-level tokenization is preferred over subword-level tokenization.The proposed explanation is that models can attend mainly to exponent embeddings rather than mantissas.
- Encoding vs Decoding numbers?: Pooled and unpooled representations cannot yet be ranked because they lack a controlled comparison.The paper also distinguishes encoding numbers into embeddings from decoding embeddings into numbers, noting that some encoders are not easily reversible.
- Rule of thumb for real-based methods?: Log-scale representations are preferred over linear-scale representations, while binning can outperform continuous value prediction when ground-truth distributions are available.The paper notes that the evidence for log scale is less rigorous than comparable cognitive-science studies.
- Can we mix-and-match multiple methods?: Combining exponent embeddings with scientific-notation DigitRNN embeddings barely outperforms exponent embeddings alone, while mixtures of real- and string-based methods remain untested.This result suggests that adding another representation is not automatically beneficial in the reported encoding experiment.
- Which methods for which tasks?: Abstract tasks can serve as early probes for grounded tasks, with arithmetic finetuning helping GenBERT on downstream question answering.High DICE scores on numeration and magnitude comparison are presented as indicators of possible gains in numeric language modelling.
- Which methods for which tasks?: Real-based methods work well for approximate tasks but not for exact arithmetic word problems or commonsense tasks.DigitRNNs are described as broad-purpose encoders, whereas distribution-modelling methods such as DExp are effective decoders.
6 Vision for Unified Numeracy in NLP
The paper envisions holistic numeracy through broad evaluation, transfer across numerical skills, explicit representation trade-offs, and applications beyond numbers. It argues that unified numerical understanding should connect continuous numeric meaning to broader language understanding.
- Evaluation: A unified numeracy benchmark should cover exact and approximate skills across abstract and grounded tasks, enabling evaluation of distinct numerical abilities.The proposed evaluation parallels aggregated NLP leaderboards and extends preliminary efforts such as Numbergame.
- Evaluation: Transfer studies should test whether training on one numeracy skill improves performance on another, ideally across every pair of tasks.Arithmetic training improves DROP QA performance, while numeration training improves Numeracy600K order-of-magnitude prediction.
- Design Principles: Number representations trade off inductive bias against data-driven variance, while coverage choices determine whether models represent fixed ranges or broader numerical domains.Lookup embeddings permit greater variance; value embeddings and DICE impose continuity, while fixed bins restrict coverage and string methods can fall back to subword or character representations.
- Design Principles: Future work should determine whether exact and approximate numeracy need specialized modules or can share one representation.This design question follows parallel proposals in cognitive science and remains unresolved in the paper’s vision.
- Broader Impact: Numeracy can extend beyond number reasoning by modeling other ordered or continuous concepts, including gradable adjectives, weekdays, and months.The paper discusses value-based models for gradable adjectives and periodic functions for calendar concepts.
- Broader Impact: Numerical understanding can support world knowledge by relating quantities to concepts and transferring that knowledge to contexts without explicit numbers.The paper illustrates this with salary comparisons between programmers and researchers and a masked occupation prediction.
7 Conclusion
The paper concludes that current number representations are inadequate and that holistic numeracy remains an open research problem. Its survey identifies unresolved choices about task specificity, numerical coverage, and inductive bias.
- 7 Conclusion: Default subword segmentation with lookup embeddings is clearly suboptimal for representing numbers.This is one of the paper’s two major conclusions from surveying and contextualizing recent numeracy work.
- 7 Conclusion: A holistic solution to numeracy still requires answers about the needed specificity, coverage, and inductive bias.The paper presents these as unresolved research questions rather than settled design choices.
Ethical Considerations
The paper’s scope is limited to the Hindu-Arabic numeral system and English number words. It calls for future work to include other number systems still in use.
- Ethical Considerations: The work focuses on Hindu-Arabic numerals and English number words, excluding other number systems that remain in use.The authors explicitly encourage follow-up research to consider those additional systems.
A Other Numeracy Tasks
Several numeracy-related tasks fall outside the paper’s main taxonomy, including paraphrasing, entailment, numeral categorization, fused-head resolution, counting, and domain-specific reasoning.
- A Other Numeracy Tasks: Numeric paraphrasing maps different surface forms, such as twelve, 12, and dozen, to the same number before later reasoning.The task cuts across the paper’s other numeracy tasks because equivalent forms must be identified first.
- A Other Numeracy Tasks: Quantity entailment extends numeric equivalence to relations such as entailment and contradiction between quantity statements.An example is that being 16 years old entails being a teenager.
- A Other Numeracy Tasks: Numeral understanding categorizes numbers into types such as percentages, prices, dates, times, and quantities.The task also includes subcategories within these numerical forms.
- A Other Numeracy Tasks: Fused-head resolution grounds implicit numerical meanings by identifying contextual heads such as a.m. or o’clock for the number 11.The example connects numerical interpretation to the sentence context rather than the numeral alone.
- A Other Numeracy Tasks: Counting requires tracking discrete instances and learning that the final counter value denotes the number of entities.The paper notes that language models may possess counting skills but still need to map counts to words or numerals for answering questions.
- A Other Numeracy Tasks: Domain-specific numerical tasks combine mathematical skills with background knowledge in areas such as physics, chemistry, and school science.Examples include calculating train distance, chemical mass percentage, and solving science problems.