Source-linked AI summary
Do Language Embeddings Capture Scales?
Xikun Zhang, Deepak Ramachandran, Ian Tenney, Yanai Elazar, Dan Roth
TL;DR
The paper asks whether pretrained language models capture the scalar magnitudes of objects, a form of knowledge not yet studied in this context. It probes representations for distributions over object attributes and examines numerical representation in pre-training. The models capture substantial scale information, but remain short of robust common-sense scale understanding.
Problem
Whether language-model representations encode distributions of object scale attributes is an unstudied question in this context and a prerequisite for common-sense reasoning.
Source: §1 · p. 1Method
Results
Language models contain significant scale information but remain far from satisfying common-sense scale understanding.
Source: §7 · p. 5Takeaways & Limitations
The findings point to context and numeracy as directions for progress, with scale information transferred from numbers to nouns during pre-training.
Source: §7 · p. 5DoQ is heuristically derived from web text and contains noise, so it serves as a starting point for evaluating models.
Source: §2 · p. 2
Abstract
from arXiv · showhide
Pretrained Language Models (LMs) have been shown to possess significant linguistic, common sense, and factual knowledge. One form of knowledge that has not been studied yet in this context is information about the scalar magnitudes of objects. We show that pretrained language models capture a significant amount of this information but are short of the capability required for general common-sense reasoning. We identify contextual information in pre-training and numeracy as two key factors affecting their performance and show that a simple method of canonicalizing numbers can have a significant effect on the results.
1 Introduction
The paper studies whether language-model representations encode object scale, a prerequisite for common-sense reasoning. Its results show scale information in embeddings, while contextual encoding, number representation, and transfer experiments shape the observed performance.
1 Introduction
- Scalar probing asks whether an object representation predicts a distribution of values for a continuous attribute, accommodating polysemy and natural variation.The introduction frames scalar magnitude prediction as another prerequisite for common-sense reasoning.Source: §1 · p. 1
- Contextual encoders outperform noncontextual ones on scalar probing, despite the task being non-contextual.Source: §1 · p. 1
- Scientific notation for numbers in pre-training significantly affects results, but the direction depends on the evaluation metric.Source: §1 · p. 2
- Zero-shot transfer to relative comparisons and product-price prediction provides evidence that the results extend across datasets.Source: §1 · p. 2
2 Problem Definition and Data
Scalar probing predicts a distribution over an object's values for a continuous attribute, represented through logarithmically spaced magnitude buckets. The study uses DoQ as its primary resource and focuses on nouns and three scales.
2 Problem Definition and Data
- Scalar values are mapped into 12 logarithmically spaced buckets, making the task equivalent to predicting their order of magnitude.Models can predict either a full distribution or a point estimate concentrated in one bucket.Source: §2 · p. 2
- DoQ provides empirical web-derived scalar-value distributions for over 350K nouns, adjectives, and verbs across 10 attributes.Source: §2 · p. 2
- The experiments focus on nouns and the MASS, LENGTH, and PRICE scales, using DoQ distributions as ground truth.DoQ is heuristically derived from web text and contains noise; transfer experiments evaluate probes on Forbes and Choi (2017) and Jianmo Ni (2019) data.Source: §2 · p. 2
- To examine context, the authors also train specialized probes on MASS for animals and PRICE for household products.Source: §2 · p. 2
3 Probing Model
The probing setup extracts object representations from embedding models and trains linear probes to predict scalar magnitudes. It compares point-estimate regression with distributional multi-class classification.
3 Probing Model
- Word2Vec, ELMo, and BERT provide the object embeddings, with contextual encoders receiving objects in artificial sentences.Word2Vec uses averaged name embeddings; BERT and ELMo use sentence representations, including the CLS token for BERT and final state for ELMo.Source: §3 · p. 2
- Linear probes are used to test information readily accessible in representations, and alternate sentence templates yield very similar performance.Source: §3 · p. 2
- Figure 2 illustrates empirical DoQ distributions and MCC predictions for runner length and bill price, including multiple peaks or modes.Source: Figure 2 · p. 3
- Regression predicts the log median value, while multi-class classification predicts a full distribution over scalar-value buckets using soft labels.The classification model uses a softmax output and dense cross-entropy against the DoQ empirical distribution.Source: §3 · p. 3
4 Numeracy through Scientific Notation
The authors investigate whether representing numbers in scientific notation improves scalar probing by making their magnitudes easier for BERT to associate with objects. They introduce NumBERT by replacing numbers in pre-training data with exponent-and-mantissa representations.
4 Numeracy through Scientific Notation
- NumBERT replaces every pre-training number with scientific notation, representing 314.1 as 3141[EXP]2.The representation combines a mantissa and exponent, with [EXP] added as a vocabulary token.Source: §4 · p. 3
- The design aims to help BERT associate objects with number magnitudes through the exponent while ignoring the relatively insignificant mantissa.Source: §4 · p. 3
- NumBERT converged to a similar loss on the original BERT Masked LM+NSP pre-training task and a standard NLP task suite as BERT-base.The authors report this as evidence that NumBERT was not over-specialized for numerical reasoning tasks.Source: §4 · p. 3
5 Evaluation
The evaluation compares scalar-probing models with an aggregate baseline using accuracy, MSE, and Earth Mover’s Distance (EMD). It also tests transfer to relative comparisons and product prices.
5 Evaluation
- Models are evaluated against an aggregate baseline using three metrics: accuracy, MSE, and EMD.The baseline predicts each test object with the empirical distribution over buckets across training objects for that attribute.Source: §5 · p. 3
- Accuracy compares the predicted bucket with the ground-truth mode of the empirical distribution.For mcc, the predicted bucket is the highest-scoring bucket; for rgr, it is the bucket containing the predicted scalar.Source: §5 · p. 4
- MSE compares probability distributions without accounting for how far apart their buckets are, whereas EMD penalizes moves between distant buckets more.MSE is upper-bounded by 1; EMD measures the work needed to move probability mass between distributions.Source: §5 · p. 4
- Transfer tests use human-labeled relative comparisons and product-price data, with DoQ-trained price models retrained using 12 power-of-4 buckets.Relative comparisons include judgments such as whether one object is heavier than another.Source: §5 · p. 4
6 Results
Scalar information is present in pretrained representations, but extracting it robustly remains difficult. Performance varies with encoder, training objective, numeric representation, domain context, and evaluation metric.
6 Results
- MCC over the best encoders captures about half the accuracy distance and a third of the MSE and EMD distance to the upper bound, leaving substantial room for robust commonsense reasoning.These relative encoder, metric, and probe patterns hold across mass, length, and price despite differences in their raw scores.Source: §6 · pp. 4–5
- MCC consistently beats regression, which frequently fails to improve on the Aggregate baseline; extracting scale information requires the full data distribution, not just the median.The full empirical distribution is used as soft labels in MCC.Source: §6 · pp. 4–5
- Contextual encoders outperform Word2Vec on this non-contextual task, and ELMo exceeds BERT on scalar probing despite being weaker on downstream NLP tasks.The authors associate ELMo’s scalar-probing advantage with better numeracy due to character-level tokenization.Source: §6 · p. 5
- NumBERT performs better than ELMo and BERT on EMD but worse on MSE and accuracy, making its apparent effect sensitive to the evaluation metric.The authors identify EMD as the most robust metric a priori while emphasizing the need to examine the full range of metrics.Source: §6 · p. 5
- Training only on objects from a narrow domain significantly improves scalar prediction on Animal Masses.Restricting “crane” to its animal-domain meaning simplifies the mass distribution the model must predict.Source: §6 · p. 5
- On VerbPhysics, rgr+NumBERT approaches the DoQ oracle but trails specialized models; on consumer prices, mcc+NumBERT does best on both distributional metrics.NumBERT is the best encoder across configurations on both zero-shot transfer tasks.Source: §6 · p. 5, Table 2 · p. 5, Table 3 · p. 5
7 Conclusion
Scalar probing finds scale information in object embeddings, but the signal remains insufficient for common-sense scale understanding. The analysis points to context and numeracy as directions for progress.
7 Conclusion
- Object embeddings contain a significant amount of scale information, but a sizable gap remains before language models meet this prerequisite for common-sense reasoning.The authors conclude that the observed signal is still weak for common-sense scale understanding.Source: §7 · p. 5
- Modeling context and numeracy are identified as directions for progress, with scale information transferred from numbers to nouns.The authors propose extending prediction to sentences whose context bears on scalar magnitudes.Source: §7 · p. 5
A Model Hyperparameters
- The appendix provides model hyperparameters for reproducibility.Source: A Model Hyperparameters · p. 6
A.1 Probing Layer of the Scalar Probing Model
The probing layer uses ridge regression for regression and a regularized linear classifier for MCC; narrow-domain experiments reduce embeddings with PCA before training.
A.1 Probing Layer of the Scalar Probing Model
- Regression (rgr) uses ridge regression with regularization strength 1.Source: §A.1 · p. 7
- MCC uses a linear classifier with softmax activation and regularization strength 0.01.Source: §A.1 · p. 7
- For narrow-domain experiments with smaller datasets, PCA reduces embeddings to 150 dimensions before probing-model training.Source: §A.1 · p. 7
A.2 NumBERT
NumBERT uses BERT-Base settings and the original BERT pretraining corpora, with a 15% masking rate and 1,000,000 training steps. It reaches performance similar to a matched BERT-base reimplementation on standard NLP benchmarks.
A.2 NumBERT
- NumBERT uses the Wikipedia and Books corpora and the BERT-Base configuration: 12 layers, hidden size 768, 12 attention heads, and 110M parameters.Source: §A.2 · p. 7
- Pretraining applies 15% masking after WordPiece tokenization and runs for 1,000,000 steps, about 40 epochs over a 3.3-billion-word corpus.Source: §A.2 · p. 7
- NumBERT and a reimplementation of BERT-base with identical settings reach similar performance on a suite of standard NLP benchmarks.Source: §A.2 · p. 7
B Data Statistics
The paper uses three datasets or resources, including DoQ subsets filtered by dimensions or object types and a cleaned F&C dataset with established splits.
B Data Statistics
- The paper uses three datasets or resources, and DoQ subsets are formed by filtering the original resource by dimensions or object types.Source: B Data Statistics · p. 7
- DoQ includes only objects with more than 100 values in the resource.Source: B Data Statistics · p. 7
- The F&C Cleaned dataset uses data and train/dev/test splits from Elazar et al. (2019).Source: B Data Statistics · p. 7
C Complete Experiment Results
The appendix describes how scalar-value distributions are bucketed and how DoQ scales are examined for multiple modes. Table 6 presents evaluations across datasets, including multimodal experiments.
C Complete Experiment Results
- Scalar attributes are modeled as 12-category distributions by taking log10 values, rounding to integers from -2 to 9, and normalizing bucket counts.Each integer corresponds to one bucket.Source: C Complete Experiment Results · p. 7
- DoQ data for each scale are divided into unimodal and multimodal sets using a hill-climbing algorithm to identify peaks.The distinction reflects cases where words have multiple meanings or objects vary substantially in their possible forms.Source: C Complete Experiment Results · p. 7
- A separate multimodality estimate smooths bucketed DoQ distributions into a density function and counts its local maxima.More than one maximum is classified as multimodal.Source: C Complete Experiment Results · p. 7
- Table 6 contains evaluations on all datasets, including the multimodal experiments.The passage identifies the table's scope but provides no cell values.Source: C Complete Experiment Results · p. 7, Table 6 · p. 8