Source-linked AI summary
Augustinian BabyLM: What Ostensive Definition Can and Cannot Teach a Small Language Model
Lisa Bylinina
TL;DR
This paper asks what visual grounding can teach a small masked language model beyond corpus-based learning. It tests vision-derived initialization and finds persistent, word-specific gains for object-property knowledge, while most benchmarks miss effects on abstract and function words.
Problem
The study examines whether ostensive visual grounding leaves measurable knowledge in a small language model and whether existing benchmarks detect that knowledge.
Method
The paper initializes selected token embeddings from image-region features, trains a masked language model on 10M words, and evaluates corpus-tailored visual-property and other benchmarks.
Results
Visual initialization persists through training; VP-Swap shows replicated, seeded-word-specific advantages, and synthetic grounding transfers the benefit to newly grounded words.
Takeaways & Limitations
Visual grounding measurably shapes specific lexical knowledge, but its effects depend on the vocabulary and evaluation used.
Takeaways & Limitations
Most configurations are single runs, the corpus is not the official 2026 strict-small distribution, and all results use one masked-language-model architecture.
Abstract
from arXiv · showhide
A language model normally begins training with random word embeddings: whatever 'banana' means must be learned from training corpora. I implement St. Augustine's picture of word learning, meaning by ostension, for a small masked language model (DeBERTa) trained on 10M words: before training, visually grounded tokens receive embeddings derived from the image regions they label; other tokens start random. Visual initialization leaves a measurable imprint that lasts until the end of training. At the same time, the effect remains invisible under most BabyLM benchmarks, which probe abstract grammatical knowledge: visual initialization does not affect performance there. The only zero-shot exception is object-property knowledge (COMPS, Misra et al. 2023), where seeding helps in every configuration. To follow up on this result, I build a corpus-tailored version of the Visual-Property Swap benchmark (Lin et al., 2026), which tests color, material, size, and shape knowledge, with per-item training frequency and seeded status. Here, vision-seeded models have a persistent, seed- replicated advantage, confined to the seeded words. As a causal test, I show that synthetic grounding of previously unseeded words transfers the advantage to exactly those words. Function words and abstract vocabulary also receive strong visual seeds and retain them throughout training, and the training objective draws on them: held-out mask-prediction loss falls for these words in every seed. However, no benchmark I run registers this. What evaluation would pick this up remains an open question.
1 Introduction
The paper tests Augustine’s ostensive account by initializing selected language-model embeddings from visual regions before text-only training. Visual seeding leaves lasting representational and objective-level effects, improves object-property knowledge, but remains largely invisible to standard benchmarks.
- Visual initialization leaves a measurable embedding imprint through the end of training and makes the training objective easier for grounded words.The intervention supplies visual information before text-only learning, rather than continuing multimodal training.
- Object-property knowledge is the sole standard BabyLM zero-shot area that improves consistently with visual seeding.COMPS improves in all configurations, whereas most benchmarks probe aspects of abstract grammatical knowledge.
- A corpus-tailored VP-Swap benchmark reveals a persistent, seed-replicated advantage restricted to grounded words and transferable through synthetic grounding.The benchmark targets visual properties and tracks each item’s training frequency and seeded status.
- Function words and abstract vocabulary receive strong, retained visual seeds that matter to the objective, yet existing benchmarks do not detect their effects.This scope result motivates evaluations designed around the specific vocabulary affected by grounding.
2 Related Work
The paper situates its intervention among multimodal BabyLM work, grounded word representations, and targeted evaluations of lexical meaning. It argues that visual grounding should be assessed with probes of denotation and referent properties rather than mainly abstract structural benchmarks.
- The approach supplies visual information once through initial embedding values, avoiding continued caption training, architectural changes, and changes to the language-model objective.This contrasts with multimodal training regimes that can harm language-only performance through catastrophic forgetting.
- Frozen vision features initialize selected word embeddings, unlike methods that inject grounding through auxiliary objectives or cross-modal attention.The visual encoders are used only to determine starting embeddings.
- Visual grounding is expected to affect lexical-semantic knowledge, while dominant small-model benchmarks mostly probe abstract structural competence.The relevant targets include what words denote and what their referents are like.
3 The Augustinian Setup
The Augustinian setup replaces random initialization for visually grounded token embeddings with projected region features, then trains a DeBERTa masked language model on text only. Seeding reaches a minority of word types but most token occurrences, with coverage skewed toward concrete words.
- 3.1 Region Features and Seeding: Visual seeds are formed by averaging frozen-encoder features from every image region labeled by a target word and writing the result into its initial embedding row.Words without image-region support retain random initialization; subword tokens share the word’s visual embedding.
- 3.2 Coverage: What Gets Seeded: About 10% of ∼163k corpus word types are seedable, but they cover 85% of text.Subword tokenization raises seedable vocabulary coverage to 37.7% at 50k merges, 29.1% at 75k, and 23.8% at 100k, while token coverage stays near 87%.
- 3.2 Coverage: What Gets Seeded: Seedability increases with concreteness, with seededness correlating with concreteness norms at r = 0.46.Figure 2 bins word types by Brysbaert concreteness and shows the fraction appearing in visual grounding data.
- The experiments train DeBERTa-v3-base masked language models on the ∼9.9M-word bb24.train corpus across three vocabulary sizes and three image encoders.Each model is trained for 10 epochs alongside a same-vocabulary random-initialized baseline.
4 Official BabyLM Evaluation
Visual initialization produces a selective, configuration-robust gain in object-property knowledge, while most official BabyLM tasks remain near zero or inconsistent. COMPS is the only zero-shot task with a stable positive effect across configurations and seeds, although GLUE is also positive across configurations but remains difficult to attribute.
- Official task results: +1.30 COMPS accuracy points were positive in all 9 configurations, while most other tasks were near zero or inconsistent.GLUE also reached +1.08 in all 9 configurations; BLiMP and EWoK split roughly evenly, while entity-tracking and supplement tasks were slightly negative.
- Interpretation: GLUE’s positive result is harder to attribute because it is evaluated after fine-tuning and the paper does not investigate how initialization is exploited there.This limits interpretation of the GLUE effect compared with the zero-shot COMPS result.
- Robustness: The three vision encoders behaved alike, with COMPS and GLUE positive in all nine encoder-by-vocabulary configurations.The remaining analyses therefore used SAM as a representative encoder.
- Stability: +1.46/+1.05/+0.77 COMPS deltas remained positive across three random seeds, unlike the other zero-shot tasks.The replicated headline pair was 75k-SAM versus 75k, with other zero-shot task signs changing across seeds.
5 VP-Swap: A Visual-Property Probe
The corpus-tailored VP-Swap benchmark makes visual-initialization effects visible, persistent, and word-specific. Seeded models lead throughout training, with gains concentrated on seeded nouns rather than unseeded placebo items.
- Benchmark construction: VP-Swap tests color, material, relative size, and shape knowledge using minimal pairs annotated with corpus frequency and seeded status.The benchmark contains 7,416 items across four syntactic frames and uses masked-language-model pseudo-log-likelihood scoring.
- Results: +2.2/+2.7/+2.9 VP-Swap points were the final deltas across three seeds, with the seeded model leading from 1M words onward.All three final differences were significant by McNemar tests, with z = 4.37/5.44/5.79.
- Word specificity: +0.034/+0.031/+0.030 were the seeded-noun deltas across random seeds, while neither-seeded placebo pairs were near zero at +0.012.The frequency-binned difference-in-differences was positive in six of seven sufficiently populated bins, reducing concern that frequency explains the effect.
- Robustness and scope: The replicated gain did not harm unseeded words and held across attributive, existential, and relative frames, but was near zero in the short copular frame.The largest property-specific effect was for color at +0.035, while rare words remained near chance.
6 Synthetic Grounding
Synthetic grounding tests whether the VP-Swap advantage follows a new grounding intervention rather than merely correlating with naturally seeded words. The advantage transfers to synthetically seeded words while leaving already seeded controls essentially unchanged.
- Method: Synthetic grounding added 737 seeded tokens by grounding 1,155 previously unseeded words from generated scenes, detected objects, and pooled SAM features.The procedure began with 1,986 concrete words lacking image support, dropping words the detector could not find.
- Results: +0.012/+0.020/+0.010 VP-Swap deltas appeared for synthetically grounded words across three seeds, with mean +0.014.These words had shown mean −0.002 under the original SAM initialization, supporting transfer after the intervention.
- Robustness: COMPS remained positive under all three random seeds at +1.57/+1.16/+0.40 after the synthetic extension.This indicates that the extension did not disturb the original object-property effect.
- Interpretation: The synthetic benefit was roughly half the real-grounding benefit, but its direction was unambiguous and its scope matched the synthetically grounded words.Generated images and automatic detection were noisier than human region annotations.
7 Mechanism: A Prior with a Lasting Imprint
Visual initialization reshapes the relational geometry of seeded embeddings even after individual vectors drift far from their visual starting points.
- 7 Mechanism: A Prior with a Lasting Imprint: Embedding drift does not explain the VP-Swap advantage: per-word drift is essentially uncorrelated with advantage across 816 seeded words.The reported correlation is r = −0.017 for words with at least three items.
- 7 Mechanism: A Prior with a Lasting Imprint: At 100M words, seeded embeddings retain visual relational structure despite individual embeddings becoming nearly orthogonal to their initial vectors.Cosine similarity to initialization falls from 1.00 to about 0.15, while seeded-word RSA remains 0.31 versus 0.10 for the baseline.
- 7 Mechanism: A Prior with a Lasting Imprint: The visual prior therefore preserves how seeded words are arranged relative to one another, not their original absolute positions.
8 The Limits of Ostension
Ostensive initialization affects abstract and function-word representations and helps the training objective, but existing benchmarks largely fail to detect those effects.
- 8 The Limits of Ostension: Function-word seeds are as strong as concrete-word seeds and retain visual structure after training, with function-word RSA 0.45 versus approximately 0 for baseline.Their absolute similarity to visual anchors is also 0.44 versus 0.34 for concrete controls.
- 8 The Limits of Ostension: Benchmark performance does not reflect this retained abstract-word information: BLiMP deltas average −0.8 points for abstract-word contrasts versus +0.8 for concrete-word contrasts.The concreteness correlation is r = +0.20 over 54 fast-subset phenomena.
- 8 The Limits of Ostension: Held-out masked-LM loss improves for vision-initialized function words in all three seeds, with a mean reduction of 0.080 nats.Never-seeded tokens remain at approximately zero change under identical masking.
- 8 The Limits of Ostension: The only consistent abstract-domain improvement is EWoK number, at +5.8, although the author treats this result as speculative because samples are small.
- 8 The Limits of Ostension: Overall, existing evaluations mostly test abstract structural competence and remain blind to visual effects that the training objective uses.
9 Discussion
The discussion argues that targeted, vocabulary-specific evaluations reveal effects that aggregate BabyLM scores miss and suggests mechanism-informed directions for future interventions.
- 9 Discussion: The paper frames visual grounding as a case of intervention–evaluation scope misalignment, since its effect is real but barely visible in aggregate evaluation.
- 9 Discussion: Targeted, metadata-rich probes expose intervention effects that aggregate BabyLM evaluation largely misses by tracking training frequency and seeding status.
- 9 Discussion: Because visual information survives as relational structure rather than retrievable anchors, richer initialization is more promising than periodically reinjecting visual embeddings.
- 9 Discussion: Figure 7 compares held-out mask-prediction-loss changes across token classes and shows seeded classes becoming easier to predict while the never-seeded placebo remains near zero.
- 9 Discussion: The paper identifies evaluation designed around visual-prior contributions as a next step, especially for strong retained function-word seeds that current benchmarks do not reward.
Limitations
The study’s evidence is constrained by limited replication, a nonstandard corpus, generated benchmark items, restricted analyses, and one masked-LM architecture.
- Limitations: Most configurations are single runs, the corpus is not the official 2026 strict-small distribution, and VP-Swap inherits properties generated and filtered by LLMs.Three model types receive three seeds each, but broader direct comparison and generalization are limited.
- Limitations: Several analyses use restricted subsets, including approximately 11k of 21k seeded tokens, a 64-word function-word probe, and the BLiMP fast subset.All results come from a single masked-language-model architecture.
Ethics Statement
The study reports low ethical risk because its models are research artifacts rather than deployment systems, while its generated benchmark and synthetic-grounding pipelines introduce documented bias and error risks.
- The models are trained from scratch on a public corpus and are not intended for deployment, limiting the work’s stated ethical risk.
- The VP-Swap benchmark may reflect cultural or distributional biases from its LLM generator and judge, so it is a diagnostic probe rather than ground-truth world knowledge.
- The synthetic-grounding pipeline can hallucinate or mis-detect objects and inherit biases from its text-to-image and open-vocabulary detection components.
- Because these pipelines affect initialization and minimal-pair evaluation rather than user-facing output, the authors judge their risks contained while flagging them for downstream users.
H The Entity-Tracking Scoring Artifact
Entity-tracking gains from vision initialization were caused by a scoring artifact rather than learned competence. An untrained-model diagnostic exposed inflated baselines, inverted difficulty, and delayed collapse.
- A +17-point vision-init spike near step 1000 was a scoring artifact, so entity tracking is excluded from interpretation.
- Untrained models scored about 42%, because pseudo-log-likelihood interacted with answer-option length and frequency priors.
- Vision initialization delayed baseline collapse by about 1000 steps rather than building entity-tracking competence.
- Inverted subtask difficulty persisted across checkpoints, with 0-operation items below 5-operation items despite the opposite true ordering.
- The spike was specific to the 50k vocabulary, while corresponding peaks were only +1.0 at 75k and +1.7 at 100k.
- An untrained-model check at step 0 is a cheap diagnostic; VP-Swap instead begins near chance at 0.49–0.51.