PaperlayerView exact revision on arXiv

Source-linked AI summary

Scaling cross-situational word learning on children's naturalistic experiences

Michael C. Frank, Alvin W. M. Tan, Khai Loong Aw, Robert Z. Sparks, Victor Moreno, Grace Keene-Lee, Meesha R. Ryan, Tarun Sepuri, Jane Yang, Virginia A. Marchman, Daniel L. K. Yamins, Bria Long

arXiv:2610.31842v1cs.CL

TL;DR

The paper asks whether cross-situational learning scales to naturalistic child video and how much it depends on visual representations. It tests this with models trained on BabyView, comparing visual encoders, and finds mappings but substantially better performance with internet-trained representations.

  • Problem

    The study asks whether cross-situational word learning scales to naturalistic data and how dependent its performance is on visual representation quality.

    Source: Discussion · p. 8

  • Method

    The authors trained contrastive vision–language models on BabyView with learned word mappings and compared frozen visual encoders pretrained on internet images or BabyView frames.

    Source: Word mapping performance depends on… · p. 4

  • Results

    Cross-situational models learned mappings from egocentric video, but performance was much higher with internet-derived than developmental-data-derived visual representations.

    Source: Discussion · p. 8

  • Takeaways & Limitations

    Cross-situational word learning is a viable developmental mechanism, while developmentally appropriate models still require progress in visual representation learning and referential alignment.

    Source: Discussion · p. 9

    The models are not developmentally realistic, and estimates’ generalizability is limited by the composition of the BabyView training data.

    Source: Discussion · p. 9

Abstract

from arXiv · show

How do children learn the meanings of the simple, concrete words that form a majority of their early vocabulary? Statistical information about many word meanings can be recovered from the gradual accumulation of information about the co-occurrence of words with their referents, a proposal known as ``cross-situational learning''. Contrastive vision--language models allow this learning strategy to be tested on egocentric video data capturing children's early real-life experiences with visual and linguistic inputs. Initial successes of this method on small datasets provide a proof-of-concept, but vision models struggle to learn robust visual representations when trained on these same egocentric videos. How dependent is cross-situational word learning performance on the quality of the model's visual representations? And can cross-situational models scale to larger and more diverse datasets? To address these questions, we trained models on the BabyView corpus, including 2,441 hours of longitudinal egocentric video from 48 children. Models trained on this dataset learned object--label mappings, but performance was far higher when they used internet-pretrained visual representations as opposed to when visual representations were learned solely from the developmental video data. Further, the majority of the learning signal was carried by a small proportion of utterances that described a visible referent. Models learned lexical representations that showed recognizable human-like semantic structure, but model performance showed substantially different trajectories over training experiences than those observed in children. Cross-situational learning is a viable word learning strategy for early noun vocabulary, but constructing developmentally plausible models will require understanding how to learn high-quality visual representations from naturalistic data.

Results

Across the experiments, word-mapping performance depended strongly on visual representations and referential alignment. Learned noun representations showed some human-like semantic structure, while model scaling differed from children’s.

Word mapping performance depends on visual encoding

  • 81.6% with the largest off-the-shelf encoder, versus 25% chance; off-the-shelf encoders also outperformed BabyView-trained encoders in-domain, and data quantity mattered more than child-sample diversity.All encoders nearly perfectly categorized the test stimuli, but supported word mapping at different levels on children’s complex, noisy input.Source: Word mapping performance depends on… · p. 5

Referential alignment is a key bottleneck

  • 88.8% accuracy came from training OTS-304M on referentially aligned data, outperforming the full corpus; removing aligned data sharply reduced accuracy, while removing matched unaligned data modestly improved it.The same aligned-data advantage held for the lower-performing BabyView-trained encoders.Source: Referential alignment is a key… · p. 5

Learned representations align with human semantic structure

  • OTS-304M clusters exceeded permuted chance for 9/10 MB-CDI categories; visual relatedness correlated with human judgments (ρmax = .31), below word2vec (ρmax = .45), yet persisted after controlling for linguistic similarity.Figure 3 assesses noun representations through category cohesion and correlations with human relatedness judgments.Source: Learned representations align with… · pp. 6–7, Figure 3 · p. 6

Comparing model scaling with human learning trajectories

  • Children recognized few words at one year despite more training hours than models, then knowledge grew rapidly across year two, with scaling initially delayed but ultimately far steeper than models.Top performance on this challenging benchmark was only 48%, reflecting in part that many test items did not occur in the training data. The curves are data-scaling functions, not literal developmental curves.Source: Comparing model scaling with human… · pp. 7–8

Discussion

Cross-situational learning can support concrete-noun mapping, but performance depends on strong visual representations and does not reproduce children's learning trajectories. The study identifies visual representation learning and referential alignment as important challenges for more developmentally plausible models.

Discussion

  • Contrastive models learned mappings from naturalistic data, but performance was much higher with strong visual representations than with representations learned from developmental data.The authors conclude that cross-situational learning is viable for early noun learning only when it uses strong visual representations.Source: Discussion · p. 8
  • Visual representations help handle noisy infant environments and support generalizing learned labels to new object exemplars.The authors report that off-the-shelf representations were also more useful for in-domain evaluations based on egocentric data.Source: Discussion · p. 8
  • Referential alignment is an important learning signal, and future models could learn on the fly which utterances are most informative.The authors connect alignment to prior findings on model performance and children's input and sensitivity to social cues.Source: Discussion · p. 8
  • The models' scaling trajectories differed from children's: models learned mappings from little data, while children's learning accelerates rapidly in the second year.The authors state that grounding alone did not produce children's acceleration and offer developmental changes as possible explanations.Source: Discussion · p. 8
  • The study focuses on early noun learning and uses models that are not developmentally realistic; its data also underrepresent varied backgrounds and public settings.BabyView was sampled to maximize data volume, and many children came from high-income households; most recordings were made inside homes.Source: Discussion · p. 9
  • The authors argue that studying visual representation learning separately from word mapping cannot provide a complete picture of early lexical development.They identify developmentally appropriate visual learning and the use of referential alignment as next steps.Source: Discussion · p. 9

Materials and Methods

The study filtered BabyView recordings for English exposure, paired utterances with nearby video frames, and trained models with frozen image encoders and learned word representations.

Data

  • The English-language training set comprised 2,441 hours from 48 children after excluding videos with more than 50% non-English data.The source release contained 2,701 hours from 51 children; all utterances, including children's own speech, were retained because diarization was noisy.Source: Data · p. 9
  • Utterances were matched to the video frame nearest their temporal midpoint, and Gemini ratings of at least 50 on a 0–100 alignment scale counted as aligned.Video was downsampled to one frame per second, so a frame could be matched to multiple utterances.Source: Data · p. 9

Neural network architecture

  • Models paired a frozen image encoder with trainable word representations and an alignment layer; frame embeddings used a 4×4 grid of pooled patch features.Multiple instance learning selected the maximally aligned region for each training pass.Source: Neural network architecture · p. 9
  • Models used InfoNCE training for 20 epochs with AdamW, and the epoch selected on 117 held-out categories was evaluated on 60 main categories.Training used a learning rate of 3e-4, batch size 256, and weight decay 0.1.Source: Neural network architecture · p. 10

Language annotation

The researchers classified sampled audio windows to filter BabyView for English-language recordings, then instructed the classifier to report speech, languages, and child-directed speech.

Language annotation

  • Three 60-second windows per video were classified from audio to filter recordings that were primarily or exclusively in English.The automatic measures correlated with survey-reported English exposure at r = .91.Source: Language annotation · p. 16
  • The prompt asked the classifier to identify languages spoken in home recordings, judging from audio and ignoring wordless singing, babbling, and background media unless it was the only speech..Source: Language annotation · p. 16
  • The output schema requested whether intelligible speech occurred, language proportions summing to 1, and whether most speech was to or with a child.Language proportions were specified using ISO 639-1 codes.Source: Language annotation · p. 16

Reference annotation

The researchers used Gemini to rate whether utterances referred to visible objects, then checked a stratified sample of ratings against human judgments.

Reference annotation

  • Gemini annotated each utterance–frame pair with an alignment score and a noun for the single visible object referred to.The prompt defined alignment as a 0–100 rating of how strongly an utterance referred to a concrete object visible in the frame.Source: Reference annotation · p. 16
  • Two researchers rated a stratified sample of 1,500 frames, with half drawn from frames Gemini rated unaligned.Human raters made binary judgments about whether the referent was present and received additional instructions about proper nouns, pronouns, and onomatopoeia.Source: Reference annotation · p. 16
  • Between-rater κ was .75, representing 88% agreement, while average rater-to-Gemini κ was .64.Gemini ratings of at least 50 counted as detections for the reported precision and recall.Source: Reference annotation · p. 16

DINOv3 ViT encoders

The study trained DINOv3 vision transformers on BabyView frames and assessed encoder category separation with a nearest-prototype probe.

Training

  • The encoders were pretrained from scratch on BabyView using DINOv3 self-distillation.Source: Training · p. 17
  • ViT-S/16 and ViT-B/16 were trained for 200,000 iterations at a global batch size of 512, while ViT-L/16 used a global batch size of 384.Training used all 9,726,507 frames extracted at 1 fps from the full BabyView 2026.1 release, without filtering by language content.Source: Training · p. 17

Encoder probes

  • A head-free nearest-prototype probe tested whether each target image was more similar to its category prototype than to three foil-category prototypes.The probe used mean cached region vectors and was run on both the Konkle test set and the held-out BabyView set.Source: Encoder probes · p. 17

Language data processing and model training

The training pipeline normalized and filtered utterance text, represented retained tokens with learned embeddings, and optimized image–text alignment with a symmetric contrastive objective.

  • Utterances were lowercased, stripped of non-alphabetic characters, and truncated to 16 in-vocabulary tokens; vocabulary included word types occurring more than five times.Vocabulary was built separately for each training run; the full corpus contained 15,608 word types.Source: Language data processing and model… · p. 17
  • The model averaged learned 512-dimensional token embeddings into normalized utterance vectors and optimized a symmetric InfoNCE loss over batches of 256 image–text pairs.Similarity used the maximum cosine match between the utterance and projected frame regions, scaled by a learned temperature.Source: Language data processing and model… · p. 17

Evaluation details

Evaluation used the Konkle category benchmark, an in-domain BabyView benchmark, and LEVANTE-bench, alongside fitted scaling curves.

Konkle evaluation

  • The Konkle evaluation used 60 categories, with a separate 117-category development split for selecting the training epoch.Each 4AFC trial sampled one target and three distractor images; results averaged 100 trials per category, and out-of-vocabulary categories were scored at chance.Source: Konkle evaluation · pp. 17–18

In-domain evaluation

  • The in-domain benchmark contained 603 frames from 356 videos across 82 categories, drawn from videos excluded by the audio-language filter.It used Gemini referent annotations and excluded person terms, body parts, and generic containers; categories required at least four exemplars.Source: In-domain evaluation · p. 18

LEVANTE-bench evaluation

  • LEVANTE-bench used a 159-item 4AFC vocabulary test with a fixed photo per item and distractor, and child scores were reconstructed from ability estimates.The final child sample included 145 English-speaking children, and 142 items appeared in BabyView transcripts.Source: LEVANTE-bench evaluation · p. 18

Sigmoid curve fitting

  • Scaling curves used a three-parameter logistic over log10 training examples, with the floor fixed at 25% and upper asymptote, midpoint, and slope fitted.Fits used standard-deviation-weighted least squares; empirical 95% confidence bands came from 4,000 Monte Carlo draws.Source: Sigmoid curve fitting · p. 18

Developmental data

Wordbank reports were converted to 4AFC estimates on words matched to the Konkle categories, and frame–utterance pairs were converted to approximate years of input.

Conversion of Wordbank data to Konkle scores

  • Wordbank CDI reports matched 40 of 60 Konkle words on the Words & Gestures form and 46 on the Words & Sentences form.Children and models were compared on the same 40 Words & Gestures-matched words; reported knowledge proportions became 4AFC scores by assuming unknown words were guessed.Source: Conversion of Wordbank data to Konkle… · pp. 18–19

Developmental time conversion

  • Developmental time conversion assumed 4,000 waking hours per year and 697 utterances per hour.The waking-time estimate is approximate; the per-child median speech rate was 702 utterances per hour (95% CI 637–747), and recording may favor more talkative periods.Source: Developmental time conversion · p. 19

Analysis of learned lexicons

The analysis examines learned lexical structure through noun neighborhoods, word co-occurrence, human relatedness judgments, vocabulary assessment, and comparisons with children's item difficulty.

Analysis of learned lexicons

  • Nouns were included in neighborhood analyses only when frequency was at least 50 and their nearest-neighbor structure exceeded a random-vector threshold.The threshold required the top three neighbors’ cosine distance to exceed four times the distance between random vectors; tSNE used perplexity 30.Source: Analysis of learned lexicons · p. 19
  • Human relatedness was assessed for noun pairs and across all words, while the benchmark combined five similarity datasets and z-scored scores within each set.Evaluation was restricted to words present in the training data at each scale.Source: Analysis of learned lexicons · p. 19, Figure S14 · p. 28
  • A word2vec skip-gram baseline estimated similarity structure from word co-occurrence in the same utterances used for contrastive VLM training.Separate baselines were trained at each dataset size, and evaluation used cosine similarity between L2-normalized vectors.Source: Analysis of learned lexicons · p. 19
  • The analyses also compare model accuracy with children's comprehension and production difficulty, and assess vocabulary performance on LEVANTE-bench.The child-difficulty comparison uses matched Konkle items; the LEVANTE-bench figure distinguishes all items from items in each model vocabulary.Source: Figure S15 · p. 29, Figure S16 · p. 29
  • A separate comparison tests whether removing key-child utterances changes OTS-304M accuracy relative to a matched random draw.Source: Figure S17 · p. 30

Related papers

Loading 2610.31842v1…