PaperlayerView exact revision on arXiv

Source-linked AI summary

(V)LMs generalize beyond surface co-occurrence: Evidence from cross-modal number agreement

Zach Studdiford, Kanishka Misra

arXiv:2609.00443v1cs.CLcs.AI

TL;DR

The paper asks whether VLMs can learn abstract grammatical number beyond surface co-occurrence, a question text-only tests cannot settle because linguistic cues can reveal number directly. It teaches novel nouns through visual or linguistic cues while updating only their embeddings, then examines behavior, representations, and causal mechanisms. The results provide non-trivial evidence for cross-modal generalization and indicate that both cue types are integrated similarly into existing number-agreement mechanisms.

  • Problem

    Text-only tests cannot determine whether models encode abstract number because orthography and distributional cues can directly reveal the feature.

  • Method

    The study teaches VLMs novel noun pairs using visual or linguistic number cues, updates only their new embeddings, and evaluates behavioral, representational, and causal evidence.

  • Results

    VLMs show non-trivial number-agreement generalization from both visual and linguistic cues, with similar treatment in their internal causal mechanisms.

  • Takeaways & Limitations

    Statistical learners like VLMs can show abstraction-compatible behavior beyond surface-level co-occurrence, including for linguistic information acquired from visual evidence.

  • Takeaways & Limitations

    The study examines only two lower-parameter VLMs from one family, limited images, one language, and a single novel-word-learning method.

Abstract

from arXiv · show

Language models learn about grammatical number primarily from co-occurrence, and show frequency effects as a result---sometimes taken to indicate that they do not learn abstract ``rules'', and are instead dependent on specific lexical items. Testing generalization with text stimuli alone cannot settle this debate, since distributional cues (is/are, this/these) easily give number away. We instead use cross-modal generalization as a tool to investigate abstractions in LMs that can also accept visual inputs (VLMs), restricting the evidence that diagnoses number to an extra-linguistic modality. We teach VLMs pairs of new nouns by adding new embeddings and only updating them during learning, comparing conditions where number is diagnosed by visual cues alone against ones where it is disambiguated by text. Across behavior, representational dynamics, and causal mechanisms, we find non-trivial evidence for cross-modal generalization across both exposure conditions, and that linguistic vs. extra-linguistic cue conditions are treated in similar ways in the internal mechanisms of the model. This suggests that statistical learners like VLMs can generalize beyond surface-level co-occurrence and show genuine abstraction-compatible behavior.

1 Introduction

The paper tests whether VLMs can abstract grammatical number beyond linguistic co-occurrence by learning novel nouns whose number is diagnosed visually or textually. Across behavior, representations, and causal mechanisms, the results support cross-modal, abstraction-compatible generalization.

  • Motivation: Text-only number-agreement tests cannot isolate abstraction because orthography and distributional cues can directly reveal number.Examples include noun suffixes and accompanying forms such as is/are and this/these.
  • Approach: The study teaches VLMs novel noun pairs with number information restricted to visual cues or made explicit through language, then tests agreement on disjoint stimuli.New embeddings are added and only those embeddings are updated during learning, preventing surface-form shortcuts such as using -s.
  • Findings: Models achieve above-chance number-agreement performance for novel nouns learned from both visual and linguistic cues, with degradation only when intervening attractors reach ≥2.This provides behavioral evidence for non-trivial cross-modal generalization, including when textual co-occurrence does not diagnose number.
  • Findings: During learning, novel noun embeddings move toward representational regions occupied by known nouns sharing their number feature in both cue conditions.Singular and plural novel nouns move toward corresponding number-organized regions, with movement on average greater for language-cued nouns.
  • Findings: Pre-existing causal mechanisms for number agreement accommodate newly learned nouns, with no effectiveness difference between language- and vision-acquired nouns.Four interpretability methods show causal efficacy substantially above chance, suggesting similar integration into the existing mechanism.
  • Implication: The findings suggest that item-specific frequency effects do not rule out abstraction-compatible representations in statistical learners such as VLMs.The representations can flexibly accommodate linguistic information acquired primarily through extra-linguistic evidence.

2 Related Work

Number agreement is an established probe of syntactic competence and generalization in neural language models. This work extends novel-word learning toward cross-modal evidence and examines how newly acquired knowledge is integrated internally.

  • Established probe: Number agreement has long been evaluated in neural language models through minimal-pair judgments and representational analyses, including studies of frequency effects.The probe targets agreement behavior under syntactic and representational evaluation paradigms.
  • Novel-word learning: Novel-word learning studies expose models to new words using pictures or sentences and test their use in different contexts.The method traces to earlier connectionist approaches and has been used for noun, verb, and category learning.
  • Cross-modal extension: The study combines visual learning with analyses of novel embeddings and interpretability mechanisms discovered before the novel words were acquired.This connects behavioral cross-modal generalization to representational dynamics and causal integration.

3 General Methods

The methods add and train novel noun embeddings in VLMs, expose their number through controlled cues, and evaluate generalization with agreement-attraction minimal pairs. The experiments use two Qwen3-VL models and balanced, disjoint stimulus sets.

  • Novel embeddings: The training paradigm adds novel embeddings, freezes the pretrained model, and updates only the new noun representations.Embeddings are initialized from balanced singular and plural noun distributions, avoiding an initial number bias.
  • Cue conditions: Vision cues pair images of one or multiple chimeric creatures with novel singular or plural nouns, while the comparison condition supplies linguistic number cues.The visual stimuli are manually verified to reduce the chance that models have previously encountered them.
  • Evaluation stimuli: Evaluation uses minimal-pair agreement-attraction sentences containing zero to three intervening nouns with the opposite number feature.Attractors are designed to compete with subject agreement, and sentence pairs differ minimally in the target verb.
  • Evaluation stimuli: The evaluation set contains 2,800 sentence pairs sampled from 25 adjectives, 40 nouns, 88 attractor nouns, and 180 verb pairs.Real-noun stimuli are converted to novel-noun tests by replacing the subject nouns with [wug] and [wugs].
  • Models: The case study evaluates the 2B and 4B post-trained Qwen3-VL models, whose embedding and unembedding matrices are tied.The authors note that future work could extend the method to models with untied embeddings.

4 Behavioral Evidence

Behavioral evaluation compares novel and real nouns across language and vision cue conditions and increasing attractor counts. Novel nouns remain above chance in both cue conditions, including when number is learned from vision alone.

  • Experimental design: The main experiment evaluates novel-noun number agreement using minimal-pair agreement-attraction stimuli and compares performance with real-noun subjects.Each cue condition is evaluated over novel embeddings from 50 training seeds.
  • Experimental design: Figure 2 reports behavioral accuracy with 95% confidence intervals across Language and Vision cues, 0–3 attractors, and real versus novel subjects.The black dashed line with triangular points denotes performance on corresponding real-noun stimuli.
  • Results: Novel-noun agreement performance is above chance for all attractor counts regardless of cue condition.The result indicates successful generalization from both language and vision exposure.
  • Results: Performance decreases as the number of intervening attractor nouns increases, while real nouns show a noticeable performance advantage over novel nouns.The introduction reports degradation when attractors reach ≥2.
  • Results: Vision-trained nouns support number-agreement behavior even though textual co-occurrence does not diagnose their number.This is the key behavioral evidence for cross-modal generalization of grammatical number.

5 Representational Dynamics of Learning

The study tracks how novel-word embeddings change during learning, measuring movement along a singular–plural direction defined by real nouns. Across language and vision cues, embeddings move toward regions associated with their intended number, although language-cued movement is larger.

  • 5.1 Methods: The analysis tracks novel-word embedding movement in a low-dimensional space organized around grammatical number.A 2D PCA is fit using real singular and plural nouns, while novel embeddings are represented at initialization and after training.
  • 5.1 Methods: Movement is the change in each novel embedding’s projection onto the singular–plural direction from final to initial training state.The number direction is derived from the difference between average singular- and plural-noun embeddings; intended singular movement is positive and plural movement negative.
  • 5.2 Results: Novel singular and plural nouns move toward regions occupied by real nouns carrying the corresponding number feature in both language and vision conditions.The pattern appears in the Qwen3-VL-4B embedding-space visualization across both cue conditions.
  • 5.2 Results: All movements point in their intended directions, with positive singular and negative plural movement, both significant at p <.001.Mean movement is averaged across 50 seeds and reported separately by cue condition and number.
  • 5.2 Results: Movement is greater on average when number is diagnosed from language cues than from visual cues.Despite the absence of diagnostic language information in the visual condition, vision-trained embeddings still show non-trivial movement toward number-relevant exemplars.

6 Mechanistic Evidence

The paper tests whether previously identified internal mechanisms for number agreement transfer to novel nouns learned from language or vision. Interventions produce positive causal effects across methods and cue conditions, with little visible difference between language and vision results.

  • 6.1 Methods: The mechanistic analysis uses counterfactual interventions to identify internal components involved in producing correct number-agreement completions.Source activations are transformed and applied to base stimuli, with efficacy measured by a log odds-ratio comparing output probabilities before and after intervention.
  • 6.1 Methods: Four methods are evaluated: Distributed Alignment Search, DiffMean, linear probes, and Attribution Patching.The methods are validated on held-out real-noun stimuli before being tested with novel nouns.
  • 6.2 Results: Validation Avg. Odds are above 0 for all methods, indicating qualitatively successful interventions, while DAS has comparatively higher effects across attractors and layers.For novel-word evaluation, the best layer is selected using average log odds-ratio on real-noun stimuli, then reused after novel nouns are added.
  • 6.2 Results: Novel-noun Avg. Odds are generally above 0 across methods and both cue conditions, showing that discovered mechanisms incorporate information acquired later.Figure 5 evaluates the best layers selected from the real-noun analysis; Figure 4 reports the layer-wise validation results.
  • 6.2 Results: Except for DAS with 0 attractors, language and vision cue conditions show no visual difference in the novel-noun causal results.The comparison uses layers selected from the earlier real-noun analysis.

7 General Discussion and Conclusion

The study concludes that VLMs can generalize grammatical number across modalities, including when novel nouns are learned from visual evidence alone. Their internal mechanisms treat language- and vision-acquired nouns similarly, suggesting abstraction-compatible behavior beyond surface co-occurrence.

  • 7 General Discussion and Conclusion: VLMs demonstrate cross-modal generalization of number agreement from both linguistic and visual cues.The conclusion is based on analyses of model behavior, representational dynamics, and causal mechanisms.
  • 7 General Discussion and Conclusion: Novel nouns learned from vision alone can be integrated by linguistic abstractions already present for words in the model’s embeddings.The authors characterize this as emergent abstraction of the number feature.
  • 7 General Discussion and Conclusion: Internal number-agreement mechanisms account for data acquired after their discovery and treat language- and vision-learned nouns in the same way.This supports flexibility in the mechanisms beyond the modality through which information was acquired.
  • 7 General Discussion and Conclusion: The findings indicate that VLMs can generalize beyond surface-form co-occurrence despite item-specific frequency effects.The discussion presents abstraction-compatible behavior and item-based learning as potentially less distinct than often assumed.

Limitations

The study’s scope is limited to two relatively small VLMs from one family, few images, and English, while using one novel-word-learning method.

  • The experiments cover only two lower-parameter VLMs from the same family, a limited number of images, and one language: English.
  • The study uses a single novel-word-learning method, excluding alternatives that may involve separate models or atypical training such as meta-learning fine-tuning.The authors state this was deliberate to preserve conclusions about the target model rather than method-specific idiosyncrasies.

A Additional Details of Embeddings Training

The appendix details how novel [wug] and [wugs] embeddings were initialized, trained, and evaluated under vision and language cue conditions.

  • A.1 Embeddings initialization: [wug] and [wugs] embeddings were initialized at the mean of 28 balanced singular–plural noun pairs, avoiding a singular or plural bias.The initialization nouns were single tokens, including pairs such as cat–cats, dog–dogs, and bird–birds.
  • A.2 Vision condition: Vision training paired five images of one chimeric creature with [wug] and five images of two creatures with [wugs].The images were generated from prompts designed around invented “snarples” rather than the familiar wug concept.
  • A.2 Vision condition: Vision-condition text captions omitted syntactic number cues, whereas language-condition stimuli explicitly marked whether [wug] and [wugs] were singular or plural.The language set contained 30 verified sentences with informative syntactic cues; the vision set randomly assigned 30 non-discriminating sentences across the two embeddings.
  • B.1 LR hyperparameter sweep: 10^-3 was the optimal learning rate for both models and cue conditions and was used for subsequent embedding training.The learning-rate sweep evaluated five random seeds at each rate; loss curves averaged 50 runs at the selected rate showed greater within-distribution loss for language than vision.
  • B.2 Generations: After training, Qwen3-VL-2B could productively generate next-token completions containing the learned [wug] and [wugs] embeddings.The generation analysis used minicons and examined base sentences containing either novel noun.

C.1 Replication of embeddings movement during training

Replication in Qwen3-VL-2B found that novel singular and plural embeddings moved along trajectories defined by real nouns, similarly across language and vision cues.

  • Qwen3-VL-2B replicated the embedding-movement analysis with qualitatively similar results.
  • Both [wug] and [wugs] representations diverged along trajectories defined by real singular and plural nouns.The PCA visualization tracks initial and final novel-word states against real noun embeddings organized by grammatical number.
  • Mechanistic analyses showed slightly weaker average odds for novel words in Qwen3-VL-2B than in the 4B model, but no difference between language and vision cue conditions.

D Details of Intervention Methods

The intervention analyses use several methods to identify or manipulate number-related representations, then model average odds across cue conditions, attractors, models, methods, and layers.

  • Distributed Alignment Search: Distributed Alignment Search learns a one-dimensional activation subspace that maximizes the likelihood of a counterfactual completion for paired source and base inputs.Its supervised objective is designed to capture the behavioral difference between the paired inputs when such a difference exists.
  • Figures: Across Qwen3-VL-2B layers, Figure 10 reports average odds on agreement stimuli with real nouns, where higher values indicate greater causal effect.
  • Figures: Figure 11 compares real and novel subjects at the best layer selected using real-noun results and shows generally high agreement across both cue conditions.
  • Linear Probe: Linear probes classify singular versus plural sentence activations with logistic regression, then steer completions by adding or subtracting the normalized weight vector.
  • Difference-of-Means: Difference-of-Means uses the difference between activation means to intervene toward plural or singular completions.
  • Attribution patching: Attribution patching estimates influential components with gradient-based effects and patches the top-k components between base and source sentences.The analysis focuses on MLP intermediate activations, which performed best on natural sentences.
  • Statistical analysis: The mixed-effects analysis predicts average odds from cue condition, attractors, model, and interpretability method, with layer as a random effect.DiffMeans and Probe had worse odds than DAS, and odds decreased as attractor count increased; model and cue condition were the exceptions among predictors without significant effects.

F.1 Replication of mechanistic results across multiple random seeds

Mechanistic findings generalized across additional novel-word embedding pairs and model sizes, remaining highly consistent with the main results. Attribution-patching efficacy varied by activation basis and top-k selection, with the strongest effectiveness observed for top-k MLP intermediate activations.

  • Replication: Four additional [wug]–[wugs] embedding pairs produced mechanistic results highly consistent with the main findings.This replication tested whether results depended on random initialization or the training dynamics of one embedding pair.
  • Replication: The replication covered Qwen-3-VL-2B and Qwen-3-VL-4B across all conditions.Figures 12 and 13 report the learning-rate-sweep mechanistic results for the two model sizes.
  • Activation basis and top-k selection: The activation basis and number of top-k activations were empirically compared for attribution patching.The comparison covered attention-head, residual-stream, and MLP activations pre- and post-linearity across k = 23...27.
  • Activation basis and top-k selection: Top k = 128 MLP intermediate activations yielded the greatest effectiveness for causal interventions in Qwen-3-VL-2B.These results are shown in Figure 14.
Loading 2609.00443v1…