PaperlayerView exact revision on arXiv

Source-linked AI summary

Technical Manual for a Toolkit for Measuring Contextual Individuation in Transformer Language Models

José Luciano Verçosa Marques, Frederico Jorge Heitmann, Daniel Omar Perez, Marcelo Vinicius de Paula, Tárcio André dos Santos Barros

arXiv:2609.05333v1cs.AIcs.CL

TL;DR

Testing contextual individuation is difficult because existing evidence does not isolate context sensitivity. The toolkit uses unchanged bridge forms across domains to provide a structurally clean test, while recognizing that domain labels are only proxies for sense and later-layer results require bounded interpretation.

  • Problem

    Existing evidence is compatible with context sensitivity but does not isolate it cleanly.

  • Method

    The toolkit uses bridge forms: single written words occurring unchanged across domains, with layer-wise representations compared using a deliberate domain-pairwise measurement.

  • Results

    The bridge-form construct gives a structurally clean way to ask whether a transformer language model individuates word occurrences by context.

  • Takeaways & Limitations

    The toolkit supports controlled investigation of contextual individuation without relying on a classifier or generation head to make the decision.

  • Takeaways & Limitations

    Domain labels are scalable proxies for sense rather than verified per-occurrence annotations, and later-layer results bound what can be concluded from a particular run.

Abstract

from arXiv · show

A transformer language model assigns a single, context-independent vector to a word type at its embedding layer, yet is widely believed to individuate that word's occurrences by context in its later layers. Testing this belief cleanly requires a construct that holds the word form fixed while its context and intended sense vary in a controlled, labeled way. This manual documents an open toolkit built around such a construct, which we call a bridge form: a single written word that recurs, unchanged, across two or more subject domains with a different sense in each. We describe, and justify, every stage of the pipeline: the declarative specification of bridge forms and their source domains, corpus acquisition from Wikipedia, occurrence localization, layer-wise representation extraction, a domain-pairwise silhouette measurement of separation in the model's representation space, and a paired visualization protocol. Each design choice is presented together with the methodological failure mode it is meant to avoid (sense contamination from overly broad category labels, the multi-group bias of the silhouette coefficient, subword-tokenization misalignment, and axis-comparability artifacts in dimensionality-reduced plots, among others). This manuscript is a methodological and implementation reference: it does not report or interpret empirical outcomes of running the toolkit on any particular model or bridge-form set. The toolkit, its full source, and the corpora used to exercise it are archived separately (Section 9) under a persistent identifier, and are intended to be cited as an instrument by studies that use it to produce and interpret empirical results.

1. Introduction

The manual addresses whether later-layer representations individuate occurrences of the same word by context rather than word identity. It introduces bridge forms and an end-to-end toolkit designed to isolate and measure this question while making methodological choices explicit and reusable.

  • Motivation: Existing contextual-sensitivity evidence from downstream performance and probing does not isolate whether models spontaneously separate senses of the same word type.Probes can succeed because annotated properties are recoverable, without showing that the model keeps senses distinct in its own geometry.
  • Bridge-form construct: A bridge form is one written word occurring unchanged across multiple subject domains while carrying a domain-appropriate sense in each.The construct can include examples such as current in physics, economics, and physical geography.
  • Bridge-form construct: Because all occurrences share one vocabulary entry and therefore the same embedding-layer vector up to positional variation, later-layer separation can be attributed to context rather than word identity.This provides a structural guarantee that does not depend on downstream behavior or an annotated sense inventory.
  • Toolkit design: The toolkit declares bridge forms and specific non-overlapping source categories, builds a corpus, locates occurrences, extracts every-layer hidden states, measures pairwise domain separation, and produces paired visualizations.It uses a full-width pairwise silhouette coefficient and an axis-comparable projection to represent the same occurrences.
  • Scope and purpose: The manual makes each non-forced methodological choice explicit and justified so the toolkit can be reused, audited, and extended without reverse-engineering source code.It is scoped as a methodological and implementation reference rather than an empirical study.

2. Intended Uses and Potential Contributions

The toolkit is presented as a reusable, architecture-neutral instrument for studying context-dependent word-sense organization without relying on trained classifiers. Its outputs also support empirical semantics research and downstream resources, while the manual does not undertake large-scale validation or resolve philosophical questions.

  • Scope and contribution: The toolkit’s value lies in making questions tractable and reproducible rather than in supplying a specific answer or settling philosophical questions.Scaling to more models, bridge forms, or statistical validation is beyond a single toolkit run and this manual.
  • A training-free complement to probing: The toolkit measures whether a model’s own representation geometry keeps word senses apart without a trained decoder or classifier.This distinguishes the toolkit’s question from probing, which asks what can be linearly decoded after training on labeled data.
  • Architecture-neutral model comparison: The same instrument applies unmodified across encoder and decoder architectures, model sizes, and pretraining objectives.This provides a common basis for comparing how differently trained or shaped models organize context-dependent meaning.
  • An empirical handle for lexical semantics: The bridge-form construct offers empirical evidence about whether a system without a sense inventory organizes a word’s occurrences by context in a sense-like way.It complements intuition, dictionaries, and manually sense-tagged corpora as a source of evidence for lexical semantics.
  • A reusable data-collection method: Domain-labeled occurrence sets grouped by source-document domain at Wikipedia scale can be reused for sense annotation, evaluation, or teaching.The resource is produced without manual annotation and is independent of downstream analysis.

3. Background and Related Work

Prior work established contextual representations and analyzed sense information through behavioral tasks, probing, clustering, and layer-wise studies, but does not directly test whether model geometry separates senses without a decoder. This toolkit addresses that gap with bridge forms, domain-based sense proxies, pairwise silhouette scores, and a cross-architecture pipeline.

  • Static embeddings assign one vector per word, whereas contextual models represent tokens as functions of their surrounding context.
  • Probing classifiers show that properties are linearly decodable, but not that the network itself uses those properties downstream.
  • The toolkit instead asks whether a model’s own geometry spontaneously separates senses without a classifier or generation head.
  • Bridge forms use topical document domains as a scalable, low-cost sense proxy, trading sense-level precision for scale and avoiding a fixed sense inventory.
  • Prior clustering studies use gold sense inventories and standard cluster metrics, whereas this toolkit uses a pairwise silhouette protocol.
  • Existing layer and ambiguity studies do not provide a reusable toolkit spanning bidirectional and causal architectures for this targeted question.

4. Design Principles

The toolkit fixes the word form while varying context and intended sense, then uses controlled labels, full-width geometry, pairwise scoring, token offsets, and shared visualization axes to isolate contextual effects and avoid methodological artifacts.

  • 4.1. Fix the form, vary the context: A bridge form holds the written word fixed while context and intended sense vary across domains, making later-layer separation interpretable as a context effect.
  • 4.2. Fix the labels before extraction: Specific source categories are chosen in advance because broad labels can mix senses and contaminate occurrence pools.
  • 4.3. Measure separation pairwise: Pairwise silhouette scores prevent close domain pairs and isolated third domains from distorting one another’s multi-group scores.
  • 4.4. Keep measurement and visualization distinct: Separation is reported in the full hidden width using cosine distance, while two-dimensional plots remain illustrative rather than evidential.
  • 4.5. Use comparable visualization axes: A shared PCA basis makes embedding- and final-layer panels comparable, trading each layer’s best two-dimensional view against axis-reorientation artifacts.
  • 4.6. Align occurrences to tokens: Character-offset mapping locates bridge-form tokens reliably across subword spellings, including byte-level BPE variation.
  • 4.7. Account for architecture: Causal decoders have less available context for early tokens than bidirectional encoders, so lower separation scores cannot alone establish weaker context sensitivity.

5. The Toolkit

The toolkit defines bridge forms and moves through corpus construction, occurrence localization, layer-wise vector extraction, pairwise silhouette measurement, and visualization. Its stages are modular, with safeguards for category drift, API truncation, token misalignment, and multi-group silhouette ambiguity.

  • Pipeline structure: The six-stage pipeline passes each stage’s output to the next and permits stages to be rerun or substituted through a shared text interface.A different corpus source can replace Wikipedia without changing later stages if it maps document identifiers to plain text.
  • Individuation measurement: Silhouette scores are computed in cosine distance for each unordered domain pair at every layer, avoiding ambiguity from applying the coefficient directly to three or more groups.Pairwise restriction makes each score compare only the two domains in that pair.
  • Visualization: The toolkit pairs layer-wise silhouette curves with shared-basis PCA plots of the first and last layers.The paired display separates full-width quantitative measurement from two-dimensional visualization.
  • Declaration and corpus construction: Bridge forms are declared as domain-to-source-category mappings before text retrieval, providing the single source of truth for downstream sense labels.Occurrences are later grouped by domain label rather than by the source category used to collect text.
  • Declaration and corpus construction: Wikipedia corpora use direct category members plus one level of subcategories and article introductions, balancing coverage against topic drift and article-length skew.The introduction-only API setting is required for batched extraction; omitting it silently truncates batches to a small, potentially non-representative fraction.
  • Localization and extraction: Occurrences are found by whole-word matching, while the form’s token is located by character offsets and excluded if it falls outside the tokenizer’s truncation window.Only the first occurrence per document is retained, preventing long documents from disproportionately contributing points.
  • Localization and extraction: A single forward pass returns the bridge form’s vector at the embedding layer and every transformer block, enabling layer-wise analysis.The toolkit uses the token’s own position and hidden-state outputs rather than matching a subword string after tokenization.

6. Implementation and Reproducibility

The reference implementation supports several encoder and decoder families through a standard hidden-state interface and uses pinned, reproducible dependencies. Runs are otherwise deterministic for fixed corpus snapshots and checkpoints, but architecture comparisons require separate full executions.

  • Supported model families: The toolkit has been designed and checked against bidirectional encoders, causal decoders, and encoder variants differing in objective, size, and attention mechanism.Any model exposing the standard hidden-states-per-layer interface can be substituted without modifying the pipeline.
  • Implementation: The reference notebook uses transformers and torch for inference, scikit-learn for silhouettes and PCA, and matplotlib/numpy for visualization and array handling.Dependency versions are pinned in the archived environment file.
  • Reproducibility: PCA uses a fixed random seed, while acquisition and inference are deterministic given a fixed corpus snapshot and model checkpoint.Wikipedia category contents can change between runs, so the operated-on corpus should be archived with derived results.
  • Scope of execution: Each execution processes exactly one model, so comparing architectures requires rerunning the full pipeline with a different model selection.Multi-model batching is outside the toolkit’s current scope because of the computational cost of end-to-end processing.

7. Interpreting the Toolkit’s Output

The output combines embedding- and final-layer PCA panels with pairwise silhouette curves to guide interpretation across layers. It is a reading guide for the instrument, not a report of empirical outcomes for particular models or bridge forms.

  • Scope of interpretation: Figure 2 illustrates the output format and makes differing pairwise curve shapes legible without reporting or interpreting the displayed patterns as findings.The examples are not evidence about any particular model, bridge form, or layer range.
  • Embedding-layer panel: The embedding-layer PCA panel is expected to show no domain-related structure and serves as a manipulation check on the bridge-form construct.A failure to collapse toward the same region indicates that occurrences may not share the assumed vocabulary entry.
  • Final-layer panel and curves: The final-layer panel and silhouette curves show whether domain-related organization is present in later representations.A curve departing from zero quantifies pairwise separability at that layer, while its shape shows when separation appears or persists.
  • Reading PCA axes: PCA axis percentages report the share of full-hidden-width variance captured by each displayed axis, not domain separation.Separation claims are carried by full-width silhouette scores rather than by the reduced plot.

8. Scope and Limitations

The toolkit’s conclusions are bounded by evolving Wikipedia coverage, proxy domain labels, sampling and inference design choices, and the absence of built-in significance testing. These constraints limit what later-layer results from an individual run can support, while leaving the embedding-layer structural guarantee intact.

  • Statistical support: Silhouette scores are reported as point estimates without permutation baselines or confidence intervals, and the toolkit does not enforce a minimum sample size.Readers should interpret departures from zero in light of the reported occurrence counts.
  • Corpus coverage: Wikipedia category structure and changing contents bound corpus coverage, size, and stability because retrieval uses direct members and one-level subcategories.The external resource can contain few or no directly attributed articles beyond what its subcategories supply.
  • Sense labeling: Domain labels are scalable proxies for sense rather than verified per-occurrence annotations, so atypical uses within domain-typical documents are not filtered.The toolkit groups occurrences by source-document domain but does not independently verify each occurrence’s sense.
  • Model comparison: Cross-architecture comparison requires one full pipeline rerun per model rather than an automated multi-model execution.The one-model-per-run boundary is deliberate and reflects end-to-end computational cost.
  • Sampling: Only the first occurrence per document is retained, bounding the sample below the total number of available corpus occurrences.This sampling rule limits the amount of evidence contributed by each document.
  • Interpretive boundary: These limitations bound claims about later-layer results from particular runs but do not compromise the embedding-layer structural guarantee.The guarantee does not depend on sample size, category coverage, or model selection.

9. Data and Code Availability

The toolkit’s source code, software environment, dependencies, and corpus snapshots are archived independently under a persistent identifier. The release is licensed under CC BY 4.0, with separate citation guidance for the toolkit and this manuscript.

  • The full source code, environment specification, dependencies, and corpus snapshots are archived independently under a persistent identifier.
  • The archived toolkit is identified as Verçosa Marques et al. (2026), Toolkit for measuring contextual individuation in transformer language models.
  • The archive provides a concept DOI resolving to the latest release and a version DOI for v1.0.0 deposited in August 2026.
  • The toolkit and manuscript are released under the Creative Commons Attribution 4.0 International license.
  • Users should cite the archived toolkit when reporting results produced with it and cite this manuscript when referring to its design rationale.

10. Conclusion

The bridge-form construct fixes a word’s written form while varying its domain-appropriate sense, providing a clean test of contextual individuation. The toolkit operationalizes this construct with interpretable pairwise measurements and aligned visualizations, but empirical findings for particular models and bridge forms remain outside this manual.

  • The bridge-form construct fixes word form while varying domain-appropriate sense, so later representational separation cannot be attributed to word identity.
  • The toolkit operationalizes the bridge-form construct end to end and justifies the design choices governing its output.
  • Its interpretability choices include pairwise rather than pooled comparison, full-width rather than projected measurement, offset-based token localization, and a shared basis across layers.
  • What the instrument reads out for specific models and bridge forms is a separate empirical question addressed by subsequent work.
Loading 2609.05333v1…