PaperlayerView exact revision on arXiv

Source-linked AI summary

Technical Manual for a Toolkit for Measuring Contextual Individuation in Transformer Language Models

José Luciano Verçosa Marques, Frederico Jorge Heitmann, Daniel Omar Perez, Marcelo Vinicius de Paula, Tárcio André dos Santos Barros

arXiv:2609.05333v1cs.AIcs.CL

TL;DR

Existing evidence is compatible with context sensitivity but does not isolate it cleanly. This paper documents a toolkit built around bridge forms to examine contextual representations; it is an implementation reference, not a report of empirical outcomes for a particular model or bridge-form set.

  • Problem

    Existing evidence is compatible with context sensitivity but does not isolate it cleanly.

    Source: §1 · p. 2

  • Method

    The toolkit uses bridge forms—unchanged written words occurring across domains with different senses—and pairwise silhouette measurements to examine representation-space separation.

    Source: §1 · p. 2, §2 · p. 3

  • Results

    The toolkit has been exercised against architectures spanning bidirectional encoders and causal decoders, but the paper reports no model- or bridge-form-specific empirical outcomes.

    Source: §6.1 · p. 21

  • Takeaways & Limitations

    The bridge-form construct provides a structurally clean way to ask whether transformer language representations vary with context.

    Source: §10 · p. 25

    Source-document domain is a scalable proxy for sense, not a verified per-occurrence annotation.

    Source: §8 · p. 24

Abstract

from arXiv · show

A transformer language model assigns a single, context-independent vector to a word type at its embedding layer, yet is widely believed to individuate that word's occurrences by context in its later layers. Testing this belief cleanly requires a construct that holds the word form fixed while its context and intended sense vary in a controlled, labeled way. This manual documents an open toolkit built around such a construct, which we call a bridge form: a single written word that recurs, unchanged, across two or more subject domains with a different sense in each. We describe, and justify, every stage of the pipeline: the declarative specification of bridge forms and their source domains, corpus acquisition from Wikipedia, occurrence localization, layer-wise representation extraction, a domain-pairwise silhouette measurement of separation in the model's representation space, and a paired visualization protocol. Each design choice is presented together with the methodological failure mode it is meant to avoid (sense contamination from overly broad category labels, the multi-group bias of the silhouette coefficient, subword-tokenization misalignment, and axis-comparability artifacts in dimensionality-reduced plots, among others). This manuscript is a methodological and implementation reference: it does not report or interpret empirical outcomes of running the toolkit on any particular model or bridge-form set. The toolkit, its full source, and the corpora used to exercise it are archived separately (Section 9) under a persistent identifier, and are intended to be cited as an instrument by studies that use it to produce and interpret empirical results.

1. Introduction

The manual introduces bridge forms to test whether a transformer separates occurrences of the same word by context, and documents an end-to-end toolkit as a methodological instrument rather than reporting empirical findings.

1. Introduction

  • Existing evidence is indirect: downstream improvements with contextual representations (Peters et al., 2018) and probe-recoverable properties (Tenney et al., 2019; Belinkov and Glass, 2019) do not isolate spontaneous sense separation.Source: §1 · p. 2
  • A bridge form is one unchanged written word that occurs across two or more subject domains with a different domain-appropriate sense in each.Examples include current in physics, economics, and physical geography.Source: §1 · p. 2
  • Because all occurrences share one vocabulary entry and essentially the same embedding-layer vector, later-layer separation across domains reflects context rather than word identity, without requiring an annotated sense inventory.Source: §1 · p. 2
  • The toolkit specifies bridge forms and non-overlapping source categories, builds a corpus, locates occurrences, extracts hidden states at every layer, measures pairwise silhouette at full width, and visualizes occurrences with paired axis-comparable projections.Source: §1 · p. 2
  • The manual makes design choices explicit and justifies them so the toolkit can be reused, audited, and extended without reverse-engineering its source code.Source: §1 · p. 2
  • The manuscript is a methodological and implementation reference; it does not present or interpret results from running the toolkit on a particular model or bridge-form inventory.Source: §1 · p. 3

2. Intended Uses and Potential Contributions

The toolkit is presented as an instrument, not a claim: it makes four complementary questions about contextual meaning tractable without asserting answers from any particular run.

A training-free complement to probing

  • The pairwise silhouette measure tests whether a model’s own representation geometry separates senses without training, tuning, or relying on a classifier.This addresses a different question from what a trained probing classifier can decode from representations.Source: §2 · p. 3

Architecture-neutral model comparison

  • Treating models as black boxes that return per-layer hidden states lets the same instrument compare encoders and decoders of different sizes and pretraining objectives.This provides a common, repeatable basis for comparing how differently trained or shaped models organize context-dependent meaning.Source: §2 · p. 3

An empirical handle for lexical semantics

  • Bridge forms provide complementary, reproducible evidence about whether a system without a sense inventory organizes a word’s occurrences by context in ways resembling sense.This supplies an empirical handle on lexical semantics but does not settle philosophical questions on its own.Source: §2 · p. 4

A reusable data-collection method

  • Domain-labeled bridge-form occurrences collected from Wikipedia without manual annotation are a reusable resource for further sense annotation, evaluation, or teaching.The occurrence sets are grouped by source-document domain and are independent of downstream analysis.Source: §2 · p. 4

Scope

  • Scaling these uses across more models or bridge forms, or adding statistical validation, is beyond the scope of a single toolkit run or this manual.Source: §2 · p. 4

3. Background and Related Work

This section situates the toolkit’s question about sense separation in contextual representations within research on lexical ambiguity and layer-wise representations. It positions the bridge-form, domain-pairwise protocol among related work, highlighting differences in targets and designs without relying on a probe.

3.1. Classical Foundations

  • Static embeddings assign one vector per word and conflate its senses, whereas contextual models map a token and its surrounding context to a vector.The expectation that contextual models individuate senses is often supported indirectly through probing or task performance.Source: §3.1 · pp. 4–5
  • Probe success establishes that a property is decodable, not that a model uses it; this toolkit instead measures whether its geometry separates senses without a decoder.The bridge-form approach uses source-document topical domain as a scalable, naturalistic sense proxy rather than manually labeling occurrences, trading sense-level precision for scale.Source: §3.1 · p. 5
  • The toolkit computes silhouette scores one domain pair at a time rather than pooling three or more domains, avoiding a use that is not the coefficient’s default.The silhouette coefficient is a bounded, interpretable measure of how well labels match representation-space geometry.Source: §3.1 · p. 5

3.2.1. Behavioral word-sense disambiguation with LLMs

  • Meconi et al. (2025) report that leading LLMs match specialized WSD systems and generalize better across domains, while Basile et al. (2025) find fine-tuned mid-sized models win on classification.These behavioral evaluations assess outputs rather than whether internal representations already separate senses without a classifier or generation head.Source: §3.2.1 · p. 6

3.2.2. Unsupervised clustering of contextual embeddings against gold sense inventories

  • Mosolova et al. (2025) revisit clustering against manually sense-tagged resources, and Liétard et al. (2024) induce per-lemma senses from BERT embeddings; bridge-form uses naturally occurring topical domains as an unlabeled, low-cost proxy.The cited studies use neither a silhouette-style index nor a pairwise protocol; the proxy trades gold-standard precision.Source: §3.2.2 · pp. 6–7

3.2.3. Layer-wise probing and representation geometry

  • Liu et al. (2024) find lexical semantics peaks around layers 6–8 in Llama-2’s 32-layer decoder; on embedding benchmarks, Skean et al. (2025) find training objective, not modality, governs representation-quality distribution.Liu et al. probe with Word-in-Context and anisotropy-corrected cosine similarity; Skean et al.’s finding concerns embedding benchmarks rather than lexical ambiguity specifically.Source: §3.2.3 · p. 7

3.2.4. Ambiguity-focused geometric and causal analyses

  • Scott et al. (2026) report that ambiguous-word senses become most distinct in middle layers and reconverge by final layers even as output predictions diverge.Their activation patching and layer ablation support a causal account; the result cautions against treating late-layer cosine geometry as the natural locus of sense separation, so the toolkit reports all layers.Source: §3.2.4 · p. 8

3.2.5. Where do the bridge forms stand?

  • The section positions bridge forms as combining a naturalistic, unlabeled domain proxy for word sense with a domain-pairwise silhouette measurement, a combination absent from the reviewed studies.The supplied passage begins a list of combined features but provides only these two items.Source: §3.2.5 · p. 8
  • Nearby work is behavioral, gold-sense-annotation-based, or focused on general layer-wise information rather than lexical ambiguity specifically.The subsection also describes the closest work as studying model outputs rather than representation organization.Source: §3.2.5 · p. 8
  • The contemporaneous study by Scott et al. (2026) is decoder-only and causal-interventionist, unlike the toolkit’s unsupervised geometric-clustering design across architecture families.The subsection presents this distinction as part of the specific gap the toolkit addresses.Source: §3.2.5 · p. 8
  • The toolkit targets a gap with a protocol designed to avoid pooled-silhouette bias and a reusable pipeline spanning bidirectional encoders and causal decoders.The subsection identifies this as the specific gap the manual’s toolkit is built to fill.Source: §3.2.5 · p. 8
  • Section 8 states plainly what the toolkit still does not do.Source: §3.2.5 · p. 8

4. Design Principles

The toolkit’s design principles pair each measurement or visualization choice with the methodological failure it is meant to prevent. Together, they define how bridge forms are selected, localized, and compared across domains, layers, and model architectures.

4.1. Fix the form, vary the context

  • Holding a bridge form’s written form fixed while varying domain context and intended sense lets later-layer separation be attributed to context rather than word type.Source: §4.1 · p. 9

4.2. Specific, non-overlapping category labels

  • Each domain uses a specific category chosen before extraction or scoring, preventing broad labels from mixing senses and category choices from being made post hoc.Source: §4.2 · p. 9

4.3. Pairwise, not pooled, comparison

  • Pairwise silhouette scores compare exactly two domains, avoiding pooled-group artifacts in which nearby groups lower one another’s scores and isolated groups appear artificially clean.Source: §4.3 · pp. 9–10

4.4. Full representation width, not a low-dimensional projection

  • Separation is measured with cosine distance at full hidden width; paired PCA plots illustrate the geometry but cannot substitute for the score because projection is lossy.Source: §4.4 · p. 10

4.5. A shared basis for comparable layers

  • The toolkit fits one PCA basis to pooled embedding- and final-layer points so panel axes remain comparable, avoiding apparent movement caused by independently reoriented projections.The shared basis trades each layer’s own best two-dimensional view for directly comparable axes.Source: §4.5 · p. 10

4.6. Offset-based, not string-based, token localization

  • Character-offset mapping locates a word occurrence’s token index, avoiding unreliable subword-string matching when byte-level BPE spellings vary by sequence position.Source: §4.6 · pp. 10–11

4.7. Architecture-neutral design, stated caveats

  • The pipeline applies without modification to bidirectional encoders and causal decoders, but lower separation under a causal model alone does not establish weaker context-sensitivity without controlling for sentence position.Early bridge forms have structurally less preceding context in causal models.Source: §4.7 · p. 11

5. The Toolkit

The toolkit is a modular six-stage pipeline for specifying bridge forms, acquiring and searching domain corpora, extracting layer-wise vectors, and measuring contextual individuation. It presents pairwise, layer-wise silhouette scores alongside paired PCA visualizations as methods and outputs, without reporting empirical findings about later-layer behavior.

5. The Toolkit

  • The six-stage pipeline passes each stage’s output to the next, while allowing stages to be rerun or substituted if they retain the document-ID-to-plain-text interface.For example, a different corpus source can replace Wikipedia without requiring changes to later stages.Source: §5 · p. 11

5.1. Bridge-form declaration

  • Each bridge form is declared as a mapping from domain labels to specific source categories, and that declaration drives later stages before text or vectors are inspected.It serves as the single source of truth for what counts as an occurrence of each sense.Source: §5.1 · p. 12

5.2. Corpus acquisition

  • Corpus acquisition uses articles directly in each source category and in its direct subcategories one level down, then uses plain-text introductions to limit article-length skew.The one-level expansion adds coverage without traversing the full category tree and risking topic drift.Source: §5.2 · p. 13
  • For batched Wikipedia requests, requesting introductions is essential: otherwise the API silently returns extracts for only a small, non-representative fraction of intended titles.The passage specifies that requesting introductions allows batches of up to twenty titles to return twenty extracts; without it, the corpus may be truncated without an error.Source: §5.2 · pp. 13–14

5.3. Occurrence localization

  • Occurrence localization searches each domain corpus for whole-word, case-insensitive matches and groups them by domain label, the grouping used downstream.Source categories supply the text but are not the groups compared by the silhouette computation.Source: §5.3 · p. 15

5.4. Representation extraction

  • For each located occurrence, the toolkit tokenizes the document once with character offsets and uses one model pass to collect the matched token’s vector at every layer.The outputs include the embedding layer and one representation per transformer block; only the first occurrence in a document is retained to prevent long documents from dominating a domain’s sample.Source: §5.4 · p. 16
  • If a matched occurrence falls outside the tokenizer’s truncation window, it is excluded rather than assigned to a misaligned token.The character-offset mapping identifies the token whose span contains the match start.Source: §5.4 · p. 16

5.5. Individuation measurement

  • Individuation measurement is organized by bridge form and uses domain-pairwise silhouette computations.The supplied passage begins its description by specifying independent computation for each bridge form’s domains.Source: §5.5 · p. 18
  • For each domain pair and layer, the toolkit reports the mean cosine-distance silhouette over occurrences in that pair, rather than pooling all domains.The pairwise design avoids ambiguity from pooled comparisons, where nearby domains can lower each other’s scores while an isolated domain appears artificially distinct.Source: §5.5 · pp. 18–19
  • Cosine distance discards vector magnitude, which can reflect frequency, position, and layer-normalization scaling rather than lexical sense; later-layer behavior remains an empirical question.The manual expects embedding-layer scores to be near zero for every pair, but does not report later-layer outcomes.Source: §5.5 · p. 18

5.6. Visualization

  • For each bridge form, the toolkit plots embedding- and final-layer PCA panels with shared symmetric axes and aspect ratio, alongside one silhouette-by-layer curve per domain pair.Distinct marker shapes and colors identify domains, while blended colors identify the domain pair for each curve.Source: §5.6 · pp. 19–20
  • Figure 2 illustrates the visualization format; its displayed pairwise curve patterns are not reported, interpreted, or generalized as findings.A shared PCA basis keeps projected layers on the same axes for direct comparison rather than using independently optimized views.Source: §5.6 · p. 20, Figure 2 · p. 21

6. Implementation and Reproducibility

The section describes the toolkit’s model compatibility, software dependencies, reproducibility conditions, and data-provenance practices. Its scope is one model per run, with results tied to the corpus snapshot and checkpoint used.

6.1. Supported model families

  • The toolkit was exercised across encoder and decoder architectures and accepts any model exposing a standard per-layer hidden-states interface without pipeline modification.The tested architectures span pretraining objectives, model sizes, and attention mechanisms.Source: §6.1 · p. 21

6.2. Software dependencies

  • The reference implementation is a sequential notebook using transformers, torch, scikit-learn, matplotlib, and numpy, with dependencies pinned in the archived environment file.Scikit-learn supplies silhouette and PCA functionality (Pedregosa et al., 2011).Source: §6.2 · p. 22

6.3. Determinism

  • With a fixed corpus snapshot and model checkpoint, acquisition and inference are deterministic; PCA uses a fixed random seed, but changing Wikipedia categories can change the corpus.The corpus used for each run should therefore be archived with derived results rather than fetched again on demand.Source: §6.3 · p. 22
  • Each run processes one selected model; architecture comparisons require rerunning the full pipeline with another selection rather than using a built-in batch operation.This is a deliberate scope boundary given the computational cost of running multiple transformer models end to end.Source: §6.3 · p. 22

6.4. Data provenance and etiquette

  • Corpus text is fetched live from Wikipedia using an identifying User-Agent and contact address from a local untracked configuration file; the toolkit does not redistribute text beyond license permissions.Requests follow the API’s etiquette, and archived materials include no Wikipedia text beyond what its license allows.Source: §6.4 · pp. 22–23

7. Interpreting the Toolkit’s Output

Section 7 explains how to read the toolkit’s panels and curves as an instrument guide, not as a report of empirical findings for a particular model or bridge-form set.

Interpreting the Toolkit’s Output

  • The embedding-layer panel should show occurrences of a fixed bridge form collapsing together across domains, serving as a manipulation check before claims about later layers.If they do not collapse, the occurrences may not share the assumed vocabulary entry, so the construct’s structural guarantee does not hold in that case.Source: §7 · p. 23
  • Final-layer panels and pairwise silhouette curves show domain-related organization; nonzero scores quantify pair separability, while curve trajectories indicate when separation appears and whether it is sustained.The curve’s shape across layers conveys information beyond its value at any single layer.Source: §7 · p. 23
  • PCA-axis percentages report the share of full-hidden-width variance shown in the two-dimensional panel, not domain separation, which is measured by the full-width silhouette score.The axis percentages may be small in spaces with several hundred dimensions.Source: §7 · p. 23
  • The manual does not present or interpret results for any particular model, bridge form, or layer range; those patterns are empirical questions for studies using the toolkit.This section is a reading guide for the instrument rather than a report of what it found in the illustrated examples.Source: §7 · pp. 23–24

8. Scope and Limitations

The toolkit’s explicit limitations constrain interpretation of later-layer results, while leaving its embedding-layer structural guarantee intact. These boundaries are stated for future work, not as claims the toolkit makes.

Scope and Limitations

  • Silhouette scores are point estimates without permutation baselines or confidence intervals, and their departure from zero should be considered alongside reported occurrence counts.The toolkit does not gate results based on those counts.Source: §8 · p. 24
  • The toolkit computes and plots silhouette curves regardless of sample size, leaving users to assess whether occurrence counts are adequate.Domain counts can be small because they depend on how often a bridge form appears in a domain’s corpus.Source: §8 · p. 24
  • Corpus coverage and stability depend on Wikipedia’s evolving category structure, including its directly attributed articles and one-level-deep subcategories.A declared source category may contain few or no directly attributed articles beyond those supplied by its subcategories.Source: §8 · p. 24
  • Source-document domain is a scalable proxy for sense, not verified per-occurrence annotation, so atypical uses in domain-typical documents can go undetected.The toolkit does not detect or filter those atypical senses.Source: §8 · p. 24
  • Each run uses one model, requiring separate pipeline runs for cross-architecture comparisons, and retains only each document’s first bridge-form occurrence.Keeping only the first occurrence bounds the sample below the corpus total.Source: §8 · p. 24
  • These limitations bound conclusions from a run’s later-layer results but do not compromise the embedding-layer structural guarantee, which is independent of sample size, category coverage, and model choice.The manuscript does not make claims about particular later-layer empirical outcomes.Source: §8 · p. 25

9. Data and Code Availability

The toolkit’s source, reproducibility environment, and exercise corpora are archived separately from the manuscript, with guidance on how to cite the toolkit and paper.

9. Data and Code Availability

  • The toolkit’s full source code, environment specification, and corpus snapshots are archived under a persistent identifier independently of the manuscript.The environment specification covers the software dependencies needed to reproduce the toolkit.Source: §9 · p. 25
  • The toolkit is released under a CC BY 4.0 license; its concept DOI resolves to the latest release, currently v1.0.0, with version DOI 10.5281/zenodo.22213867.The toolkit’s DOI is 10.5281/zenodo.22213866.Source: §9 · p. 25
  • Cite the archived toolkit when reporting results produced with it, and cite the manuscript when referring to its design rationale.The manuscript is intended for archival on arXiv under a CC BY 4.0 license.Source: §9 · p. 25

10. Conclusion

The bridge-form construct isolates contextual sense by holding word form fixed while varying domain-appropriate sense, and the toolkit operationalizes it with design choices intended to make outputs interpretable. Applying the instrument to particular models and bridge forms remains a separate empirical question not addressed in this manuscript.

10. Conclusion

  • Bridge forms test contextual individuation by keeping the written word fixed while varying its domain-appropriate sense, so separation beyond the embedding layer cannot be attributed to word identity.The construct provides a structurally clean way to ask whether a transformer individuates a word’s occurrences by context.Source: §10 · p. 25
  • The toolkit operationalizes the construct end to end and justifies choices intended to make its output interpretable, including pairwise comparisons, full-width measurement, offset-based token localization, and a shared visualization basis.The manual presents these design choices as safeguards for interpreting toolkit outputs.Source: §10 · p. 25
  • Whether particular models and bridge forms exhibit contextual individuation is a separate empirical question for work using the instrument, not a result reported in this manuscript.The conclusion distinguishes documenting the instrument from applying it to produce empirical findings.Source: §10 · p. 26

Related papers

Loading 2609.05333v1…