Are In-Context Images Worth 10 Dimensions?

1 INTRODUCTION

• Prior work has studied how induction circuits use linear representations for classification, while this paper focuses on how those representations form.The authors use the continuous, diverse visual input space to trace feature formation across classification tasks.Source: §1 · p. 1
• The Shared Discriminative Geometry (SDG) is a low-dimensional text-token space shared across image-classification tasks where classes are linearly separable.The model uses this geometry to make predictions.Source: §1 · pp. 1–2
• The authors posit that early LVLM layers reduce vision-token dimensionality by projecting discriminative features onto principal components of the in-context examples.The resulting representations transfer to text tokens and are later used to attend to similar in-context examples.Source: §1 · pp. 1–2, Figure 1 · p. 2

2 RELATED WORK

• Prior text-classification work identified early-layer circuits that form linearly separable representations and later induction circuits that use them for query classification.The mechanism by which those high-quality representations form has received less attention.Source: §2 · p. 2
• Unlike prior fixed semantic subspaces in LLMs, the SDG is tied to vision and has a fixed subspace with context-dependent inner geometry.Earlier work identified low-dimensional subspaces for numerical, calendar, and refusal-related features.Source: §2 · p. 2
• The paper presents dimensionality reduction by projecting examples onto the current context’s principal components as a new in-context capability of linear transformers.Prior work had studied linear-transformer implementations of regression, expectation maximization, and reinforcement-learning algorithms.Source: §2 · p. 2

3.1 EXPERIMENT SETTING

• The ten datasets span diverse domains, including five fine-grained datasets, giving the study a broad range of visual classes and inputs.The authors argue that image diversity makes LVLM mechanisms shared across classification tasks easier to probe than text does.Source: §3.1 · pp. 2–3
• The setup samples 3-way, 3-shot tasks from ten image-classification datasets, using abstract label tokens to reduce the influence of label semantics.Each task supplies ten images: nine labeled support examples and one unlabeled query.Source: §3.1 · p. 3
• The study probes forerunner and average-pooled vision-token representations across layers, using Ministral 3 (3B) for the reported results.The forerunner token is the ':' after “label”; average pooling is used as a proxy for vision-token representations.Source: §3.1 · p. 3

3.2 MOTIVATION

• Raw forerunner representations are not class-linearly separable, whereas vision-token representations are; forerunner NCC accuracy is near random while vision-token accuracy is comparable to model accuracy across most layers.NCC accuracy serves as a proxy for class separability and feature quality.Source: §3.2 · p. 4

4.1 LEARNING THE SHARED DISCRIMINATIVE GEOMETRY

• The meta-training protocol learns layer-specific linear projections that make projected query representations align with their class centroid and be orthogonal to other-class centroids.The LVLM is frozen; the same protocol learns Text SDG from forerunner tokens and Vision SDG from average-pooled vision tokens.Source: §4.1 · pp. 4–5, Figure 2 · p. 4
• At the peak layer, Text SDG has an effective dimension of ∼10, and a fixed 10-dimensional subspace achieves 90% accuracy across 10 domain-specific datasets.The projections are trained to dimension 128; the paper notes effective dimensionality is a proxy that may vary with training setup.Source: §4.1 · p. 5
• Ablating the top 15 SDG dimensions decreases model accuracy by 12.9%, while boosting them increases accuracy by 3.9%; ablating a random subspace has negligible effect.The authors conclude that the SDG is low-dimensional and necessary for in-context image classification.Source: §4.1 · p. 5, Figure 3 · p. 5

4.2 TEXT SDG FORMATION AS A TWO-STEP PROCESS

• The proposed two-step circuit first reduces dimensionality in vision tokens through layer ℓ∗−1, then transfers the resulting geometry to the forerunner token at layer ℓ∗.The authors report that the low-dimensional vision projections become more class-discriminative even though full-space accuracy is not higher at layer ℓ∗−1.Source: §4.2 · pp. 5–6
• Vision SDG falls from 32 dimensions at the input to a minimum of 10 at layer ℓ∗−1 while preserving similar accuracy; its top 10 dimensions show 29% error reduction versus input representations.The top 10 dimensions at layer ℓ∗−1 achieve 90% accuracy using ∼2 times fewer dimensions.Source: §4.2 · p. 6, Figure 4 · p. 6
• At layer ℓ∗, the forerunner token matches the previous layer’s pooled vision-token feature quality and effective dimension, consistent with transfer of vision representations to text.Ablating Vision SDG at ℓ∗−1 reduces accuracy by 6.4 points versus 3.2 points for Text SDG ablation there; ablating vision-to-forerunner attention reduces accuracy by 5.2 points.Source: §4.2 · p. 6, Table 1 · p. 6

5.1 EXPLAINING WHAT DRIVES IN-CONTEXT DIMENSIONALITY REDUCTION

• Randomizing support-label assignments barely changes Text SDG formation accuracy or peak accuracy, although correct labels improve accuracy after formation.Source: §5.1 · p. 6
• Because NCC depends only on support-example similarities, a low-dimensional representation preserving X^T X leaves its accuracy unchanged and can use X’s principal components.Source: §5.1 · p. 7

5.2 DIMENSIONALITY

• Each linear self-attention layer implements a gradient step toward minimizing the mismatch between compressed and original sample similarities, so convergence requires multiple layers.Source: §5.2 · pp. 7–8
• The theoretical analysis assumes non-causal linear self-attention, suitable initialization, and a sufficiently small learning rate, while most LVLMs are causal.Source: §5.2 · pp. 7–8

5.3 EVIDENCE OF A SIMILAR MECHANISM IN LVLMS

• Ablating cross-image vision attention in the 2% of heads matching the proposed DR circuit removes the NCC accuracy boost in 10 dimensions.The comparison is shown after relearning the Vision SDG with the attention ablated.Source: Figure 5 · p. 8, §5.3 · p. 9

5.2 DIMENSIONALITY

• Under idealized assumptions, linear self-attention locally drives sample-to-sample similarity error toward zero at an exponential rate.Source: §5.2 · p. 8

5.3 EVIDENCE OF A SIMILAR MECHANISM IN LVLMS

• DR heads concentrate before ℓ*−1 and correlate with greater attention to other in-context images, though attention is not part of their selection criterion.Source: §5.3 · p. 8

• On SST-2, forerunner-token NCC accuracy is close to random before projection and increases after projection onto the Text SDG.Source: §6 · p. 9
• Modifying the SDG has the same effect as in image classification on multimodal classification tasks, while ablating it improves matching-task performance.Source: §6 · p. 9
• For matching tasks, support examples indicate the task, but the answer depends on matching inputs within the query rather than their visual similarity to support examples.Source: §6 · p. 9

7 CONCLUSION

• Across LVLM layers, the number of dimensions needed to preserve linear class separability decreases, and representation formation cannot be reduced to simple transfer from vision tokens to text tokens.Source: §7 · p. 9
• Whether the mechanism generalizes to LLMs or tasks beyond classification remains future work.Source: §7 · p. 9
• The proposed theoretical circuit is not directly identified in trained LVLMs; the authors identify only a potentially more sophisticated similar circuit.Source: §7 · p. 9

AI USE STATEMENT

• The authors report using generative AI for writing support, related-work retrieval, result interpretation, mathematical proof assistance, and code generation, with review of AI-assisted work.Source: AI USE STATEMENT · p. 10

REPRODUCIBILITY STATEMENT

• The code is included in supplementary material and is to be publicly released; the datasets are publicly available for download.Source: REPRODUCIBILITY STATEMENT · p. 10

Supplementary material overview

• The supplement includes the experimental setup, label-space details, training details, dimensionality analyses, and training on one dataset only.It also includes the theoretical analysis, circuit analysis, other-model experiments, and other-task details.Source: SUPPLEMENTARY MATERIAL OVERVIEW · p. 15
• Additional sections cover more ablations, vision-to-forerunner information flow, SDG alignment across layers, and dimensionality-reduction heads.Source: SUPPLEMENTARY MATERIAL OVERVIEW · p. 15
• Figure 6 plots model performance against 0, 2, 4, or 6 support examples for two label spaces in the 2-way setting.These support counts correspond to 0, 1, 2, or 3 shots per class.Source: Figure 6 · p. 16

A.1 LABEL SPACE

• The {Rud, Baz} label space avoids the nonzero cross-label probabilities observed with {Foo, Bar}, and improves performance by removing that bias.For 3-way classification, the label space adds Gip.Source: §A.1 · p. 16, Table 3 · p. 16

B.1 TRAINING DETAILS

• Training uses 32,768 episodes sampled uniformly across 10 datasets, with three classes, three support images per class, and one query per episode.Support prefixes provide one-, two-, and three-shot losses in a single forward pass; the LVLM remains frozen and no image augmentation is applied.Source: §B.1 · p. 16

B.2 DIMENSIONALITY

• The Text SDG’s top 10 directions account for 91.06% of projected variance, versus 45.36% for the input Vision SDG.The Text SDG spectrum drops sharply around rank 10, while the input Vision SDG decays more gradually; mean effective dimension across model projection types is about 90.Source: §B.2 · p. 17, Figure 7 · p. 17, Figure 8 · p. 18
• Across four training configurations, Vision SDG effective dimension remains close to 10, with only a slight increase for 4-way 4-shot training.Removing intermediate losses has little effect on effective dimension.Source: §B.2 · p. 17, Table 4 · p. 17

B.3 TRAINING ON ONE DATASET ONLY

• Textures-only training yields an effective dimension of 9.71, near the all-dataset value of 9.69, while Aircraft-only training yields 6.57.Thus, the approximately 10-dimensional geometry can emerge from one domain, but its effective dimensionality depends on the training domain.Source: §B.3 · p. 19, Table 5 · p. 19
• Between layers 7 and 15, single-domain SDGs gain NCC accuracy on other domains, and both become increasingly aligned with the all-dataset Vision SDG.The accuracy increases on the full set of 10 datasets as well as across Textures and Aircraft, providing complementary evidence of shared discriminative directions.Source: §B.3 · p. 19, Figure 10 · p. 19
• The training protocol combines features shared across domains with domain-specific features; each single-domain SDG performs better on its own domain than the other domain’s SDG.Training on 10 diverse domains should diminish this domain-specific effect; disentangling the two feature types is left for future work.Source: §B.3 · p. 19

C.1 PROOF OF THE CONVERGENCE RATE

• The convergence proof proceeds through descent and boundedness, convergence to zero error, and then exponential convergence once the error is sufficiently small.The stated rate is E(Zℓ) = O(exp(−4ηλnℓ)).Source: §C.1 · p. 20

C.1.3 EXPONENTIAL CONVERGENCE RATE. E(Zℓ) = O(exp(−4ηλnℓ))

• Near zero error, the proof gives a linear contraction with rate 1 − 2ηλn and a quadratic remainder.This yields the stated exponential error bound.Source: §C.1 · p. 20, §C.1.3 · p. 24

C.2 NUMERICAL EXPERIMENTS

• Causal masking slows convergence, while normalization and softmax have negligible impact under the tested conditions.The softmax result uses scale matching and centered inputs and initial states.Source: §C.2 · pp. 24–25
• Canonical initialization yields an error approximately ten times smaller than Gaussian and projection initializations, which behave similarly.Source: §C.2 · p. 25
• All tested numerical settings reach relative Gram error below 10−5 after all transformer layers, even with a 32× dimensionality reduction.The experiments test robustness to initialization, causal masking, normalization, and softmax attention.Source: §C.2 · pp. 24–25

D.1.1 ABLATING THE TOP SDG DIMENSIONS

• Ablating the top 15 SDG dimensions affects accuracy more than ablating all 128 for 2–5 shots; ablating dimensions 16–128 slightly improves accuracy, while boosting them reduces it.Ablating the SDG only from forerunner tokens has less effect; later layers may copy the SDG back from other text tokens.Source: §D.1.1 · p. 25, Table 6 · p. 25

D.1.2 ABLATION IMPACT ON HEADS ACCURACY

• After SDG formation, 14.77% of all heads and 23.29% of induction heads exceed 80% accuracy; ablating it reduces these shares to 1.42% and 1.37%.The accompanying figure tracks induction-head counts and head accuracy across layers, before and after ablation.Source: §D.1.2 · p. 27, Figure 12 · p. 27

D.2 ANALYSIS OF FLOW FROM VISION TO FORERUNNER

• The figure compares attention from the forerunner token to the query image with NCC accuracy across image-patch and newline-token positions.It marks three patch-token sinks and shows per-token accuracy after projection onto the Vision SDG.Source: Figure 14 · p. 28

D.3 SDG ALIGNMENT ACROSS LAYERS

• Vision SDG alignment weakens across more distant layers, Text SDG alignment stabilizes after ℓ*, and Vision–Text alignment remains low, consistent with a rotation during transfer.The authors identify separate layer-specific down-projections as useful for tracking changing Vision SDG geometry.Source: §D.3 · p. 28

D.4 ANALYSIS OF DIMENSIONALITY REDUCTION HEADS

• The selected dimensionality-reduction heads attend more to previous sink tokens from similar in-context examples, consistent with the derived circuit.The s_OV score is invariant to orthogonal rotations between subspaces, yet the authors report that it reliably selects these heads.Source: §D.4 · pp. 28, 30

E OTHER MODELS

• The additional-model analyses cover Ministral 3, Qwen 3, and Qwen 3.5, reporting SDG-projection NCC accuracy, effective dimension, and accuracy by projection dimension.The listed models include Ministral 3 (3B and 8B), Qwen 3 (4B), and Qwen 3.5 (4B and 9B).Source: E OTHER MODELS · p. 33, Figure 21 · p. 34, Figure 22 · p. 35

E.2 NCC ACCURACY IN THE SHARED DISCRIMINATIVE GEOMETRY; E.3 EFFECTIVE DIMENSION

• The appendix figures identify before-and-after SDG-projection NCC accuracy and effective dimension across layers as separate analyses.The supplied captions do not state their numerical outcomes.Source: Figure 21 · p. 34, Figure 22 · p. 35

SST-2

• SST-2 is evaluated as binary sentiment classification by replacing each image with its movie-review sentence’s text-token sequence.The protocol substitutes the image marker with a text marker and uses sentence-token embeddings.Source: F OTHER TASKS DETAILS · p. 36

Open MI; Matching MI

• Open MI evaluates concept-name classification on ImageNet images, while Matching MI asks whether image pairs satisfy relationships illustrated by demonstrations.Both benchmarks use all 200 provided tasks; Matching MI samples positive and negative pairs from the support set.Source: F OTHER TASKS DETAILS · p. 36

VLGuard; MHaluBench

• VLGuard tests harmful-versus-unharmful image-instruction pairs, and MHaluBench tests whether image-text claims are hallucinations.The evaluations use the first 200 prepared VLGuard pairs and 200 generated MHaluBench tasks.Source: F OTHER TASKS DETAILS · p. 36

VizWiz; SugarCrepe; NaturalBench

• VizWiz asks whether a question about a photograph is answerable; SugarCrepe and NaturalBench test image-caption matching as yes/no questions.VizWiz uses 200 generated tasks, as do SugarCrepe and NaturalBench.Source: F OTHER TASKS DETAILS · p. 36

Task formatting

• The formatting table distinguishes Matching MI’s two-image examples from the one-image format used by other benchmarks.Demonstrations are newline-separated, and evaluation matches the greedy next token after the query’s “Answer:” prompt.Source: F OTHER TASKS DETAILS · p. 36

Additional accuracy analysis

• The figure compares top-k SDG-dimension NCC accuracy for input vision tokens at layer 0 and the forerunner token at ℓ*.Source: Figure 23 · p. 37