PaperlayerView exact revision on arXiv

Source-linked AI summary

Intuition vectors

Shahar Haim, Daniel C. McNamee

arXiv:2609.32619v1cs.LGcs.AI

TL;DR

The paper asks whether pretrained visual representations can support abstract reasoning without task-specific training. It tests frozen embeddings with simple geometric operations across Bongard and ARC tasks, finding concept discrimination and task-level transformation signals alongside a distinction between representing transformations and generating outputs.

  • Problem

    The paper asks how far abstract reasoning can be supported by relational structure already present in pretrained representations, without task-specific representation or generation training.

    Source: §1 · p. 1

  • Method

    The authors apply nearest-centroid classification and demonstration-derived input-output embedding differences to frozen visual encoders, using random pixel projections as a control.

    Source: §1 · p. 2

  • Results

    Frozen representations expose abstract concept information beyond pixel similarity, with Bongard–LOGO reaching 64.2% accuracy for DINOv3 at 224 px and ARC intuition vectors providing an operational task-level signal.

    Source: §4.1 · pp. 4–5, §5 · p. 9

  • Takeaways & Limitations

    Representing an abstract transformation and executing its instance-specific consequence appear to be distinct capacities.

    Source: §5 · p. 9

    The findings’ generality to other architectures, model sizes, and domains remains for future work, and the authors do not claim an ARC solution or include a generative component.

    Source: §2.3 · p. 3, §5.1 · p. 10

Abstract

from arXiv · show

Large self-supervised vision models learn representations that support scene segmentation and the semantic decomposition of physical objects. We ask whether their representational geometry supports transfer to visual reasoning problems without any task-specific fine-tuning. We hypothesized that relational representations may bridge perception and abstract reasoning by encoding similarities and transformations among visual inputs such that an intuitive, implicit form of reasoning may be performed via latent vector arithmetic. Specifically, we examine DINOv3, MAE, and random pixel projections on abstract and naturalistic Bongard problems, ARC-AGI-1 and ARC-AGI-2, and novel ARC-GEN instances. On both Bongard benchmarks, the accuracy of a simple nearest-centroid readout of frozen visual embeddings is within four percentage points of the task-specific baselines reported with the original benchmarks. In ARC, latent difference vectors summarizing demonstration input-output transformations, which we refer to as intuition vectors, show greater alignment with test vectors from the same task, whereas those from unrelated tasks are near orthogonal. This latent geometry is operational: transporting a query along its intuition vector consistently improves exact-output retrieval, reaching 70.7 on ARC-AGI-2 evaluation. Across 397,000 ARC-GEN instances from 794 tasks, single-pair intuition vectors identify the generating task with approximately 87\% leave-one-out accuracy. These findings suggest that latent vector arithmetic over frozen visual representations supports implicit rule inference across varied problem domains without a generative model component, indicating that inferring an abstract transformation and generating its instance-specific consequence may be separable capacities.

1 INTRODUCTION

The paper asks whether relational structure already present in pretrained representations can support abstract reasoning without task-specific training. It probes this question with simple geometric operations on frozen visual embeddings.

1 INTRODUCTION

  • Prior work reports structured representations in language, vision, and video models, alongside growing efforts to reuse pretrained representations in modular architectures.The cited examples include geometric organization of semantic concepts and physics-related structure in video models.Source: §1 · p. 1
  • The motivating gap is that abstract reasoning systems are commonly tailored to tasks, benchmarks, or individual instances, unlike emerging evidence of structured pretrained representations.The question is how far those existing representations can support abstract reasoning without task-specific representation or generation training.Source: §1 · p. 1
  • The study probes concept discrimination, rule representation within and across tasks, and rule application using nearest-centroid classification and input-output embedding differences.Random pixel projections serve as a low-level control for separating task-level structure from instance-specific information needed for exact outputs.Source: §1 · p. 2

2 RELATED WORK

The related work frames the study around whether pretrained representations have functionally useful structure for abstract reasoning. It contrasts the paper’s frozen-representation approach with task-specific Bongard and ARC methods.

2.1 EMERGENT REPRESENTATIONAL STRUCTURE

  • Prior work documents representational structure but debates whether it is coherent and functionally meaningful; this paper focuses on usefulness beyond the setting where representations were constructed.The target application is general abstract reasoning.Source: §2.1 · p. 2

2.2 BONGARD PROBLEMS

  • Bongard problems require inferring a rule from positive and negative examples; this study instead applies nearest-centroid discrimination directly to pretrained representations.LOGO uses procedurally generated geometric images, while OpenWorld uses natural images and open-vocabulary concepts.Source: §2.2 · p. 2

2.3 ARC-AGI 1&2 AND ARC-GEN

  • ARC tasks apply a rule inferred from a few grid input-output examples, while prior approaches include executable or latent programs, specialized predictors, fine-tuning, and test-time optimization.The cited compression-based transductive strategy avoids pretraining but remains ARC-specific and requires task-specific optimization at test time.Source: §2.3 · p. 2
  • The study examines whether frozen representations reflect within-task transformations and guide exact-output retrieval, separating relational reasoning from output generation.The authors do not claim an ARC solution and leave adding a generative component to future work.Source: §2.3 · pp. 2–3
  • Figure 1 depicts support-set centroids and nearest-centroid assignment, then compares best observed accuracy across encoders, pixel controls, resolutions, and Bongard benchmarks.Its reference lines mark published results and chance; the caption does not provide the plotted accuracy values.Source: Figure 1 · p. 3

3 METHODS

The methods evaluate frozen visual representations with fixed geometric readouts and pixel controls. They use centroid classification for Bongard tasks and demonstration-derived intuition vectors for ARC retrieval and task identification.

3.1 REPRESENTATIONS AND CONTROLS

  • The study evaluates frozen DINOv3 and MAE encoders at 224 and 480 pixels without fitting model parameters on benchmark data.It records class-token representations after each of 24 transformer blocks and uses fixed Gaussian pixel projections as a low-level control.Source: §3.1 · pp. 3–4

3.2 BONGARD CONCEPT DISCRIMINATION

  • For each Bongard problem, six images per class form support centroids and one held-out query per class is assigned to its nearer centroid.The reported layer is selected by the highest benchmark-level mean accuracy for each encoder, resolution, and benchmark.Source: §3.2 · p. 4

3.3 ARC INTUITION VECTORS AND OUTPUT RETRIEVAL

  • An ARC intuition vector is the mean layerwise difference between output and input embeddings across a task’s K demonstration pairs.Alignment is assessed against a held-out pair’s embedding difference, with cross-task vectors and shuffled answer grids as controls.Source: §3.3 · p. 4
  • ARC retrieval transports the test-input representation by the intuition vector and ranks candidate outputs by Euclidean distance, against untranslated-query retrieval.For ARC-GEN, 79,400 held-out generated pairs from 794 tasks are retrieved from a shared pool of 368,442 unique outputs, using layers selected on original ARC evaluation pairs.Source: §3.3 · p. 4
  • The population-level ARC-GEN analysis assigns 397,000 generated task-instance intuition vectors from 794 tasks to nearest task centroids under leave-one-out evaluation.Task classification uses the same ARC-selected layers as retrieval.Source: §3.3 · p. 4

4 RESULTS

Frozen visual representations support abstract concept discrimination and encode task-specific input-output transformations. Moving a query along a support-derived intuition vector improves exact-output retrieval, while task identity and exact-output retrieval remain distinct abilities.

4.1 EMBEDDING-SPACE NEIGHBORHOODS DISCRIMINATE ABSTRACT CONCEPTS

  • 64.2% on Bongard-LOGO and 60.5% on Bongard-OpenWorld were the strongest DINOv3 results, respectively 3.0 and 3.5 points below the published references.At 224 px, the strongest results were observed at layers 23 and 21, respectively.Source: §4.1 · pp. 4–5

4.2 INTUITION VECTORS ALIGN ACROSS EXAMPLES WITHIN TASKS

  • Matched intuition and test vectors formed angles of 29.4°–56.5° across ARC settings, while cross-task angles remained near orthogonal at 85.4°–89.1°.Deeper layers increasingly distinguished correct spatial organization from cell-shuffled answers that preserved grid dimensions and color counts.Source: Figure 2 · p. 5, §4.2 · pp. 6–7

4.3 INTUITION VECTOR TRANSPORT GUIDES OUTPUT RETRIEVAL

  • 70.7% top-1 accuracy on ARC-2 evaluation followed intuition-vector transport, a 14.4-point gain over the original query; ARC-1 rose from 32.0% to 63.2%.At the best observed DINOv3-224 layer, each dataset's outputs competed in a same-dataset candidate pool.Source: Figure 3 · p. 6, §4.3 · p. 7
  • 11.6–21.5-point top-1 gains on ARC-GEN with encoder transport exceeded the approximately 4.4-point pixel gain across 79,400 queries from 794 tasks.Encoder transport reduced median query-answer distance to 53%–71% of its original value, while pixel transport left approximately 103%.Source: §4.3 · p. 8

4.4 PIXEL PROJECTIONS AND INTUITION VECTORS SUPPORT DIFFERENT TASK REGIMES

  • Encoder-space transport gains often peaked in small-to-moderate-change tasks and exceeded pixel gains across the change range, while pixel advantage increased with newly introduced colors.Encoder gains eventually declined for the most extensive transformations; pixel retrieval was most competitive with substantial low-level overlap or salient palette changes.Source: Figure 4 · p. 7, §4.4 · pp. 8–9

4.5 TASK-LEVEL IDENTIFIABILITY DISSOCIATES FROM EXACT-OUTPUT RETRIEVAL

  • 73.4%–87.2% leave-one-out task-classification accuracy for pretrained representations exceeded pixels at 43.9%, but task identification and exact-output retrieval were not equivalent.In pixel space, only 10 tasks exceeded 90% on both classification and transported retrieval, versus 31.5 expected under independence; precise instance-level execution remains a limitation.Source: Figure 5 · p. 8, §4.5 · p. 9

5 CONCLUSION

Frozen, off-the-shelf visual representations support abstract reasoning through simple geometric operations, but representing a transformation and generating its exact consequence appear to be distinct capacities. The paper also bounds its evidence: local-operation analyses are exploratory, and generality to other architectures, model sizes, and domains remains open.

5 CONCLUSION

  • Frozen visual representations support abstract reasoning through simple geometric operations, beyond pixel similarity.The conclusion describes intuition vectors as an operational task-level signal, not merely a decodable one.Source: §5 · p. 9
  • Representing a transformation and executing its instance-specific consequence appear to be distinct capacities.The paper does not claim an ARC solution and leaves adding a generative component to future work.Source: §5 · p. 9

5.1 LIMITATIONS

  • Single-cell analyses are exploratory and limited to DINOv3 at 480 px; generality to other architectures, model sizes, and domains remains future work.Whether local context dependence occurs in other encoders, and how it relates to stable task-level signatures, remains open.Source: §5.1 · p. 10

A.2 NEAR-MISS ARC 1&2 RETRIEVAL

The near-miss analysis tests whether intuition-vector transport helps distinguish a true ARC answer from recolored alternatives, using DINOv3 and pixel controls under specified corruption and sampling conditions.

A.2 NEAR-MISS ARC 1&2 RETRIEVAL

  • The candidate pool contains the true output and 99 independently recolored variants at five corruption fractions, from 0.01 to 0.60.Cells are sampled without replacement and replaced uniformly by one of the other nine ARC colors; corruption precedes rendering and leaves padding unchanged.Source: §A.2 · p. 13
  • The analysis evaluates DINOv3 at 480 px on layers 6, 12, 17, and 22, alongside pixel controls averaged over ten projection seeds.MAE and 224-px renderings are not evaluated in this setting.Source: §A.2 · p. 13
  • All test queries are included regardless of whole-pool retrieval success, and error bars capture query-level sampling variation rather than task-clustered variation.Pixel uncertainty combines binomial standard error with the SEM across projection seeds.Source: §A.2 · p. 13

A.3.1 SCOPE, CONSTRUCTION, AND CONTROLS

The supplementary analyses describe how synthetic-grid transformations and contextual compositions are constructed and controlled, alongside figures showing benchmark performance and ARC vector alignment and retrieval across settings.

A.3.1 SCOPE, CONSTRUCTION, AND CONTROLS

  • Synthetic-grid analyses use frozen DINOv3-Large/16 at 480 px, a 30 × 30 canvas, and layer 17 unless stated otherwise; MAE and 224-px renderings are not evaluated.Translation, recoloring, and addition vectors are endpoint-embedding differences; composition is assessed by cosine similarity and relative residual.Source: §A.3.1 · p. 13
  • The composition analysis tests 458,496 joint transformations per layer, with a shuffled control permuting displacement and recoloring assignments while preserving both marginals.The control averages ten permutations; displacement and recoloring primitives are averaged over specified positions and colors.Source: §A.3.1 · p. 13
  • Contextual analyses use nine interior anchors and 32 layouts per anchor, with addition-transfer and shared-context composition tests spanning specified cell occupancies and color combinations.The addition-transfer analysis tests 23,328 transitions per occupancy and 116,640 overall; layerwise analyses use 16 layouts per anchor.Source: §A.3.1 · p. 13
  • Contextual estimates average colors within anchor-layout units before spatial-unit aggregation, with 95% intervals from 2,000 spatial-unit bootstrap resamples.Position decoding uses grouped four-fold cross-validation, keeping color variants of a layout in the same fold.Source: §A.3.1 · p. 13
  • Figure S1 compares DINOv3, matched-size MAE, and pixel controls across Bongard subgroups; OpenWorld’s overlapping subgroup partitions should not be averaged together.The figure’s selected encoder layer is a descriptive dataset-level maximum on evaluation data, and its intervals omit selection uncertainty.Source: Figure S1 · p. 14
  • Figure S2 plots nearest-centroid accuracy across 24 transformer blocks, while Figures S3–S4 show ARC intuition–test angles and Figures S5–S7 and S8 describe pixel, generated-instance, and near-miss analyses.Figure S3 distinguishes matched-task, cross-task, and cell-shuffled controls; Figure S4 reports wider evaluation uncertainty for its smaller ARC-2 evaluation set.Source: Figure S2 · p. 15, Figure S3 · p. 16, Figure S4 · p. 16
  • Figure S6 compares intuition-transported and unperturbed retrieval across ARC training sets, with larger task counts than evaluation sets and no layer, encoder, or resolution selected from these curves.Figure S7’s ARC-GEN retrieval and classification diagnostics use 100 queries per task and a 368,442-grid candidate pool.Source: Figure S6 · p. 17, Figure S7 · p. 18

A.3.2 CONTEXT-DEPENDENT COMPOSITIONALITY OF SINGLE-CELL TRANSFORMATIONS

On synthetic grids, translation and recoloring have separable, approximately additive geometry in empty contexts, but individual transformation vectors change with context. Measuring the primitives in the same context preserves their near-additivity.

A.3.2 CONTEXT-DEPENDENT COMPOSITIONALITY OF SINGLE-CELL TRANSFORMATIONS

  • Opposite unit translations are strongly opposed at layer 17, while orthogonal translations and translation–recoloring pairs are near orthogonal.Opposite-direction cosines are −0.90 for up–down and −0.85 for left–right; recoloring directions are mutually aligned to varying degrees.Source: §A.3.2 · p. 19
  • Across 458,496 balanced empty-grid transformations per layer, summed translation and recoloring primitives achieve cosine 0.948 and residual 0.298 at layer 17.In early layers, matched predictions have mean cosine near 0.999 and relative residual 0.03–0.05; shuffled assignments have near-zero cosine and residuals of approximately 1.4–2.6.Source: §A.3.2 · p. 19
  • Context reduces transfer of individual transformations from empty grids, especially for matched colors; at five context cells, matched-color translation cosine is 0.239 and matched-endpoint recoloring cosine is 0.528.Context sensitivity increases after approximately layer 12 in the five-cell layerwise analysis.Source: §A.3.2 · p. 19
  • At layer 17, joint transformations measured from the same contextual source remain nearly additive, with cosine decreasing only from 0.9984 to 0.9977 as context grows from one to five cells.Their relative residual remains 0.049–0.053, while empty-grid primitives at five cells fall to cosine 0.646 and residual 1.166.Source: §A.3.2 · p. 19

A.3.3 CONTEXT DEPENDENCE OF CELL ADDITION

Cell-addition vectors depend strongly on their source context, and combining them does not yield magnitude-additive transitions. Despite this, the added cell’s location remains linearly decodable.

A.3.3 CONTEXT DEPENDENCE OF CELL ADDITION

  • The experiments cross nine anchor positions, contexts with one to five cells, and all ordered foreground-color pairs.The authors describe context as a boundary of primitive transfer and say directly linking this controlled setting to individual ARC tasks requires further experiments.Source: §A.3.3 · pp. 19, 21
  • At layer 17 with five context cells, addition-vector cosine is 0.044 for shared colors versus 0.610 for different colors.The context effect appears after the first block and persists thereafter.Source: §A.3.3 · p. 19, Figure S10 · p. 21
  • At layer 17 with five additions, context-conditioned vectors sum to a joint transition with cosine 0.905 but relative residual 2.249.Across layers, cosine declines to approximately 0.85 while residual remains near 2.5; isolated additions perform worse, with cosine 0.571 and residual 5.060 at k = 5.Source: §A.3.3 · pp. 19, 21, Figure S10 · p. 21
  • 97.8% accuracy decodes the added cell’s nine-way location from its context-dependent vector, versus 11.1% chance.A wrong-location control remains comparatively close to the joint transition, indicating aggregate measures are dominated by occupancy- and color-related components.Source: §A.3.3 · p. 21, Figure S10 · p. 21, Figure S11 · p. 22

Related papers

Loading 2609.32619v1…