Source-linked AI summary
Emergent Object Binding Has a Finite Spatial Horizon
Mayank Singal
TL;DR
High IsSameObject accuracy from frozen patch embeddings does not reveal how the signal is structured. The paper probes binding across patch distances while controlling relevant confounds, finding a finite spatial horizon and associated behaviors. It also notes that its depth observation is preliminary and that the mechanism behind the decay remains unresolved.
Problem
High IsSameObject accuracy shows that same-object relations are decodable, but does not establish whether binding is equally reliable across distances and object arrangements.
Source: §1 · pp. 1–2Method
The authors probe frozen patch embeddings across pair distances, control confounds such as object size through matching, and compare probe families and datasets.
Source: §1 · p. 2Results
Binding probability decays monotonically with distance to a nonzero floor, following an exponential with finite length scale across object sizes, three probe families, and ADE20K and COCO.
Source: §1 · p. 2Takeaways & Limitations
The horizon account explains weaker binding on large objects, less reliable separation of same-class instances, part–whole grouping, and no occlusion effect after size control.
Source: §1 · p. 2The spatial mechanism is not identified: candidate explanations are not separated by the evidence, and the authors make no mechanism claim.
Source: §7 · p. 13
Abstract
from arXiv · showhide
Pretrained Vision Transformers encode whether two image patches belong to the same object. This IsSameObject signal is decodable from frozen patch embeddings at high accuracy, which suggests that object binding emerges from self-supervised pretraining alone. We show that this single accuracy number hides the structure of the signal. Binding is local: the probability that two patches of the same object are decoded as bound falls off monotonically with the distance between them and levels off at a nonzero floor, a falloff well described by an exponential with a finite length scale. This decay holds across object sizes, across three families of probe, on both ADE20K and COCO, and across DINO and CLIP backbones, which indicates that it is a property of the representation rather than of the decoder. Reading binding as local spatial coherence with a finite range accounts for a set of behaviors that the aggregate score leaves unexplained: binding weakens on large objects, separates distinct objects of the same class less reliably than objects of different classes, and groups object parts with their wholes. It is, by contrast, unaffected by occlusion once object size is controlled. We map each behavior with confounds controlled. As a preliminary observation, the horizon and its floor are organized at different depths in DINOv2 and DINOv3, which we report as suggestive given the small number of layers probed and the confound between the two models.
1 Introduction
The paper looks beyond high IsSameObject accuracy to characterize what the signal represents. It finds distance-limited binding and shows how that pattern relates to object size, class, parts, and occlusion.
1 Introduction
- A high aggregate accuracy establishes that the signal exists but can hide distance-dependent behavior because random same-object pairs are usually nearby and scores average across object properties.The paper frames its contribution as analyzing the probed signal rather than introducing the probing framework.Source: §1 · pp. 1–3
- The horizon accounts for weaker binding on large objects, less reliable separation of same-class objects, part–whole grouping, and no occlusion effect after controlling for object size.The proposed account links large-object weakening to patch pairs that exceed the horizon and the residual floor to class.Source: §1 · p. 2
- The central distance decay is reported for DINOv2 and DINOv3 on ADE20K and COCO, while the different depths of the horizon and floor are presented only as suggestive.The depth observation rests on few probed layers and a two-model comparison.Source: §1 · pp. 2–3
- Binding probability falls monotonically with patch distance to a nonzero floor, following an exponential with a finite length scale across object sizes, three probe families, and ADE20K and COCO.The authors call this finite range the spatial horizon of binding.Source: §1 · p. 2
2 Related Work
Prior work studies what pretrained vision features support and shows that same-object relations can be decoded from frozen patch embeddings. This paper uses that probe to examine the structure of the relation and checks whether findings persist across probe capacities.
2 Related Work
- Self-supervised ViTs support dense-feature tasks, while this paper focuses on whether two patches encode membership in the same object and how that relation is structured.DINO-family features have been reported to expose object boundaries and cluster by object.Source: §2 · pp. 3–4
- Li et al. show that IsSameObject is linearly decodable from frozen ViT patch embeddings, strongest in DINO-family models and weak in masked-autoencoder models.Their work establishes the signal and compares it across pretraining methods; this paper takes the probe as a measurement instrument.Source: §2 · p. 4
- Because expressive probes can fit structure not readily available in the representation, the paper repeats central measurements across three probe capacities and compares a feature-similarity baseline.The authors report which effects remain stable across probes and which do not.Source: §2 · p. 4
3 Method
The method trains probes on frozen Vision Transformer patch embeddings to predict whether pairs belong to the same object instance. It tests alternative probe families, evaluates on annotated datasets, and controls for correlated factors through stratified matching.
3 Method
- A probe maps a pair of frozen patch embeddings to a score trained with binary cross-entropy to predict whether the patches belong to the same object instance.The default is a bilinear form with a symmetrized weight matrix scaled by 1/D.Source: §3 · p. 4
- To test dependence on probe choice, the authors also use a linear probe on elementwise products and a two-layer network on products and absolute differences.These alternatives are compared with the default bilinear probe.Source: §3 · p. 4
- The study uses DINOv2-Large and DINOv3-Large patch tokens, ADE20K instance, class, part, and occlusion annotations, and repeats the distance measurement on COCO.Annotations are mapped to the patch grid by majority vote, with training and evaluation on disjoint images.Source: §3 · p. 4
- Balanced positive and negative instance pairs are sampled per image; results use three seeds, with AUC for a single summary and class-resolved accuracies elsewhere.Balancing avoids a trivial majority-class solution, while class-resolved accuracies avoid reliance on one fixed decision threshold.Source: §3 · p. 5
- Stratified matching bins pairs by a potential confounder and averages effects across bins to control size, distance, or class as appropriate.It controls object size for distance and occlusion analyses, and distance and size for the same-class comparison.Source: §3 · p. 5
4 Binding Has a Finite Spatial Horizon
In pretrained ViTs, same-object binding probability falls with patch distance and approaches a nonzero floor; the decay is well described by an exponential with finite length scale. The section tests whether this structure follows distance across sizes, probes, datasets, and pretraining objectives.
4 Binding Has a Finite Spatial Horizon
- R2 > 0.9; same-object binding falls monotonically with distance toward a nonzero floor, fitting an exponential with finite length scale.Source: §4 · p. 5
- Across object-size bands, binding curves stay close, indicating that distance rather than size organizes the decay.Source: §4 · p. 5, Figure 2 · p. 6, Table 2 · p. 8
- On COCO, binding falls from 0.69 at the nearest bin to 0.44 beyond fifteen patches, with λ≈5.5, c≈0.43, and R2 = 0.97.Source: §4 · p. 6
- On COCO, CLIP has AUC 0.85 and shows the horizon, whereas MAE's AUC 0.54 is near chance and has no binding structure to decay.Source: §4 · pp. 6–7, Table 1 · p. 7
- Across linear, bilinear, and MLP probes in DINOv2 and DINOv3, decay's presence and direction persist, although DINOv3 near-to-far drops vary from 0.19 to 0.44.Source: §4 · pp. 5–6, Figure 3 · p. 7, Table 3 · p. 8
- The authors treat local falloff as expected; the distinctive evidence is a class-tracking floor whose same-class effect exceeds a similarity baseline by about twofold.A thresholded cosine classifier recovers less than half of the effect shown by the trained probe.Source: §4 · pp. 8–9
5 The Horizon Accounts for Where Binding Breaks
The spatial horizon helps explain where binding succeeds and breaks: binding weakens on large objects, favors same-class instances, groups parts with wholes, and is unaffected by occlusion after controlling for size.
Large objects
- Binding accuracy falls monotonically from 97.5% on small objects (2–4 patches) to 61.6% on the largest (64+ patches), consistent with patch pairs exceeding the horizon.The underlying variable is distance: large objects contain more pairs beyond the binding range.Source: §5 · p. 9
Same-class instances
- Distinct same-class objects are rejected as different less often, with a 12–13-point gap that survives matching for distance and object size.The size-matched gap is 12.6 points after matching distance and 13.1 points after matching size; the effect also exceeds a cosine-similarity baseline and reproduces on COCO.Source: §5 · pp. 9–10
Parts and wholes
- A patch on a part is grouped with its parent whole at the same rate as two patches within that whole, and far more than unrelated objects.The part-whole comparison uses ADE20K’s part hierarchy.Source: §5 · p. 10
Occlusion
- After matching for object size, the apparent occlusion advantage falls to about zero in both models, so binding is unaffected by occlusion once size is controlled.The matched effects are 0.3 ± 0.7 in DINOv3 and −1.3 ± 0.5 in DINOv2; matching leaves the same-class effect intact.Source: §5 · p. 10
6 A Preliminary Note on Depth
The horizon appears at different depths in DINOv2 and DINOv3, while the floor and same-class effect co-vary across depth. These observations are preliminary: only four layers per model were probed, and the models differ in architecture.
6 A Preliminary Note on Depth
- The depth pattern is exploratory: four layers per model, unreliable intermediate fits, and architectural differences limit interpretation and support no causal conclusion.DINOv3 adds rotary position embeddings, register tokens, and gated feed-forward blocks, which the comparison cannot disentangle.Source: §6 · p. 10
- DINOv3 forms a clean, narrow horizon early, while DINOv2’s horizon becomes clean late and widens toward the output; intermediate fitted length scales are unreliable.The reported trajectory is based on endpoints and fit quality, not individual fitted values.Source: §6 · p. 10, Figure 4 · p. 11
- The floor and size-matched same-class gap co-vary across four layers per model, with coincident peaks in DINOv3 and adjacent peaks in DINOv2.Their correlation is illustrative, not statistical, given the four-layer samples.Source: §6 · p. 11, Table 5 · p. 12
- In DINOv2, the floor falls from 0.53 to near zero as the length scale grows, while DINOv3 reaches a low floor by mid-network.The authors interpret these observations as consistent with the floor reflecting binding’s semantic component.Source: §6 · pp. 11–12
7 Limitations
The reported horizon and depth patterns have measurement and scope limits, and the paper establishes the spatial law empirically without identifying its cause.
7 Limitations
- Euclidean patch-grid distance approximates but does not equal distance along an object, so the fitted length scale is best read as an upper bound for irregular or elongated shapes.The exact decay for such shapes remains open.Source: §7 · p. 12
- The fitted length scale is sensitive to binning and probe family, so it is reported as a range rather than a precise value.The presence of decay and its independence from size are the robust claims.Source: §7 · p. 12
- The depth analysis uses four layers per model and locates the horizon only coarsely; cross-model checks use one layer and one probe per model.Part-whole, occlusion, and depth analyses are limited to ADE20K and DINO models.Source: §7 · p. 12
- The work establishes the spatial law empirically but does not isolate its cause; candidate explanations remain unresolved, and a causal account would require targeted intervention.The authors identify perturbing positional information as a possible next step.Source: §7 · pp. 12–13
8 Conclusion
The paper finds that object binding in pretrained ViTs is spatially local, with a finite range and a distance-independent component. This structure explains several behaviors hidden by aggregate accuracy, while its mechanism remains an open direction.
8 Conclusion
- The probability that same-object patches are bound decays with distance to a nonzero floor, following an exponential with a finite range.The paper interprets emergent binding as local spatial coherence together with a semantic component.Source: §8 · p. 13
- This account explains why binding weakens on large objects, favors distinct same-class objects, and groups parts with their wholes, while remaining robust to occlusion.The occlusion result holds once object size is controlled.Source: §8 · p. 13
- The structure appears in CLIP with comparable strength and is absent in MAE, where the binding signal is near chance.The conclusion states that the pattern is not specific to the DINO objective.Source: §8 · p. 13
- A causal account and a fuller sweep across layers, probes, and architectures remain future directions.The paper points to intervention on positional information as a way to pursue a causal account.Source: §8 · p. 13
Use of AI Tools
AI-based assistants supported manuscript preparation, coding, and experimental analysis, while the authors retained responsibility for research claims, design decisions, and final interpretations.
Use of AI Tools
- AI-based assistants supported manuscript drafting and editing.Source: Use of AI Tools · p. 13
- The assistants also helped write and debug analysis code and supported experimental analysis.Source: Use of AI Tools · p. 13
- The authors made the research claims and experimental design decisions and retained ownership of final interpretations.Source: Use of AI Tools · p. 13