Source-linked AI summary
Understanding Clustering in Slot Attention via Particle Dynamics
Vasudev Joy, Rajat Rasal, Avinash Kori, Anthea Monod, Ben Glocker
TL;DR
Existing attention dynamics are known to produce token clustering, but learned components in slot attention obscure how much of its clustering comes from the attention mechanism itself. The paper introduces parameter-free simplified slot attention (SSA), relates it to soft k-means, and reports performance close to slot attention on Pascal VOC.
Problem
Learned components in slot attention obscure whether its competitive assignment mechanism alone is sufficient for object-centric segmentation.
Source: §1 · p. 1Method
SSA isolates slot attention’s competitive assignment dynamics in a parameter-free model and relates them to soft k-means clustering.
Source: §3.1 · p. 3, §3.2 · p. 3Results
Takeaways & Limitations
The results suggest that much of slot attention’s clustering can be explained by its dynamics rather than its learned components.
Source: §1 · p. 1, §5 · p. 5The empirical study is restricted to one dataset, a frozen feature backbone, and a reconstruction-based training setup, while the energy function characterizes USSA rather than SSA’s weighted-mean dynamics.
Source: §5 · p. 5
Abstract
from arXiv · showhide
Studying attention through the lens of interacting particle dynamics has shown how token clustering can emerge from the underlying dynamics. We extend this perspective to slot attention, a method for object-centric image segmentation and representation learning in which learned components obscure how much of the clustering behaviour is intrinsic to the attention dynamics. We therefore introduce simplified slot attention (SSA), a parameter-free variant whose dynamics are connected to soft $k$-means clustering and which provides a straightforward mechanistic explanation for the emergence of object-centric representations. On the Pascal VOC dataset, SSA achieves performance comparable to that of slot attention, demonstrating that competitive object-centric segmentation can be achieved without learned neural-network components.
1. Introduction
Prior work shows that attention dynamics can produce emergent token clusters. This paper asks whether slot attention’s competitive assignments alone can account for object-centric segmentation, and reports that a parameter-free variant performs close to slot attention on Pascal VOC.
1. Introduction
- Interacting-particle attention models have been shown to produce emergent token clustering, including analyses that use identity query, key, and value projections.Geshkovski et al. [8] use identity projections to characterize clustering dynamics and analyze associated energy functions.Source: §1 · p. 1
- Slot attention groups image features into a fixed set of latent slots intended to capture scene objects, and its competitive assignments produce segmentation masks.The learned components obscure how much of this clustering is intrinsic to the attention dynamics.Source: §1 · p. 1
- SSA achieves segmentation and reconstruction performance close to slot attention on Pascal VOC, suggesting much of its clustering is explained by dynamics rather than learned components.The paper connects SSA to soft k-means, soft spherical k-means, and an unnormalised energy-based surrogate.Source: §1 · p. 1
2. Preliminaries
The preliminaries define input-feature and slot notation, describe slot attention’s competitive grouping and update pipeline, and contrast its assignment dynamics with soft k-means clustering.
2. Preliminaries
- Inputs are feature vectors X, and S_t denotes the M slots at iteration t.The notation also defines Q, K, and V as projection matrices and w^t_ij as the assignment weight from input j to representation i.Source: §2 · p. 1
2.1. Slot Attention
- Slot attention groups encoder features into M slots using queries from slots and keys and values from fixed encoder features, then updates slots with a GRU and residual MLP.Column-wise softmax makes slots compete to explain each input feature; masks come from assignments or decoder alpha masks.Source: §2.1 · p. 2
2.2. Particle Dynamics
- Cross-attention evolves slots against fixed input features, while soft k-means and soft spherical k-means normalize assignments over clusters so centroids compete for each input.Soft spherical k-means constrains data and centroids to a sphere; the standard k-means limit is recovered as β → ∞.Source: §2.2 · p. 2
3. Simplified Slot Attention
The section derives simplified slot attention (SSA) by isolating competitive assignments, relates its updates to soft clustering methods, and introduces USSA as an energy-based surrogate with distinct clustering behavior.
3.1. Dynamical Model
- SSA retains slot attention’s column-stochastic assignments and competitive grouping, while replacing learned transformations with a parameter-free weighted-mean update.The reduction assumes p = d = d_slots and Q = K = V = I, replacing LayerNorm, the GRU, and residual MLP with the weighted mean.Source: §3.1 · p. 3
3.2. Relationship to Soft Clustering Methods
- SSA and soft k-means share competitive assignments and weighted-mean updates but differ in similarity functions; on the unit sphere, SSA and SSKM assignments coincide for β′ = 2β.SSKM projects each updated centroid onto the unit sphere, whereas SSA does not.Source: §3.2 · p. 3
3.3. Clustering Properties
- USSA is an unnormalised surrogate with an energy whose hard-assignment clustering geometry follows dot-product similarity, but its weighted sum lets slots shrink to zero and die.For fixed slot positions, SSA and USSA induce identical assignments; SSA’s weighted mean prevents this slot collapse.Source: §3.3 · p. 4
4. Experiments
The experiments test whether SSA’s competitive clustering dynamics suffice for object-centric segmentation on Pascal VOC. SSA closely tracks slot attention, while comparisons among variants indicate that both dot-product assignments and weighted-mean updates matter.
Setup
- The evaluation uses frozen DINO features on Pascal VOC, grouping 196 patch features into six slots and reporting segmentation and reconstruction measures.Results are reported for assignment-based and decoder alpha-mask segmentations, with patch reconstruction MSE.Source: §4 · p. 4, Table 1 · p. 5
Results
- SKM tends to under-segment, SSKM produces much poorer segmentations, and USSA progressively collapses toward fewer active slots.These comparisons suggest that both dot-product assignments and the weighted-mean update contribute to SSA’s clustering behaviour.Source: §4 · p. 4
- SSA is best or second-best on every metric except FG-ARI, closely tracking slot attention on Pascal VOC.FG-ARI requires caution because it ignores background and collapsed USSA achieves the highest cluster-based FG-ARI.Source: §4 · p. 4, Table 1 · p. 5
Setup
- Figure 1 shows cluster assignments above decoder alpha masks, with colours denoting predicted slots that are not matched across methods.Additional qualitative examples are provided in Section A.1.Source: Figure 1 · p. 5
5. Discussion
SSA combines soft k-means’ competitive weighted-mean update with dot-product assignments, and both components matter to its clustering behaviour. The analysis is limited by the surrogate energy characterization and a restricted empirical setting.
5. Discussion
- SSA combines soft k-means’ competitive weighted-mean update with dot-product assignments; Euclidean assignments, spherical projection, and weighted sums each yield poorer clustering behaviour.The energy function characterises USSA’s surrogate dynamics, not SSA’s weighted-mean dynamics.Source: §5 · p. 5
- The empirical study is restricted to Pascal VOC, a frozen feature backbone, and reconstruction-based training.The roles of slot-attention components omitted from SSA are discussed heuristically; broader datasets, representations, and tasks remain to be evaluated.Source: §5 · p. 5
A.1. Additional Qualitative Results
Figure 2 provides additional qualitative segmentation examples across all grouping modules in the same format as Figure 1.
A.1. Additional Qualitative Results
- Figure 2 presents additional qualitative segmentation examples across all grouping modules in the same format as Figure 1.Source: Figure 2 · p. 8
A.2. Derivation of USSA Update
The derivation obtains the USSA update by minimizing an objective with respect to assignment weights and slots in turn. An L2 term is added because the unconstrained objective is unbounded below.
A.2. Derivation of USSA Update
- Without a constraint on slot norms, the direct dot-product analogue of the soft k-means objective is unbounded below, so an L2 regularization term is added.Source: §A.2 · p. 9
- Minimizing first with respect to assignment weights and enforcing their normalization recovers the familiar softmax weight update.The quadratic regularization term does not depend on the weights and vanishes during this differentiation.Source: §A.2 · p. 9
- With weights fixed, minimizing with respect to each slot gives a weighted-sum slot position, identified as the USSA update.Source: §A.2 · p. 9
A.3. Fixed point of USSA
The USSA fixed-point analysis reduces the energy to hard slot assignments and characterizes when splitting a cluster lowers it. The splitting criterion is a predominantly negative sum of cross-cluster dot products.
A.3. Fixed point of USSA
- At a USSA fixed point, substituting each slot’s weighted-sum expression into the energy yields an objective whose minimizers are hard assignments.Because the objective is concave in the weights subject to their sum constraint, the minimizers lie at assignment-simplex vertices, with weights equal to 0 or 1.Source: §A.3 · p. 10
- A cluster is better split when the sum of its cross-cluster dot products is predominantly negative.The analysis considers a partition of the data into two disjoint parts.Source: §A.3 · p. 10
A.5. GRU, QKV and LayerNorm
The section offers hypotheses about the roles of the components omitted from simplified slot attention. It relates GRU damping to a partial gradient step, learned Q/K projections to similarity, and LayerNorm to constrained slot growth.
A.5. GRU, QKV and LayerNorm
- Under eq. (11), slots progressively die out as a dominant slot accumulates mass, leaving one active slot by iteration 12.The example uses 10 slots and 1,000 Gaussian-initialized points; bright matrix entries indicate high attention weights.Source: Figure 3 · p. 11
- On DINO features, eq. (11) leaves two slots by the end: one for the cat and one for the background.The displayed segmentations are given by cluster assignment.Source: Figure 4 · p. 11
- The GRU’s learned damping is hypothesized to resemble a gradient-descent step size, making its update closer to a partial gradient step on a bounded energy than SSA’s coordinate descent.This comparison omits the GRU’s reset gate and nonlinear activations, and the hypothesis is not tested empirically.Source: §A.5 · p. 11
- Learned Q/K projections are heuristically viewed as a similarity function that can align dot-product-dissimilar points, while V is hypothesized to transform slots for the downstream task.These are proposed interpretations of the projections’ roles.Source: §A.5 · p. 11
- LayerNorm is hypothesized to relate to spherical k-means by constraining slot growth without fully restricting slots to a unit sphere.The authors do not test this hypothesis empirically.Source: §A.5 · p. 12
A.6. Limitations
The analysis is bounded by an incomplete account of learned components, a narrow experimental setting, and a gap between USSA’s energy and SSA’s updates. Future work proposes energy-based sampling and further study of metastable clusters.
A.6. Limitations
- The precise mathematical roles of the GRU and Q, K, V projections remain only partially characterised.The study uses a single parameter-free simplified slot attention variant.Source: §A.6 · p. 12
- Experiments use one dataset and frozen encoder, and three-seed averages leave generalisation and randomness unquantified.The study uses Pascal VOC 2012 and DINO ViT-B/16; whether parameter-free clustering remains competitive on more demanding tasks is open.Source: §A.6 · p. 12
- The bounded energy characterises USSA’s surrogate dynamics, not SSA’s weighted-mean updates, whose fixed points and convergence remain to be characterised.Source: §A.6 · p. 12
A.7. Future Work
- Future work proposes extending a probabilistic clustering view to slot attention and exploring sampling-based training, potentially addressing the fixed-K assumption.The passage relates this direction to Dirichlet-process mixture inference, which can infer the number of clusters [12], and contrastive divergence.Source: §A.7 · p. 12
- Further work could test whether metastable clusters observed under USSA’s unnormalised dynamics also occur in self-attention.Under USSA, slots grow in magnitude, form metastable clusters, and eventually die out to zero, with each death taking longer.Source: §A.7 · p. 12
Appendix B. Implementation and Experiment Details
The implementation compares segmentation modules on frozen DINO features from Pascal VOC 2012 within a shared training pipeline. It specifies the decoder, training schedule, evaluation metrics, and reporting setup.
Dataset and Preprocessing
- Models group 196 patch tokens from frozen DINO ViT-B/16 features on Pascal VOC 2012, with trainaug for training and val for evaluation.Images are resized to 224×224; training uses random crops and horizontal flips, evaluation uses center crops, and ImageNet channel statistics normalize images.Source: Appendix B · pp. 12–13
Segmentation Modules
- Six segmentation modules map 196 features of dimension 768 to six slots, with the same encoder, decoder, and training pipeline.Slot attention uses three routing iterations; SSA retains weighted-mean aggregation without Q, K, V projections, GRU, LayerNorm, or residual MLP.Source: Appendix B · p. 13
Decoder Architecture
- The token-wise MLP decoder repeats each slot across positions, adds shared learned positional embeddings, and predicts feature reconstructions and mask logits.The decoder outputs a 768-dimensional reconstruction and one scalar mask logit per slot per position.Source: Appendix B · p. 13
Training
- Training uses AdamW for 150 epochs, with a 10,000-step linear warmup followed by exponential learning-rate decay.All segmentation-module variants share the training recipe, and results are averaged over three seeds.Source: Appendix B · p. 13
Evaluation and Metrics
- Evaluation reports FG-ARI, instance- and class-level mBO, and reconstruction MSE, using both module assignments and decoder alpha masks for segmentation.mBO accounts for background and object-boundary fit; instance- and class-level mBO distinguish instance- from semantic-level segmentations.Source: Appendix B · p. 13