PaperlayerView exact revision on arXiv

Source-linked AI summary

Canonical locks that encode part-whole hierarchies

Rajat Modi, Yogesh Singh Rawat

arXiv:2609.26046v1cs.AI

TL;DR

Part–whole relationships can change with input and may be unknown, making their representation in neural nets a challenge. The paper presents canonical locks as a geometric primitive, reports single-sample generalization with competitive accuracy, and describes scope extending to rotations in 90-degree increments.

  • Problem

    Part relationships can change dynamically with input, and their structure may not be known in advance.

    Source: §1 · p. 1

  • Method

    Canonical locks are a geometric primitive for encoding hierarchies, with latent vectors storing information through their relative magnitudes and phases.

    Source: §3.3 · p. 5, §9 · p. 22

  • Results

    The authors report training with one sample without pretrained weights and generalizing to test images with competitive accuracies.

    Source: §9 · p. 22

  • Takeaways & Limitations

    The reported generalization extends to rotational groups in increments of 90 degrees.

    Source: §9 · p. 22

    The 4 × 4 transformation-matrix view is low-dimensional, while a d × d matrix for d ≥ 4 is difficult to visualize.

    Source: §6.5 · p. 13

Abstract

from arXiv · show

One of the challenges in representational learning is how to encode part-whole hierarchies in a neural net. Prior works rely on flattening tree-like structures into string-like sequences and training a sequence-to-sequence model via autoregression. While such a representation works for parse-trees in NLP, it is not entirely clear how to make it work for images. Thus, we propose a geometric primitive called canonical locks. The key idea is that parts/wholes can be modelled as higher-dimensional vectors ($d \geq 4$), and information can be encoded in their relative phase differences. Inductively, the net consists of positionally-bound bottom-up and top-down neural fields, which drive each other to achieve a state of thermal equilibrium. Additionally, we show the existence of a few symmetrical configurations in the net. The computational iterations taken to break these symmetries depend on the angle between parts/wholes arranged on a disk (or more precisely a ring) in higher dimensions. It also appears to have connections to the psychological phenomenon of mental rotation.

1. The Problem of Part-Whole Hierarchies

The section asks how neural nets can represent part–whole hierarchies whose structure changes with input and is not known in advance. It presents second-generation APMs and frames this paper as an engineering-oriented account of GLOM’s ideas.

1. The Problem of Part-Whole Hierarchies

  • Real-world structures can be modeled as graphs whose part relationships change with input and may need to be inferred.A face’s part graph can change when the face is rotated or occluded.Source: §1 · p. 1
  • Earlier APMs relied on test-time training, collapsed on purely supervised tasks, and failed at dense segmentation.Attempts to address collapse with regularization, local contrastive learning, or masked-region reconstruction were slow to converge.Source: §1 · p. 2
  • Earlier APM work reduced training time through teacher-based codistillation but did not address part–whole hierarchies and still relied on a teacher trained at scale.The section also says the assumption that transformer layers naturally encode hierarchies proved incorrect.Source: §1 · p. 2
  • The paper proposes second-generation APMs that aim to encode part–whole hierarchies and perform dense segmentation, with features robust to occlusion and masking.The authors report learning from one training sample without pre-training and generalizing to a test set with competitive accuracies.Source: §1 · p. 2
  • This paper interprets GLOM’s philosophy as engineering constraints and establishes a vector-space terminology for thinking about parts, wholes, compositions, and decompositions.The authors position the paper as groundwork for later implementation on parallel distributed memories.Source: §1 · p. 2

2. Two ways of encoding part-whole hierarchies

The section contrasts encoding part–whole relationships inside a neural net with encoding them outside it in latent representations. It identifies failure modes for internal routing and presents canonical locks after describing limitations of the external scheme.

2.1. Encoding hierarchy inside the net

  • Internal encoding uses attention weights between parts and wholes to form an adjacency-like hierarchy, but faces routing, changing-graph, and representation-collapse problems.Routing requires N^2 part-to-whole assignments, and stacking layers can worsen the noise in routing information.Source: §2.1 · p. 3
  • Large-scale pretraining is described as a way to drown out system noise, while the paper argues for constraining neural nets by geometric laws.The section contrasts a scaling-oriented view with a geometrical view that treats canonical frames, mental rotations, hierarchies, and equivariance as foundational.Source: §2.1 · p. 3

2.2. Encoding hierarchy outside the net

  • The outside-net scheme computes location-specific embeddings in parallel, reusing the same neural hardware as hierarchy depth increases.A location-specific query performs the binding.Source: §2.2 · p. 3
  • The outside-net scheme had been proposed and inductively realized, but collapses when multiple hierarchy levels are considered, motivating canonical locks.The section frames geometry as a way to combine scaling with faster search.Source: §2.2 · p. 3, §2.1 · p. 3

3. Canonical locks for learning machines

The section introduces canonical locks as fixed relative angles between part and whole vectors, then considers static and dynamical systems for representing them. It also discusses how information may be stored in vector magnitude and phase.

3.1. A toy problem

  • The toy model places one part and whole vector on separate disks on a unit hypersphere, with continuous angular states and chirality ignored.The part and whole can each occupy any of infinitely many states.Source: §3.1 · p. 4
  • A canonical lock fixes a particular relative angle between part and whole even as their vectors rotate to different disk positions.Source: §3.1 · p. 4
  • The setup asks how to choose an unsupervised relative angle, lock multiple part–whole pairs, and keep locks invariant while representations rotate equivariantly.Source: §3.1 · p. 4

3.2. Static vs Dynamical systems

  • Static systems settle to fixed part and whole vectors, whereas dynamical systems let both rotate while preserving their relative angle.Source: §3.2 · p. 5
  • Static systems may lose downstream performance through overthinking, while dynamical systems appear more robust despite continued rotation.The best downstream accuracy need not occur at the fixed point.Source: §3.2 · p. 5

3.3. Information storage in dynamical systems

  • The hierarchy-generating final layer can use both relative magnitude and phase to encode information, and phase can carry more information than scalar magnitude.Source: §3.3 · p. 5

4. The Part-Whole Symmetry

The section examines difficult symmetric arrangements of multiple parts around one whole and considers how their geometry affects canonical-lock formation. It also notes that perfect locks may be difficult in high-dimensional spaces.

4. The Part-Whole Symmetry

  • The section tests difficult symmetric arrangements with one whole and three parts, assuming the parts belong to the same object.Source: §4 · pp. 5–6
  • For perpendicular parts at 90°, 180°, and 270° to a fixed whole, the text considers convergence to the central part or a 2/3 preference for either outer part.Source: §4 · p. 6
  • If the whole and parts balance around the origin, the part vector could become zero, leaving its direction arbitrary and canonical locking useless.Source: §4 · p. 6
  • When all parts coincide with the whole, their identical vectors provide no relative angle and encode no meaningful hierarchy.Source: §4 · p. 6
  • Perfect locking may be difficult in dimensions d ≥ 4 because the volume of the space is high.Source: §4 · p. 6
  • The authors motivate symmetry breaking as a route to self-organized structure, drawing an analogy to biological development.Source: §4 · p. 6

5. The Part-Whole Asymmetry

The experiments examine symmetry breaking from anti-symmetric and collapsed part–whole arrangements, then relate the iterations required to break symmetry to angular configurations. The section also discusses how the machine might select among canonical axes.

5.1. Breaking the anti-symmetry

  • With anti-symmetric inputs, the net brings mean cosine similarity to zero in about 30 iterations, yielding part outputs orthogonal to the fixed whole vector.The setup has one whole and three parts, including one part at 180° to the whole.Source: §5.1 · p. 6
  • The section notes that random vectors in higher dimensions may be orthogonal, while arguing that a machine can be programmed to learn explicit geometric constraints.It presents this as a response to interpreting the canonical lock as a random coincidence.Source: §5.1 · p. 6

5.2. Breaking the collapsed symmetry

  • With collapsed inputs, the net also produces part outputs orthogonal to the whole, but takes about 60 iterations rather than 30.The section suggests the collapsed case takes longer because its part vectors initially have no relative structure.Source: §5.2 · pp. 6–7

5.3. A Linear Model for Predicting the Time It Takes to Break Symmetry

  • At 225°, the model predicts symmetry breaking at about 22.5 iterations, where cosine loss approaches zero; later zero crossings occur near iterations 40 and 60.The section associates the 40-iteration crossing with a 120° center of gravity and the 60-iteration crossing with collapsed symmetry.Source: §5.3 · p. 7, Figure 11 · p. 7
  • The machine is said to prefer the axis with the greatest rotational degree of freedom, but canonical-frame numbers alone do not encode that degree of freedom.The section proposes attaching a learnable token so the machine can select an axis without brute-forcing all plausible axes.Source: §5.3 · pp. 7–8

6. Existence of Other Symmetries

This section examines how part–whole representations encode hierarchy through relative angles, synchrony, higher-dimensional structures, and lattice configurations, including challenges in forming and reading them. It also describes replicated whole vectors, paired bottom-up and top-down nets, and a bottom-up transformation intended to avoid mean-field representational collapse.

6. Existence of Other Symmetries

  • Other configurations oscillate among anti-symmetry, collapsed symmetry, and tri-fold symmetry before equilibrium, even with random inputs and weights across multiple seeds.The text treats these three as fundamental symmetries and notes that other manifolds could yield additional higher-dimensional symmetries.Source: §6 · p. 8

6.1. Synchrony vs Asynchrony across levels of the part-whole hierarchy

  • Synchronized part and whole vectors have zero relative angle and encode no useful information; nonzero angles between levels encode information.The examples contrast collapsed cross-level vectors with parts aligned among themselves and wholes aligned among themselves, but at a nonzero relative angle.Source: §6.1 · p. 8

6.2. Relaxing the orthogonality of coordinate frames

  • A single canonical lock can propagate across a hierarchy, while relaxing coordinate-frame orthogonality models relative angles between frame normals without body-part vectors.The proposed reduction removes the intermediate part vectors; the text says a part’s lock to its whole can yield locks across the hierarchy.Source: §6.2 · p. 9

6.3. Preserving the orthogonality of relaxed coordinate frames

  • In higher dimensions, independent random directions become nearly orthogonal, so driving internal canonical locks to 60° may require many computational iterations.The text also notes that pixels can be represented as vectors, but how they form objects depends on granularity.Source: §6.3 · pp. 9–10

6.4. The part-whole grammar

  • The part–whole grammar represents a problem’s solution as a lattice configuration, but the text says proving that a learned configuration implements an algorithm remains difficult.The proposed lattice is encoded in the net’s weights; the text distinguishes known algorithms, pattern-recognition problems, and problems whose algorithms provably do not exist.Source: §6.4 · pp. 11–12

6.5. Reading the wholes of a part in the hierarchy

  • Reading a part’s whole is framed as transforming its vector upward, but capsule-style routing requires N × M decisions and its transformation-matrix representation has limitations.The text criticizes 4 × 4 matrices for restricting representation quality and higher-dimensional matrices for being difficult to visualize; even reduced matrices may need costly alignment to interpret.Source: §6.5 · pp. 12–13
  • Replicating each whole vector across its parts replaces transformation matrices and routing; vector similarity supports post-recognition clustering without prior K, bounding boxes, or Hungarian matching.For M whole vectors, the passage proposes assigning membership to K clusters with k-means.Source: §6.5 · p. 13

6.6. Brain as a generative model

  • Separate bottom-up and top-down neural nets synchronize, removing the assumption that the network can look at an input image only once.Source: §6.6 · p. 13

6.7. Issues with the mean field approximation

  • Mean-field averaging collapses the hierarchy; a shared bottom-up net Wf must differ from identity, and nonlinear layers with random, nonidentical weights can ensure this.The authors introduce Asynchronous Perception Machines with canonical locks as an upgrade over (Modi & Rawat, 2024).Source: §6.7 · p. 14, Figure 23 · p. 14

7. Asynchronous Perception Machines

This section describes APM’s part–whole image representations and its label-free inner agreement process alongside outer-loop training. It examines how wiring and spatial weight sharing affect convergence, and outlines related symmetry, amplification, memory, and information-sharing tradeoffs.

7. Asynchronous Perception Machines

  • A shared encoder maps an image into vectors spanning the part–whole hierarchy, and bottom-up and top-down nets connect successive levels before a decoder predicts the task output.The architecture describes H × W × L vectors of d dimensions for an image of height H, width W, and L hierarchy levels.Source: §7 · p. 14

7.1. The Weight Update rule

  • The inner loop runs agreement for T iterations, updating only top-down weights without task labels; the outer loop updates encoder and bottom-up weights using task-specific loss.The paper also notes that the outer loop could use a self-supervised task such as masked image reconstruction.Source: §7.1 · pp. 14–15

7.2. The agreement procedure

  • Parts and wholes exchange local-neighborhood information; part votes are projected onto whole vectors, which combine those votes with neighboring wholes while top-down weights update.During cross-level agreement, bottom-up weights are frozen.Source: §7.2 · p. 15

7.3. The Slurp Update

  • The slurp update moves a part vector incrementally along a unit-sphere geodesic toward its top-down output, avoiding a direct replacement that would eliminate further top-down learning.Because bottom-up gradients weaken across layers, the paper recommends more iterations for equation 1a than for equation 4.Source: §7.3 · p. 17

7.4. Promoting Perfect Agreement

  • Deep Weight Alignment constrains layer-wise symmetric bottom-up and top-down nets to promote perfect agreement, and its inner-loop losses require no labels.The authors report that this setup works better than training both nets together, while acknowledging its biological implausibility.Source: §7.4 · p. 18

7.5. Hinton diagrams for synchronization

  • Across varied initializations and more than a million iterations, agreement produced signed symmetry and drove top-down weights toward the bottom-up inverse, though magnitudes differed.With random vectors, the procedure converged to their mean; with structured image data, islands of agreement emerged.Source: Figure 30 · p. 19, §7.5 · p. 19

7.6. Alternating signed symmetry and amplification circuits

  • Alternating weight signs and weaker top-down weights motivate the authors’ standing-wave account, in which top-down and bottom-up resonance may strengthen decoded crests.To prevent crests from growing without bound and disrupting convergence, the paper clamps weights to a higher-dimensional spherical manifold.Source: §7.6 · p. 20
  • Alternating backpropagation between the two sides makes top-down weights mirror bottom-up weights on average; because bottom-up weights are larger, the authors measure their Lyapunov energy and amplify top-down weights.The top-down weights are amplified by a fixed factor to compensate for their weaker magnitude.Source: §7.6 · p. 20
  • The rate at which learned representations converge depends on the architecture’s wiring pattern, despite the universal approximation theorem’s abstraction of architectures as a function.The authors base this observation on their experience with different architectures.Source: §7.6 · p. 20

7.7. Replication of top-down and bottom-up nets

  • Replicating both nets across spatial locations enables high-bandwidth information exchange through weight sharing, but quickly exhausts memory on a single GPU.The passage contrasts this exchange with knowledge distillation (Hinton, 2021; 2022).Source: §7.7 · p. 20
  • Keeping one copy of each net and batching location queries saves memory on a single device (Modi & Rawat, 2024); across multiple GPUs, this approach can combine memory efficiency with weight-sharing benefits.The multi-device arrangement keeps one copy across devices and parallelizes within each device by feeding different location queries.Source: §7.7 · pp. 20–21

8. The visual hierarchical parsing

The section defines a five-level face hierarchy and describes how the network predicts and reads part-whole relationships without assigning parts to wholes. Unlike autoregressive parsing, it predicts the full tree in one forward pass and refines it through agreement iterations.

8. The visual hierarchical parsing

  • The face hierarchy groups 11 finest-level classes into progressively coarser levels, while existing datasets are described as reaching only three levels despite GLOM requiring five.The five levels contain 11, 7, 5, 3, and 2 classes, with hair absorbed into face at the coarsest level.Source: §8 · p. 21
  • Given face images with varying cluster sizes and locations, the network predicts a hierarchy that generalizes to unseen images.The section describes backward parsing by recursively examining the representation at the next finer level.Source: §8 · p. 21
  • Part-whole parsing emerges from an agreement procedure: placing subpart, part, and object vectors in one column makes parent-child relationships directly readable without assignment.This arrangement is intended to speed parsing.Source: §8 · p. 22
  • The entire parse tree is predicted in one forward pass and refined through bottom-up and top-down agreement, rather than generated token by token through autoregression.In the described seq2seq approach, decoded activity is fed back at each timestep, requiring another network run.Source: §8 · p. 22

9. Conclusion

The paper presents canonical locks as a geometric primitive for encoding part-whole hierarchies. It reports one-sample training without pretrained initialization and generalization to test images, and proposes compositional learning as a hypothesis for this behavior.

9. Conclusion

  • Canonical locks are presented as a geometric primitive for encoding part-whole hierarchies, with one-sample training without pretrained initialization and competitive test-set accuracy, including generalization to rotations in 90-degree increments.The conclusion also reports generalization to rotational groups that increment in multiples of 90 degrees.Source: §9 · p. 22
  • The authors hypothesize that overfitting on one sample produces compositional learning that combines parts into wholes dynamically, while constrained weights reduce the search space and speed learning.The passage frames these claims as a running hypothesis.Source: §9 · p. 22
  • The paper proposes exploring higher-bandwidth, lower-parameter learning machines and suggests APM/s might meta-learn multiple hierarchy levels, architectures, algorithms, optimizers, and pretraining tasks.It also mentions meta-learning from the meta learner itself.Source: §9 · p. 22

A. Decoding the part-whole hierarchy

The decoder reads each pixel’s predicted class at each hierarchy level through that level’s name table. Its five entries form a part-whole path directly from classifier outputs, with no post-processing to cluster features, match a tree, or solve an assignment.

A. Decoding the part-whole hierarchy

  • At each pixel, the APM predicts a class at each of five levels, with class counts 11, 7, 5, 3, and 2; each level has its own name table for decoding.Source: A. Decoding the part-whole hierarchy · p. 24
  • The five decoded class names give each pixel’s part-whole path directly, without clustering features, matching a tree, or solving an assignment.The passage says multiple hierarchies could be decoded together by aligning an open-world text model to locations.Source: A. Decoding the part-whole hierarchy · p. 24
  • Across levels, pixels may change part-whole meaning, retain a meaning such as hair through L4, or correspond to a class unused in the face, such as inner mouth.A pixel can be a part at one level and a whole at another; the level determines which interpretation is read.Source: A. Decoding the part-whole hierarchy · p. 24
  • Sample level accuracies were 0.919 (L1), 0.900 (L2), 0.916 (L3), 0.969 (L4), and 0.982 (L5).The values are reported for one sample.Source: A. Decoding the part-whole hierarchy · p. 24
Loading 2609.26046v1…