PaperlayerView exact revision on arXiv

Source-linked AI summary

How to represent part-whole hierarchies in a neural network

Geoffrey Hinton

arXiv:2102.12627v1cs.CV

TL;DR

The paper asks how a fixed-architecture neural network can parse each image into a different part-whole hierarchy without dynamically allocating neurons. It proposes GLOM, which represents parse-tree nodes as islands of similar vectors and combines this idea with shared neural networks and local attention. The paper concludes that this representation could make parses in vision or language easier to inspect, while noting important limitations in biological efficiency and interpretability.

  • Problem

    A fixed neural network lacks an obvious way to represent image-specific part-whole graphs when neurons and connection weights cannot be dynamically allocated or copied.

  • Method

    GLOM assigns universal capsules to every image location across multiple levels and represents each parse-tree node as an island of similar vectors, using neural transformations and local smoothing.

  • Results

    The paper presents GLOM as an imaginary system that explores how islands of similar vectors can represent image parse trees and unify field-like and structural descriptions.

  • Takeaways & Limitations

    If made to work, GLOM should improve interpretability of transformer-like representations for vision or language by making parse structure visible through vector islands.

  • Takeaways & Limitations

    GLOM requires replicated bottom-up and top-down networks at every location, which is computationally advantageous but potentially wasteful in biological synapses.

Abstract

from arXiv · show

This paper does not describe a working system. Instead, it presents a single idea about representation which allows advances made by several different groups to be combined into an imaginary system called GLOM. The advances include transformers, neural fields, contrastive representation learning, distillation and capsules. GLOM answers the question: How can a neural network with a fixed architecture parse an image into a part-whole hierarchy which has a different structure for each image? The idea is simply to use islands of identical vectors to represent the nodes in the parse tree. If GLOM can be made to work, it should significantly improve the interpretability of the representations produced by transformer-like systems when applied to vision or language

1 Overview of the idea

GLOM proposes representing image part-whole hierarchies with islands of similar embedding vectors in a fixed architecture. Columns combine bottom-up, top-down, temporal, and local same-level interactions so these islands can emerge over time.

  • Architecture: The architecture consists of weight-shared columns, each containing spatially local autoencoders that learn multiple levels of part-whole representation.Adjacent levels communicate through multilayer bottom-up and top-down networks.
  • Interactions: Same-level interactions use local attention-weighted smoothing across columns, while within-column interactions implement part-whole transformations.The architecture shares interaction networks across columns and time steps, though networks may differ between adjacent levels.
  • Update dynamics: Each level updates from four contributions: bottom-up prediction, top-down prediction, the prior embedding, and an attention-weighted average from nearby columns.These contributions are combined as a weighted average at every discrete time step.
  • Core idea: GLOM uses islands of nearly identical vectors at different levels to represent nodes in an image parse tree.Higher-level islands should be larger, allowing image-specific hierarchies without dynamically allocating neurons or preassigning groups for every possible node.
  • Emergence: For static images, repeated-frame processing lets embeddings settle into distinct islands, while agreement regularizers encourage local island formation.The regularizers compare the new embedding with bottom-up and top-down predictions.

2 Introduction

The introduction frames GLOM as a fixed-architecture solution to representing image-specific part-whole hierarchies without dynamically allocating neurons. It combines universal location-specific capsules, neural fields, and attention-weighted averaging around islands of similar vectors.

  • Problem: A fixed neural network must represent different image-specific part-whole graphs despite lacking dynamic neuron allocation and easily copied connection weights.Earlier capsule approaches addressed this by reserving type- and region-specific groups of neurons and routing active capsules into a graph.
  • Relation to capsules: Unlike mixture-based capsules, universal capsules permit distributed, value-coded part identities and can share knowledge between similar parts, but introduce symmetry breaking in routing.The cited capsule models had achieved results on small datasets but had not scaled well to larger ones.
  • Proposal: GLOM explores multiple levels of ubiquitous universal capsules, allowing each image location to participate simultaneously in scene, object, part, and sub-part representations.This is described as a profligate alternative to routing among preassigned capsule types.
  • Mathematical analogy: Attention-weighted local averaging provides a simple transformer-like interaction in which key, query, and value are all the embedding itself, while adaptation occurs in bottom-up and top-down networks.Those networks are shared across locations and time steps, but may differ across hierarchy levels.
  • Neural fields: GLOM uses a hierarchy of neural fields in which a shared decoder combines a representation with location to reconstruct spatially varying values.At the scene level, a top-down network converts a scene vector and image location into an object vector that includes 3-D pose information.
  • Coordinate structure: Top-down neural computation can handle replicated parts and symmetries more easily than table lookup, including shapes with repeated components.The paper contrasts computed part vectors with pointer-following representations of known objects.

3 Some design decisions

The design discussion considers hierarchy depth, spatial granularity, initialization, and local context. It presents choices that trade richer contextual prediction against the clean equivalence properties of purely local bottom-up processing.

  • Hierarchy depth: GLOM assumes a fixed-depth part-whole hierarchy, although people can handle much deeper hierarchies through flexible mappings between entities and levels.The paper suggests about five embedding levels for a face-centered example.
  • Spatial granularity: Embedding locations may range from pixels to larger patches, and the paper assumes unchanged spatial granularity across levels for simplicity rather than as an optimal choice.Changing granularity would require top-down predictions for multiple lower-level locations and bottom-up aggregation across them.
  • Local context: Allowing the bottom-up network to inspect nearby locations can reduce uncertainty, but it removes the property that identical part representations make exactly identical object-level predictions.The pure version preserves that equivalence because its bottom-up prediction depends only on the part representation.
  • Attention: Same-level attention uses a similarity-based local average, with β controlling attention sharpness and potentially increasing as the interpretation settles.The weight is normalized over locations attended to by the current location.
  • Input initialization: A convolutional network can initialize the lowest-level embeddings, while reconstruction-error backpropagation through a shared neural field can refine them.Coarser-scale convolutional inputs may also provide hints for higher-level embeddings.

4 Color and texture

GLOM represents an object’s distributed properties uniformly across all locations it occupies at the object level. This applies to compound color, texture, and shape while preserving part-level differences.

  • Color: An object’s object-level representation can assign every location it occupies the same compound color, such as “pale-green-or-mauve.”Attention at that level favors locations with similar compound colors.
  • Texture: The same uniform object-level representation applies to texture, despite individual texture elements having distinct shapes, poses, and spatial relationships.
  • Shape: GLOM extends this principle to shape: an object can have different parts but the same compound shape at every occupied object-level location.

5 Cluster discovery versus cluster formation

GLOM forms clusters of similar embeddings through interactions rather than discovering them from immutable part votes. Adjacent-level and same-level information progressively refine these clusters.

  • Cluster discovery: EM capsules instead seek clusters of similar whole-pose votes generated by already identified parts.
  • Cluster formation: Unlike EM capsules, GLOM does not use immutable vector votes from identified parts to activate whole representations.Each lower-level embedding contributes a revisable bottom-up signal instead.
  • Cluster formation: GLOM combines bottom-up, top-down, and attention-weighted same-level contributions to update each embedding.
  • Cluster formation: Clusters emerge as islands through interaction between an intra-level similarity process and dynamically changing suggestions from adjacent levels.

6 Replicating embedding vectors over locations

Replicating embedding vectors across locations lets GLOM represent object-level islands without dynamically allocating neurons or relying on pointers. This also supports sparse, long-range interactions.

  • Representation: Replicated object-level embeddings allow each location in an object to carry the island’s information, avoiding dynamically allocated parse-tree hardware.
  • Representation: GLOM’s island search must consider alternative location groupings and negotiate a shared vector while satisfying inter-level constraints.
  • Representation: High-dimensional vector averages remain close to their component vectors, supporting blended representations that preserve similarity.
  • Efficiency: Sparse higher-level interactions can span farther without increasing computation because one sampled location represents a distant island’s object-level information.
  • Efficiency: A Gaussian sampling scheme can select competing locations, using wider sampling distributions at higher hierarchy levels.

7 Learning Islands

GLOM proposes learning islands by reconstructing masked images while regularizing inter-level predictions toward consensus. The paper highlights unresolved questions about representation collapse and contrastive separation.

  • Learning objective: Masked-image reconstruction preserves input information and can make object identification useful for filling sufficiently large missing regions.
  • Contrastive learning: Contrastive learning must distinguish patches from different objects because same-image crops may contain different object sets.
  • Learning objective: Regularizing bottom-up and top-down networks to predict a consensus opinion makes feed-forward islands more coherent.
  • Open question: Whether consensus-based training collapses representations without negative examples for unrelated contexts remains an important unresolved question.
  • Contrastive learning: Maintaining separation does not require pushing already-different representations arbitrarily far apart.
  • Contrastive learning: Allowing locations to choose similar embeddings can form identical islands and reduce collapse, potentially removing the need for negative examples with other training techniques.

8 Representing coordinate transformations

GLOM represents part-whole coordinate transformations with learned, distributed activity vectors rather than explicit matrices or quaternions. These transformations depend on part and whole identities while preserving viewpoint-invariant spatial relationships.

  • Neural networks generally represent shape details more effectively relative to intrinsic coordinate frames than camera- or world-based frames.
  • GLOM proposes highly redundant vector representations for pose instead of conventional matrix or quaternion representations.
  • Universal capsules condition part-whole transformations on distributed part identity, requiring the full part-level vector as bottom-up input.
  • Identity and pose may be entangled so activity patterns can represent uncertainty over alternatives such as a tilted square versus an upright diamond.
  • GLOM forces part-whole coordinate transforms to be modeled correctly so whole-level averaging can exploit viewpoint-invariant spatial relationships and generalize to new viewpoints.

9 Representing uncertainty

GLOM represents uncertainty through distributed tuning curves and activity patterns rather than assigning individual neurons to interpretable variables. This supports combining probabilistic evidence while retaining interpretable islands of similar vectors.

  • Embedding components need not correspond directly to physical variables because combining information correctly requires representing each source’s uncertainty.
  • Populations of neurons with Gaussian tuning can represent probability distributions over an M-dimensional entity space, including different uncertainty widths across directions.
  • Adding neural activities combines independently supplied distributions multiplicatively, while a finite population limits the distributions that can be represented.
  • Mixtures of Gaussian and uniform tuning provide flexible unibump basis functions for representing multimodal log-probability distributions.
  • Understanding individual neurons requires jointly inferring the latent space and tuning curves, but finding islands of nearly identical vectors remains straightforward.
  • Each location assumes at most one part per hierarchy level, binding all activities in that level’s embedding to the same part.

10 Comparisons with other neural net models

Compared with capsule models, CNNs, and transformers, GLOM relocates representation from predefined object-type neurons to location-based continuous vectors and structured interactions. Its benefits include flexible island formation, viewpoint handling, and non-tree property inheritance, alongside biological and computational trade-offs.

  • Comparison with capsule models: GLOM avoids preallocating neurons to discrete part types by representing part identity in a continuous activity space.
  • Comparison with capsule models: GLOM avoids dynamic routing because each occupied location constructs its own representation of the relevant whole.
  • Comparison with capsule models: Island formation automatically performs split-and-merge-like clustering through continuous agreement, without discrete changes in cluster number.
  • Comparison with capsule models: Its main disadvantage is that object-shape knowledge is not localized to small neuron groups and must be replicated across locations, which is statistically inefficient biologically.
  • Comparison with capsule models: GLOM preserves capsule ideas such as viewpoint-aware coordinate relationships and coincidence filtering while allocating neurons to locations rather than object types.
  • Comparison with transformer models: In transformer form, GLOM uses shared recurrent weights, simplified self-attention, structured adjacent-level interactions, and coordinate-transform networks conditioned on part and whole identities.
  • Comparison with convolutional neural networks: Unlike standard CNN pooling, GLOM seeks viewpoint invariance in weights while retaining viewpoint equivariance in activities.
  • Representing the ISA hierarchy: Distributed property inheritance can combine partially overlapping classes, so types need not form a tree and conflicting evidence can be represented.

11 Video

For video, GLOM treats static images as repeated frames so embeddings can settle into islands, then extends the same process to slowly changing scenes. Rapid image changes can prevent adequate settling and impair interpretation.

  • The paper focuses on static images but motivates GLOM as a sequence-processing pipeline applicable to time-varying images.
  • Event time, representation time, and reference time distinguish when an event occurs, when its neural representation occurs, and what time that representation refers to.
  • Repeated frames allow multiple time steps for embeddings to settle into appropriate islands, whereas dynamic images require those steps to track changing occupants.
  • Small object motions require large changes at only a small subset of object-level locations, while remaining locations change slightly with pose.
  • If image changes are small and predictable, embeddings can form slowly changing islands that track the dynamic image while lower levels change more rapidly.
  • Rapid image changes leave insufficient time for iterative settling, so GLOM cannot correctly interpret complicated shapes in such conditions.

12 Is GLOM biologically plausible?

GLOM is biologically implausible in several respects, especially its reliance on weight-sharing, negative-pair contrastive learning, and backpropagation. The paper discusses distillation, offline negative learning, and local error-sign mechanisms as possible ways to address some of these tensions, while simulations remain below vanilla CNN performance.

  • 12 Is GLOM biologically plausible?: GLOM appears biologically implausible because it requires weight-sharing, negative-pair processing for contrastive learning, and backpropagation through hidden layers.These are identified as three specific biological-model concerns.
  • 12.1 Is the neocortex a giant distillery?: Distillation can replace literal weight copying by training location-specific bottom-up and top-down networks toward a consensus formed across students and contextual embeddings.The proposal requires corresponding networks to become functionally similar rather than having identical weights.
  • 12.1 Is the neocortex a giant distillery?: Averaging predictions from different locations can produce a stronger object-level ensemble than individual bottom-up predictions.The claimed benefit arises when location-specific bottom-up models differ somewhat while processing corresponding parts.
  • 12.1 Is the neocortex a giant distillery?: Distillation permits different locations to receive differently structured inputs, supporting receptive fields that enlarge away from the fovea and avoiding location inputs for top-down models.The passage contrasts this flexibility with weight-sharing in convolutional networks.
  • 12.1 Is the neocortex a giant distillery?: Using distillation across 10^4 locations addresses the synapse–fixation discrepancy by combining about 10^9 local synapses with about 10^13 shared training examples.Many of the shared examples are highly correlated.
  • 12.3 Communicating error derivatives in the brain: Simulations show that the proposal works fairly well but performs worse than vanilla CNNs, with lack of weight-sharing contributing substantially to the gap.The paper suggests co-distillation could reduce this statistical inefficiency.

13 Discussion

The discussion presents GLOM as an unimplemented design that represents parse-tree nodes with islands of similar vectors, linking field-based and structural approaches to perception. It argues that vector similarities may support compositional reasoning without a separate symbolic interface, while emphasizing that experiments are still needed.

  • 13 Discussion: Because GLOM lacks a working implementation, its ideas remain subject to serious credibility concerns that experiments could resolve.The author notes that ongoing and future testing is needed, including generalization to radically new viewpoints.
  • 13 Discussion: Islands of similar vectors unify Gestalt fields with structural descriptions by representing percepts continuously while giving each parse-tree node an embedding-space address.The shared vector for a whole differs from those representing its parts.
  • 13 Discussion: The discussion proposes learned high-dimensional vector similarities as a basis for analogy-driven reasoning rather than requiring a neurosymbolic handoff.The passage frames this as the author’s view of the primary mode of reasoning.
  • 13 Discussion: The paper connects GLOM’s hierarchy-oriented interactions and contrastive island regularization to transformer-based sentence parsing.It cites BERT’s success and earlier demonstrations that neural networks can output parse trees when required by the task.
Loading 2102.12627v1…