Source-linked AI summary
TPR-Attention for Combinatorial Generalization
Melisa Civelekoğlu, Isabeau Prémont-Schwarz
TL;DR
Deep networks struggle with combinatorial generalization because they rely on statistical patterns rather than explicit compositional structure. The paper introduces TPR-Attention, an attention mechanism over structured tensor-product representations, and finds substantially better out-of-distribution compositional generalization than classical attention and ResNets on interacting-factor tasks.
Problem
Deep networks struggle to generalize to novel combinations because they rely on statistical patterns rather than compositional rules.
Method
TPR-Attention operates directly on role–filler bindings, using structured object-centric representations with binding and unbinding operations within attention.
Results
TPR-Attention generalized compositionally out of distribution substantially better than classical attention and ResNets in a controlled setting with interacting factors.
Takeaways & Limitations
The results indicate that explicit compositional structure in attention supports combinatorial generalization in the difficult interacting-factor setting.
Takeaways & Limitations
The experiments use manually provided latent factors and controlled settings, and compare combinatorial generalization for only a single layer.
Abstract
from arXiv · showhide
Systematic generalization remains a significant challenge in deep learning. In particular, combinatorial generalization - generalizing to new configurations of known factors of variation - is effortless for humans but difficult for standard neural architectures that rely on statistical correlations rather than explicit structural representations. We introduce a new architectural component that embeds structured inductive bias into deep learning: an attention mechanism operating over tensor-product representations (TPRs). Through controlled experiments on compositional tasks, we show that this TPR-attention mechanism outperforms existing architectural components in combinatorial generalization. These results highlight the value of integrating explicit compositional structure into neural attention and point toward a promising path for models capable of systematic generalization.
1 Introduction
Combinatorial generalization requires recombining known factors into novel structures, but traditional deep networks struggle because they rely on statistical patterns rather than compositional rules. TPR-Attention addresses this limitation with structured object-centric representations and explicit binding and unbinding operations, achieving better generalization in difficult interacting-factor settings.
- Combinatorial generalization recombines known factors of variation into novel structures, as humans do effortlessly.
- Traditional deep networks struggle with novel combinations because they learn statistical co-occurrences rather than underlying compositional structure.
- TPR-Attention binds an object’s factors of variation into a structured object-centric representation and explicitly performs binding and unbinding within attention.
- TPR-Attention achieves better combinatorial generalization than existing architectural components, particularly when factors of variation interact.
2 Related Work
Prior work links disentangled representations and symbolic role–filler structures to compositional generalization, but their relationship remains unclear. This paper focuses on interacting factors, where existing approaches can remain compositionally unsuccessful despite achieving high disentanglement.
- Disentangled representations separate independent factors of variation so they can be recombined into novel concept compositions.
- The relationship between disentanglement and compositional generalization remains unclear, with several studies reporting limited evidence that factor decoupling improves generalization.
- Models can achieve high disentanglement yet fail to generalize compositionally when factors interact.
- Tensor Product Representations and related vector symbolic architectures provide role–filler bindings that separate feature from content.
3 Method
TPR-Attention represents objects as tensor products of role–filler bindings and processes them through structured matching, property extraction, transformation, and re-binding. Its three-stage mechanism operates over a TPR-based associative memory to produce transformed role–filler outputs.
- TPRs represent objects as vector-space embeddings of symbolic structures composed of roles and fillers.
- Each role–filler relation is bound by a tensor product, and an object is the superposition of all such bindings.
- TPR-Attention generates a structured role–filler query, matches relevant objects, extracts a target property, then transforms and re-binds it per attention head.
- TPR-SAM provides the structured associative-memory mechanism used to match and extract properties at the same index.
- The attention mechanism operates on a TPR-SAM memory represented as the superposition of past transformed objects.
- Objects receive similarity-based weights when their filler bound to the queried role matches the query filler.
- Target fillers are extracted from matching objects and weighted by similarity to the filler bound to the target role.
- 3.2.3 Transformation and Re-binding: A learned linear transformation H transforms extracted fillers, which are re-bound to a new role per attention head and superposed into the output object.
4 Experiments
The experiments evaluate TPR-Attention on controlled latent-representation composition tasks, testing unseen factor combinations across multiple OOD splits, test conditions, and interaction settings. TPR-Attention is compared with regular attention and a ResNet baseline.
- Controlled composition task: The controlled composition task operates on latent representations to isolate compositional behavior from perceptual representation learning.The output combines a transform input's specified factor with the reference input's remaining factors.
- Results: In the square red OOD condition, TPR-Attention with four heads achieves lower loss in all tests than classical attention with four heads and a one-layer ResNet.Figure 2 reports means over five seeds with ±2 standard deviation shading.
- Baselines: The evaluation compares one layer of TPR-Attention with one layer of regular attention and ResNet on unseen factor combinations.The comparison targets how well each model generalizes compositionally.
- OOD evaluation: Three OOD splits test generalization across numerical, mixed numerical-categorical, and categorical factor combinations.The splits are scale pos, square pos, and square red.
- OOD evaluation: The experiments test whether OOD factors occur in the reference input, transform input, or only emerge through their composition.These are evaluated as three separate test conditions.
- Interaction settings: Interacting-factor experiments include numerical scale-pos interactions and categorical shape-col interactions, alongside non-interacting factors.Shape-color interaction is introduced through a random-matrix linear mixing, while scale-pos interaction adds the factors.
5 Discussion
TPR-Attention applies attention directly to structured role-filler bindings, using matching, property extraction, and transformation with re-binding across heads. In controlled experiments with interacting factors, it generalized compositionally out of distribution better than classical attention and ResNets, while remaining limited by manually provided factors, controlled settings, and single-layer comparisons.
- TPR-Attention: TPR-Attention operates directly on structured role-filler bindings, giving attention explicit object-centric structure.The mechanism performs binding and unbinding operations within attention.
- Discussion: TPR-Attention generalized compositionally out of distribution substantially better than classical attention and ResNets in a controlled setting with interacting factors.The authors state that this level of combinatorial generalization had not previously been demonstrated by prior architectures.
- Limitations: The experiments rely on manually provided latent factors, use a small controlled factor setting, and compare only single-layer versions.The paper leaves scaling to higher-dimensional or noisier domains and persistence of the advantage under stacking unresolved.
- TPR-Attention: The mechanism has three stages: object matching, target-property extraction, and transformation with re-binding per attention head.These stages are summarized in Algorithm 1.
- Object matching: In object matching, filler similarity scores weight memory objects, producing a weighted sum of objects relevant to the query.Orthonormal role vectors allow role contractions to collapse through the Kronecker delta.
- Transformation and re-binding: Target fillers are extracted from matching objects, transformed by a learned matrix, and re-bound to a new role for each attention head.The transformation is represented by Tbind(f, H, rn) := H rn f = rn ⊗(f ⊤H).
- Composition task: For composition tasks, the output preserves the reference object except at an action-specified target role, which is substituted using the transform object.The model uses action-conditioned queries, object IDs, and a copy mechanism to preserve most reference features; analogous copy behavior is used in baselines.
E Composition Task Representation Details
The composition-task representation encodes objects as role-filler structures, using fixed one-hot roles and factor-specific fillers. Categorical, geometric, and interacting factors receive distinct vector constructions, including learned tensor-based interactions.
- Object representation: Each object is represented as a set of role-filler pairs, with roles specifying attribute slots and fillers specifying their values.Roles are fixed, while filler construction depends on the factor of variation.
- Roles: Roles use canonical basis vectors in R^dr, making them one-hot vectors.For each role j, rj = ej ∈ R^dr.
- Categorical factors: Color and shape, as categorical factors, are encoded with one-hot vectors.Shape categories use the same encoding strategy as the example color categories.
- Geometric factors: Orientation is represented on the unit circle, while normalized two-dimensional position is encoded as a vector representation.Scale indices are mapped to points on the unit sphere using fixed angles.
- Interacting factors: Scale and position are combined into a numerical interacting factor, while shape and color form a categorical interacting factor through tensor contraction.The categorical interaction uses fshape[i] M[k, i, j] fcolor[j].
F.1 Extended Results for Non-Interacting Setting
Extended results report loss on combinatorial generalization across out-of-distribution test sets for 4- and 8-head models. TPR-Attention consistently achieves lower loss than classical attention and ResNet baselines.
- Results: TPR-Attention consistently achieves lower loss than classical attention and ResNet baselines across out-of-distribution combinatorial-generalization test sets.The comparison is shown for both 4-head and 8-head configurations.
- Results: The figure’s left column reports 4-head results and its right column reports 8-head results.Curves show means over 5 seeds.
- Uncertainty: Shaded regions indicate ± 2 standard deviation for 4 heads and ± 2 standard error for 8 heads.These uncertainty summaries accompany the mean curves over 5 seeds.
F.2 Extended Results for Numerical Interaction Setting
Figure 6 compares loss on combinatorial generalization with scale–position interaction across five seeds for TPR-Attention, classical attention, and a one-layer ResNet.
- The figure compares TPR-Attention with eight heads, classical attention with eight heads, and a one-layer ResNet baseline.
- The evaluation focuses on combinatorial generalization with scale–position interaction.
- Shaded regions represent ± 1 standard error across five seeds.
F.3 Extended Results for Categorical Interaction Setting
Figure 7 compares loss on combinatorial generalization with shape–color interaction across five seeds for TPR-Attention, classical attention, and a one-layer ResNet.
- The figure compares TPR-Attention with eight heads, classical attention with eight heads, and a one-layer ResNet baseline.
- The evaluation focuses on combinatorial generalization with shape–color interaction.
- Shaded regions represent ± 1 standard error across five seeds.