PaperlayerView exact revision on arXiv

Source-linked AI summary

Coordination Among Neural Modules Through a Shared Global Workspace

Anirudh Goyal, Aniket Didolkar, Alex Lamb, Kartikeya Badola, Nan Rosemary Ke, Nasim Rahaman, Jonathan Binas, Charles Blundell, Michael Mozer, Yoshua Bengio

arXiv:2103.01197v2cs.LGcs.AIstat.ML

TL;DR

Structured neural models use pairwise interactions, but the paper asks how specialist modules can achieve global coordination. It introduces a limited-capacity shared workspace for communication among specialists and reports better performance across a range of visual reasoning benchmarks than pairwise self-attention schemes.

  • Problem

    Structured neural architectures commonly use pairwise interactions, which may not provide global coordination or a coherent representation for downstream tasks.

    Source: §1 · pp. 1–2

  • Method

    The method adds a shared workspace to neural architectures, where specialists compete to write information that is then broadcast to all specialists.

    Source: §2 · p. 3, §2.1 · p. 4

  • Results

    The shared-workspace approach achieves better performance across a wide range of visual reasoning benchmarks than pairwise interactions typically used in self-attention schemes.

    Source: §5 · p. 9

  • Takeaways & Limitations

    The authors conclude that limited workspace capacity supports specialization and compositionality while facilitating synchronization among otherwise independent specialists.

    Source: §4 · p. 8, §5 · p. 9

Abstract

from arXiv · show

Deep learning has seen a movement away from representing examples with a monolithic hidden state towards a richly structured state. For example, Transformers segment by position, and object-centric architectures decompose images into entities. In all these architectures, interactions between different elements are modeled via pairwise interactions: Transformers make use of self-attention to incorporate information from other positions; object-centric architectures make use of graph neural networks to model interactions among entities. However, pairwise interactions may not achieve global coordination or a coherent, integrated representation that can be used for downstream tasks. In cognitive science, a global workspace architecture has been proposed in which functionally specialized components share information through a common, bandwidth-limited communication channel. We explore the use of such a communication channel in the context of deep learning for modeling the structure of complex environments. The proposed method includes a shared workspace through which communication among different specialist modules takes place but due to limits on the communication bandwidth, specialist modules must compete for access. We show that capacity limitations have a rational basis in that (1) they encourage specialization and compositionality and (2) they facilitate the synchronization of otherwise independent specialists.

1 INTRODUCTION

Structured neural architectures separate information among specialist components, raising the question of how those components coordinate. The paper proposes a shared, bandwidth-limited workspace through which specialists exchange information and argues that this bottleneck supports flexible generalization.

1 INTRODUCTION

  • Structured models separate information among components, but achieving coherence and coordination between them remains an open question.The introduction describes modularity as beneficial for generalization, scaling, and long-range dependencies, while identifying coordination as a challenge.Source: §1 · p. 1
  • In the driving example, specialists for road position, steering, and alerts must exchange information coherently to execute the task properly.Alerts may need priority to override default driving behavior.Source: §1 · p. 2
  • The paper proposes training specialist modules to communicate through a shared workspace, drawing on Global Workspace Theory.The workspace is a common representation that specialists can modify and whose contents are broadcast to all; write access is limited to maintain coherence.Source: §1 · p. 2
  • Modular systems need compatible representations and synchronization, particularly when specialists have statistical, functional, or causal dependencies.The introduction describes a shared internal language as necessary for specialists to share compatible representations.Source: §1 · p. 2

2 SYNCHRONIZING NEURAL MODULES THROUGH A SHARED WORKSPACE

The method adds a shared workspace to modular and position-based neural architectures, replacing pairwise communication with competitive writes and broadcast reads. Its memory slots also support higher-order interactions and persistent, dynamically adjusted filtering, while the proposed communication cost is linear in the number of specialists.

2 SYNCHRONIZING NEURAL MODULES THROUGH A SHARED WORKSPACE

  • The method adds a shared workspace to Transformers and slot-based architectures, where specialists compete for write access at each computational stage.The workspace is updated and then its contents are broadcast to the specialists.Source: §2 · pp. 3–4
  • Key-value attention supplies differentiable, soft selection and combination of specialist or memory values, with permutation-invariant inputs and outputs.The experiments use multihead dot-product attention.Source: §2 · p. 3

2.1 SPECIFICS OF THE SHARED WORKSPACE.

  • The shared workspace uses a two-step process: specialists compete to write information, then workspace contents are broadcast for specialists to update their states.Softmax competition allows soft participation; top-k softmax instead selects a fixed number of highest-scoring specialists to write.Source: §2.1 · pp. 4–5
  • Workspace-mediated messages can include higher-order interactions among more than two modules, with the shared channel providing a common representation space.The workspace first stores filtered summaries; specialists then summarize slot contents and update their states.Source: §2.1 · p. 5
  • Persistent workspace memory can filter information based on prior inputs and dynamically adjust its filtering for a particular input.The paper contrasts this with fixed inducing points and describes comparisons against 2×Self-Attention and a version without memory persistence.Source: §2.1 · p. 5
  • Communication complexity is linear in the number of specialists; the experiments use a practically constant number of memory slots.The paper contrasts this with quadratic scaling for pairwise attention.Source: §2.1 · p. 5

3 RELATED WORK

The related work situates the paper among modular architectures, memory networks, mixture-of-experts models, and approaches to reducing attention complexity. It distinguishes the shared workspace by combining specialists with competitive writes and workspace reads.

3 RELATED WORK

  • The paper focuses on coherence and synchronization among specialist modules through a global shared workspace.It situates this focus within earlier work on interacting specialized modules and modular architectures.Source: §3 · pp. 5–6
  • Unlike several prior memory approaches, the proposed workspace restricts writes to be sparse and competitive.The comparison includes memory tokens, segment memories, compressed memories, and Set Transformer inducing points that are not maintained across layers.Source: §3 · p. 6
  • The proposed architecture combines modular specialists and shared-workspace coordination, while allowing multiple experts to be active at once.The paper contrasts this with Neural Turing machines’ single specialist and mixture-of-experts models’ typical single active expert.Source: §3 · p. 6

4 EXPERIMENTS

The experiments evaluate shared workspaces across visual benchmarks and modular architectures, testing whether limited-capacity communication improves performance and specialist coordination. Across the supplied results, shared-workspace models show gains on several tasks, with specialization and transfer benefits also reported.

Experiments

  • Shared-workspace models outperform or improve on baselines across triangle detection, CATER, Sort-of-CLEVR, bouncing-ball reasoning, Atari transfer, and MultiMNIST generation.The reported outcomes include faster convergence, higher accuracy or ARI, improved transfer, and lower test cross-entropy.Source: §4 · pp. 7–9

Making sense of the visual input

  • Transformers with shared workspace converge faster and achieve higher accuracy on equilateral-triangle detection; on Sort-of-CLEVR they also converge faster and outperform baselines for relational and non-relational questions.For triangle detection, the method also significantly outperforms Set Transformer; the Sort-of-CLEVR result is attributed to task sparsity.Source: §4 · pp. 7, 9, Figure 4 · p. 7, Figure 3 · p. 7

Multi MNIST Generation

  • On MultiMNIST generation, TIMs + SW produces the least test cross-entropy among TIMs and Image Transformers while dividing image regions across mechanisms more interpretably.The activation maps contrast TIMs’ mechanism 2 activation across all pixels with TIMs + SW’s more differentiated mechanism assignments.Source: §4 · p. 8, Figure 5 · p. 8

CATER: Object Tracking

  • On CATER object tracking, shared-workspace models outperform pairwise multihead attention and the Set Transformer, including variants with k = 5 specialists writing to the workspace.Table 1 compares Top-1 and Top-5 accuracy, but the supplied passages provide no cell values.Source: Table 1 · p. 7, §4 · p. 8

Shared Workspace for Physical Reasoning

  • On bouncing-ball prediction, shared workspace yields higher ARI than pairwise SCOFF, while Table 2 reports decreasing performance as workspace-slot count increases.The authors interpret higher ARI as better division of labor among specialists.Source: §4 · p. 9, Table 2 · p. 9

Shared Workspace for Atari Video Games

  • Across 36 Atari game pairs, RIMs + SW reaches median performance ratios of 1.13 on game A and 1.11 on game B.The authors attribute the gains to better forward transfer and reduced backward interference, presumably through more appropriate modularization.Source: §4 · p. 9

5 CONCLUSION

The paper proposes a shared workspace to coordinate modular neural specialists. It reports better performance across visual reasoning benchmarks than pairwise self-attention interactions.

5 CONCLUSION

  • A limited-capacity shared workspace mediates specialist communication and yields better performance across visual reasoning benchmarks than pairwise self-attention schemes.Specialists compete to post new workspace contents, which become accessible to all specialists for their updates.Source: §5 · p. 9
  • The model divides knowledge and expertise among specialists that compete to post new contents to the workspace.Source: §5 · p. 9
  • After the workspace is updated, it is accessible to all specialists for their own updates.Source: §5 · p. 9

ETHICS STATEMENT

The authors do not foresee negative social impacts, while acknowledging that machine-learning improvements could be misused.

ETHICS STATEMENT

REPRODUCIBILITY STATEMENT

The authors describe releasing the code after review and including it in the supplementary material. The appendices provide RIM and TIM pseudocode and generic hyperparameter and training details.

REPRODUCIBILITY STATEMENT

A PSEUDO CODES

  • For RIMs, specialists compete to update a shared workspace, whose contents are broadcast to all specialists instead of using direct module-to-module attention.Algorithm 1 takes the current sequence element, prior specialist states, and row-compartmentalized memory as inputs.Source: A PSEUDO CODES · p. 14
  • For TIMs, the shared workspace replaces direct module-to-module communication.Source: A PSEUDO CODES · p. 14

B HYPERPARAMETERS

  • The hyperparameter section describes workspace slot and attention settings, including trials with one or four slots and no search for broadcast-attention hyperparameters.The section identifies memory attention heads, keys, and values as broadcast parameters.Source: B HYPERPARAMETERS · p. 16
  • The appendix reports approximate V100 training times for vision, bouncing-ball, and StarCraft tasks.It also says no dataset-specific hyperparameter search was done for the bouncing-ball datasets and Curtain task.Source: B HYPERPARAMETERS · p. 17

C IMPLEMENTATION DETAILS

Workspace writing uses a gating mechanism with input and forget gates. The input-to-gate weight W1 is shared across specialists.

C IMPLEMENTATION DETAILS

D PROPERTIES OF SHARED WORKSPACE

The section tests whether higher-order interaction and persistent workspace memory contribute to Shared Workspace performance. The reported results distinguish it from repeated self-attention and show faster convergence than the compared models.

D PROPERTIES OF SHARED WORKSPACE

  • The comparison tests whether repeating self-attention can simulate higher-order interaction, while noting that it lacks a global channel enforcing a shared message representation.The section also tests a model combining self-attention with the shared workspace.Source: D PROPERTIES OF SHARED WORKSPACE · p. 17
  • Reinitializing the workspace at every layer tests whether persistent memory contributes to performance and convergence speed.The comparison is run on Sort-of-CLEVR.Source: D PROPERTIES OF SHARED WORKSPACE · p. 17
  • Shared Workspace converges faster and generalizes better than the compared models on Sort-of-CLEVR; replacing it with 2×SA significantly reduces performance.Removing memory persistence also results in significantly slower convergence.Source: D PROPERTIES OF SHARED WORKSPACE · p. 17, Figure 7 · p. 18

E TRANSFORMER TASKS

The section describes Transformer setups for triangle detection, Sort-of-CLEVR reasoning, and CATER object tracking. The tasks use different image or video inputs and classification targets.

E.1 DETECTING EQUILATERAL TRIANGLES

  • Triangle detection uses 64 × 64 images split into 4 × 4 patches, a four-layer Transformer, and a CLS token for classification.The dataset has 50,000 training and 10,000 evaluation examples; the proposed models use an eight-slot workspace.Source: §E.1 · p. 18

E.2 SORT-OF-CLEVR

  • Sort-of-CLEVR combines image patches with an encoded 11-bit question and uses a four-layer Transformer to answer relational or non-relational questions.The proposed models use an eight-slot workspace, and classification uses the CLS representation.Source: §E.2 · pp. 18–19

E.3 CATER: OBJECT TRACKING

  • CATER tracking samples 50 frames per video, stacks them into 10 inputs, and predicts the snitch’s cell in the final frame on a 6 × 6 grid.Each input is processed through an 18-layer ResNet before a six-layer Transformer.Source: §E.3 · p. 19

F RIMS TASKS

These experiments apply shared-workspace extensions to bouncing-ball prediction and modular image generation. The reported results favor the extension over baselines for ball dynamics and describe improved division of labor on MultiMNIST.

F.1 BOUNCING BALL

  • The shared-workspace extension predicts bouncing-ball dynamics more accurately than LSTM, RIMs, and RMC baselines during 35-step rollouts from 10 ground-truth frames.The reported advantage increases with rollout length and number of balls.Source: Figure 9 · p. 19, §F.1 · p. 19

G.1 TIMS

  • TIMs+SW replaces pairwise communication among four mechanisms with a shared workspace, while TIMs retains pairwise competition between mechanisms.The MultiMNIST configurations use four mechanisms; the TIMs+SW workspace has two memory slots.Source: TIMs+SW · pp. 20–21

TIMs+SW

  • In TIMs+SW, a two-slot workspace is shared across four modular middle layers, and the compared architectures have 1M–1.8M parameters.The first and final two layers are monolithic.Source: TIMs+SW · pp. 20–21

G.1.1 MULTIMNIST GENERATION

  • TIMs+SW achieves the best test-set cross-entropy in 20 epochs on MultiMNIST next-pixel prediction and yields a more interpretable division of labor than pairwise TIMs.The task uses images composed of independently sampled digits; the reported loss is measured on the test set.Source: TIMs+SW · p. 21, Table 5 · p. 21

G.1.2 USING WORKSPACE FOR LANGUAGE MODELLING

The paper evaluates TIMs and shared-workspace variants on WikiText-103 next-word prediction using validation perplexity. Shared-workspace variants consistently improve on TIMs, with the sparse variant performing best.

G.1.2 USING WORKSPACE FOR LANGUAGE MODELLING

  • The experiment trains eight-layer Transformer decoders for 15 epochs on WikiText-103 next-word prediction and reports validation perplexity.The decoder layers are arranged as two monolithic layers, four modular layers, and two final monolithic layers.Source: §G.1.2 · pp. 21–22
  • TIMs+SSW uses soft attention for mechanism activation scores, while TIMs+HSW uses top-k attention with k=2.Both are shared-workspace variants.Source: §G.1.2 · p. 21
  • TIMs+SSW and TIMs+HSW consistently improve performance over TIMs on WikiText-103 validation perplexity, and TIMs+HSW performs best.The models have comparable parameter counts, within a 10% difference.Source: §G.1.2 · p. 22, Figure 11 · p. 23

Related papers

Loading 2103.01197v2…