Source-linked AI summary
Perceiver: General Perception with Iterative Attention
Andrew Jaegle, Felix Gimeno, Andrew Brock, Andrew Zisserman, Oriol Vinyals, Joao Carreira
TL;DR
Existing perception models often rely on modality-specific architectural assumptions, while flexible Transformers scale poorly with very large inputs. The Perceiver uses iterative asymmetric attention to distill inputs through a latent bottleneck, and it performs competitively across modalities, including comparisons with ResNet-50 and ViT. Its remaining modality-specific augmentation and position encoding define an important scope boundary.
Problem
Perception models often use modality-specific architectural priors, requiring different designs for images, audio, multimodal inputs, and point clouds.
Method
The Perceiver uses a single Transformer-based architecture with iterative cross-attention and latent self-attention to process arbitrary modality configurations through a small latent bottleneck.
Results
The Perceiver performs competitively across modalities, including performance comparable to ResNet-50 and ViT on ImageNet and competitive AudioSet results.
Takeaways & Limitations
The Perceiver provides a general perception architecture that scales to more than a hundred thousand inputs and handles arbitrary sensor configurations.
Takeaways & Limitations
The model still relies on modality-specific augmentation and position encoding, so end-to-end modality-agnostic learning remains future work.
Abstract
from arXiv · showhide
Biological systems perceive the world by simultaneously processing high-dimensional inputs from modalities as diverse as vision, audition, touch, proprioception, etc. The perception models used in deep learning on the other hand are designed for individual modalities, often relying on domain-specific assumptions such as the local grid structures exploited by virtually all existing vision models. These priors introduce helpful inductive biases, but also lock models to individual modalities. In this paper we introduce the Perceiver - a model that builds upon Transformers and hence makes few architectural assumptions about the relationship between its inputs, but that also scales to hundreds of thousands of inputs, like ConvNets. The model leverages an asymmetric attention mechanism to iteratively distill inputs into a tight latent bottleneck, allowing it to scale to handle very large inputs. We show that this architecture is competitive with or outperforms strong, specialized models on classification tasks across various modalities: images, point clouds, audio, video, and video+audio. The Perceiver obtains performance comparable to ResNet-50 and ViT on ImageNet without 2D convolutions by directly attending to 50,000 pixels. It is also competitive in all modalities in AudioSet.
1. Introduction
The Perceiver addresses modality-specific architectural priors with a flexible Transformer-based model that scales high-dimensional inputs through iterative attention and a latent bottleneck. It achieves competitive classification performance across images, audio, video, multimodal data, and point clouds.
- Motivation: Modality-specific priors force architectural redesign when inputs change across images, audio, stereo, or point clouds.Examples include 2D convolutions for images, 1D convolutions or LSTMs for audio, and different strategies for multimodal fusion.
- Contribution: The single architecture is designed to handle arbitrary configurations of different modalities while retaining Transformer expressivity and flexibility.The approach targets high-dimensional inputs that standard self-attention cannot process efficiently because of quadratic scaling.
- Approach: The Perceiver uses a cross-attention bottleneck to project high-dimensional inputs into a smaller latent array before deep Transformer processing.It alternates cross-attention with latent self-attention, with input indices M much larger than latent indices N.
- Approach: Position and modality-specific features preserve spatial, temporal, and cross-modal information without imposing explicit domain structures.These features can be learned or constructed using high-fidelity Fourier features for each input element.
- Results: The Perceiver performs comparably to ResNet-50 and ViT on ImageNet and competitively on AudioSet, while also performing strongly on ModelNet-40 point clouds.The reported evaluations cover image classification, raw audio, video, audio-video combinations, and point-cloud classification.
2. Related Work
Related work establishes a trade-off between ConvNets’ scalable local processing and Transformers’ flexible but input-size-sensitive attention. The Perceiver combines cross-attention projection with latent processing to obtain linear input scaling while decoupling network depth from input size, supporting multimodal processing without separate modality-specific extractors.
- Convolutional architectures: ConvNets scale efficiently for high-resolution images through weight sharing and local 2D neighborhoods but offer limited flexibility for combining multiple signals.
- Efficient attention architectures: Standard Transformers are flexible but scale poorly because self-attention compares every input with every other input at every layer.
- Efficient attention architectures: Prior efficient Transformer approaches reduce input size through subsampling, convolutions, or factorization before applying attention.ViT, for example, reduces images to approximately 200 inputs using a 2D convolutional projection.
- Efficient attention architectures: The Perceiver uses cross-attention over a low-dimensional auxiliary array to reduce attention complexity from quadratic to linear in input size.
- Novelty: Unlike related linear-complexity methods, the Perceiver also decouples network depth from input size, enabling very deep architectures.The paper identifies this decoupling, rather than linear scaling alone, as important for challenging tasks across domains.
- Multimodal architectures: Multimodal systems commonly use separate feature extractors and require application-specific decisions about when and how to fuse modalities.
3. Methods
The Perceiver uses asymmetric cross-attention to compress large input arrays into a small latent array, where deep Transformer processing scales independently of input size. Position and modality features restore information that permutation-invariant attention cannot otherwise exploit.
- The Perceiver architecture: The architecture alternates cross-attention from a large byte array to a small latent array with a Transformer tower operating on the latent representation.The latent array is typically much smaller than the input, such as 512 latents for ImageNet.
- Taming quadratic complexity with cross-attention: Learned latent queries attend to input keys and values, reducing attention complexity from O(M^2) to O(MN) when N ≪ M.M indexes input elements and N indexes latent units.
- Uncoupling depth with a latent Transformer: Latent self-attention costs O(N^2), allowing additional Transformer layers without coupling network depth to the input size.The resulting architecture has complexity O(MN + LN^2), and the best ImageNet model uses 48 latent Transformer blocks.
- Iterative cross-attention & weight sharing: Iterative cross-attention lets the latent bottleneck extract information from inputs across multiple steps, balancing informative cross-attends against cheaper latent self-attends.More cross-attends improve performance but increase computational requirements.
- Iterative cross-attention & weight sharing: The Perceiver can share weights across Transformer and cross-attention blocks, reducing ImageNet parameters by approximately 10x while reducing overfitting and boosting validation performance.Weight sharing is enabled by the architecture’s iterative structure.
- Position encodings and input structure: Because attention is permutation-invariant, the model uses position and modality-specific features, including Fourier features, to represent spatial, temporal, and modality identity.These features let the network learn how to use or ignore positional structure rather than imposing it through convolutional architecture.
4. Experiments
The experiments evaluate Perceiver across images, audio, video, multimodal inputs, and point clouds, including settings that challenge modality-specific spatial assumptions. Results show competitive performance across these tasks, while attention visualizations and ablations examine how the model uses position information.
- Images – ImageNet: The best ImageNet Perceiver attends to all 50,176 input pixels eight times through cross-attention and latent Transformer blocks.The reported configuration uses six latent Transformer blocks per cross-attention stage.
- Images – ImageNet: Under permuted ImageNet inputs, Transformer and Perceiver results remain unaffected, whereas ViT and ResNet performance drops dramatically.The permutation preserves position features but prevents convolutional architectures from exploiting local neighborhood structure.
- Images – ImageNet: ImageNet attention maps evolve from early maps resembling image content to later high-frequency plaid patterns associated partly with Fourier position features.The visualized maps are computed before softmax, and later modules can attend to different pixel sets despite shared weights.
- Audio and video – AudioSet: AudioSet spectrogram and raw-audio scores improve by more than 3% with the described procedure, reaching 44.2 after further spectrogram tuning.Without the procedure, spectrogram and raw-audio models score 39.9% and 39.7% mAP versus 43.2% and 43.5%.
- Audio and video – AudioSet: AudioSet results are near state of the art for audio and video, while raw-audio Perceiver performance is similar to spectrogram-based performance.The raw-audio model obtains 38.4 mAP and exceeds most ConvNet models except CNN-14 with extra improvements.
- Audio and video – AudioSet: Audio-video fusion improves over single modalities and specialized fusion approaches but remains below a state-of-the-art system using separate models with late fusion.The Perceiver fuses audio and video at the input rather than using separate modality-specific models.
5. Discussion
The discussion presents Perceiver as a scalable, broadly applicable architecture, while acknowledging overfitting and remaining modality-specific components. Across the evaluated modalities, it performs competitively overall but does not eliminate all domain-specific design.
- Discussion: Perceiver scales to more than a hundred thousand inputs while making few assumptions about input structure and supporting arbitrary sensor configurations.The architecture is intended to enable fusion of information at all levels.
- Discussion: Across the modalities and combinations considered, Perceiver performs competitively and does best overall in the paper’s comparisons.The discussion reports parity with ResNet-50 and ViT on ImageNet and competitive AudioSet performance across audio, video, and combined inputs.
- Discussion: The authors identify overfitting as a consequence of flexibility and report that design decisions were made to mitigate it.They propose pre-training the image-classification model on very large-scale data as future work.
- Discussion: Despite reducing modality-specific prior knowledge, Perceiver still uses modality-specific augmentation and position encoding.The authors describe end-to-end modality-agnostic learning as an open research direction.
A. Extended related work
The Perceiver relates to efficient-attention and multimodal architectures by using cross-attention to create a task-independent latent space for deep processing. Its composition differs from Set Transformer designs by decoupling latent-transformer cost from input size and supporting re-entrant processing.
- Efficient attention architectures: Efficient-attention work reduces Transformer cost through local patterns, sparse patterns, simplified QKV computation, or computational bottlenecks.These approaches retain Transformer-like flexibility while targeting sub-quadratic computational cost.
- Relationship to the Set Transformer: The Perceiver’s latent array differs from Set Transformer’s ISAB because it remains task-independent and supports a deep latent Transformer.Set Transformer maps inputs to a low-dimensional array and immediately maps them back, whereas Perceiver retains a relatively large latent representation.
- Relationship to the Set Transformer: Perceiver scaling is O(MN + LN^2), compared with O(LMN) for a fully-ISAB model, where M is input size, N latent size, and L depth.The comparison highlights the cost advantage of decoupling latent processing from input size.
- Cross-attention and attentional latents: The Perceiver uses cross-attention to induce a latent space for deep processing while keeping each operation linear in input size.This use of cross-attention differs from prior architectures that write to or read from fixed-size arrays or memories.
- Global, re-entrant processing: Multiple cross-attentions provide re-entrant processing that can mitigate the bandwidth bottleneck introduced by global computation on very large inputs.The passage describes repeated input access as the mechanism for addressing this bottleneck.
B. Ablations
The ablations examine processing capacity, cross-attention arrangement, weight sharing, and implementation conventions. Larger models and more attention generally help, but excessive latent capacity can overfit or destabilize training.
- Hyperparameter sweep: Increasing model size generally improves ImageNet top-1 accuracy, except that the largest latent-dimension model shows signs of overfitting.The sweep used a small Perceiver architecture and trained models for 5 million steps.
- Cross-attention arrangements: Cross-attention-only models are evaluated using top-1 ImageNet validation accuracy, with 12 cross-attends exceeding available memory on the largest 64-TPU configuration.The table reports models without latent Transformers and without sharing weights between cross-attention modules.
- Cross-attention arrangements: Interleaved cross-attention places layers throughout the network for re-entrant processing, whereas the alternative places all cross-attends at the start.The comparison uses eight blocks of six latent self-attention modules, with sharing applied after the initial cross-attention.
- Reporting conventions: FLOPs are reported as unfused multiply-adds, so fused multiply-accumulate conventions would approximately halve the reported figures.This convention follows the cited literature practice and affects cross-paper comparisons.
C. Architectural details
The Perceiver combines cross-attention with a latent Transformer, using learned latent queries, residual attention blocks, and global latent pooling for classification. Architectural choices include shared weights and capacity controls that affect overfitting and validation performance.
- Core modules: The Perceiver consists of a cross-attention module followed by a Transformer, with inputs transformed into query, key, and value representations.The attention channel count is typically set by the minimum of the input channels, such as 261 for ImageNet.
- Latent initialization: The first cross-attention layer uses learned per-element latent queries, while the latent array is initialized from a truncated normal distribution.For ImageNet, the learned query array has shape 512×1024; the initialization has mean 0 and standard deviation 0.02.
- Capacity and regularization: Increasing latent index dimension, self-attends, and cross-attends generally improves accuracy, but increasing latent channel dimension can cause overfitting.These trends come from the scaled-down ImageNet hyperparameter study.
- Transformer blocks: Self-attention applies normalized inputs to QKV projections, and each attention block is followed by a dense GELU multilayer perceptron.The dense block uses layer normalization, two linear layers, and a GELU activation; dropout was omitted after degrading performance.
- Transformer blocks: All linear layers preserve input dimensionality and are tiled over index dimensions as 1 × 1 convolutions.This applies to query, key, value, and dense-block layers.
- Output construction: Averaging final latent self-attention outputs over the index dimension produces a global summary vector that is projected to class logits.This converts the latent representation into the target classification output.
- Capacity and regularization: Weight sharing between later cross-attention modules and corresponding latent Transformer blocks mitigates overfitting and improves ImageNet validation performance.The reported architecture uses eight cross-attends and six blocks per latent Transformer.
D. Position encodings and Fourier features
Position encodings use crop-relative coordinates, while Fourier features expose frequency resolution as tunable design parameters. The ablations favor more bands and higher maximum resolution up to the Nyquist limit, provided computation remains feasible.
- Fourier features: The Fourier parameterization controls maximum frequency independently from band count, allowing the number of bands to track computational budget.More bands generally helped when the resulting architecture fit in memory.
- Fourier features: Increasing Fourier bands and maximum resolution generally improves performance up to the Nyquist limit, with similar effects for linear and logarithmic band spacing.The initialization scale of 0.1 diverged during training in the corresponding experiment.
E. Audiovisual attention maps
The audiovisual Perceiver’s attention maps respond to both static and dynamic video features, while audio maps combine positional banding with content-related modulation.
- The audiovisual model visualizes attention separately over video and audio inputs across its first and second cross-attention modules.Video uses space-time patches, while audio is represented through raw audio samples and mel-spectrogram visualizations.
- Video attention maps are sensitive to both static and dynamic features and can exhibit spatiotemporal structure.The learned attention adapts to input content rather than using a fixed bank of spatiotemporal filters.
- Audio attention maps show Fourier-frequency positional banding combined with content-related modulation.Mel-spectrograms plot time on the x-axis and frequency on the y-axis.
F. Notes on changes from the original version
The revised version corrects the AudioSet mAP evaluation and reports slightly improved ImageNet results after implementation changes. The accompanying figures show attention maps across ImageNet and AudioSet settings, including video, audio, and multimodal inputs.
- The AudioSet mAP results were recomputed after an error transposed class scores and examples during sklearn average-precision calculation.The original results were unfortunately higher than the corrected numbers.
- ImageNet results slightly improved after removing dropout, removing an unintended linear layer, and changing the order of averaging and projection for output logits.
- The figures visualize ImageNet attention maps from first and final cross-attends under Fourier-feature and learned position encodings.
- The figures also visualize AudioSet attention maps for video-only, mel-spectrogram-only, and video-plus-mel-spectrogram inputs.For multimodal inputs, attention is shown separately over video and mel-spectrogram subsets, including car and plane examples.