PaperlayerView exact revision on arXiv

Source-linked AI summary

Perceiver IO: A General Architecture for Structured Inputs & Outputs

Andrew Jaegle, Sebastian Borgeaud, Jean-Baptiste Alayrac, Carl Doersch, Catalin Ionescu, David Ding, Skanda Koppula, Daniel Zoran, Andrew Brock, Evan Shelhamer, Olivier Hénaff, Matthew M. Botvinick, Andrew Zisserman, Oriol Vinyals, Joāo Carreira

arXiv:2107.14795v3cs.LGcs.CLcs.CVcs.SDeess.AS

TL;DR

Existing neural architectures often assume stereotyped modalities and tasks or scale poorly with large inputs and outputs. Perceiver IO adds queried decoding to the Perceiver’s latent architecture, enabling arbitrary structured outputs with linear scaling in input and output sizes. It achieves broad results across language, vision, multimodal, multitask, and symbolic settings, including competitive GLUE performance without tokenization and state-of-the-art Sintel optical flow.

  • Problem

    Existing architectures bake in domain and task assumptions or scale poorly when inputs and outputs become large, diverse, or structured.

  • Method

    Perceiver IO encodes arbitrary inputs into a latent array, processes the latents, and decodes arbitrary outputs through queries specifying each output’s semantics.

  • Results

    Perceiver IO achieves strong results across language, visual, multimodal, multitask, and symbolic tasks, including GLUE performance matching BERT at a given FLOPs budget without tokenization and state-of-the-art Sintel optical flow.

  • Takeaways & Limitations

    The architecture offers a unified interface for diverse inputs and outputs while simplifying multimodal and multitask neural pipelines.

Abstract

from arXiv · show

A central goal of machine learning is the development of systems that can solve many problems in as many data domains as possible. Current architectures, however, cannot be applied beyond a small set of stereotyped settings, as they bake in domain & task assumptions or scale poorly to large inputs or outputs. In this work, we propose Perceiver IO, a general-purpose architecture that handles data from arbitrary settings while scaling linearly with the size of inputs and outputs. Our model augments the Perceiver with a flexible querying mechanism that enables outputs of various sizes and semantics, doing away with the need for task-specific architecture engineering. The same architecture achieves strong results on tasks spanning natural language and visual understanding, multi-task and multi-modal reasoning, and StarCraft II. As highlights, Perceiver IO outperforms a Transformer-based BERT baseline on the GLUE language benchmark despite removing input tokenization and achieves state-of-the-art performance on Sintel optical flow estimation with no explicit mechanisms for multiscale correspondence.

1 INTRODUCTION

Perceiver IO targets a single architecture that can handle diverse input modalities and structured output tasks without task-specific engineering. It extends the Perceiver with queried decoding while retaining latent-space processing and demonstrates broad cross-domain results.

  • Existing systems commonly use modality-specific encoders, fusion networks, and task-specific readouts, with complexity increasing as inputs and outputs diversify.
  • The original Perceiver handles diverse inputs but is limited to simple output spaces such as classification.
  • Perceiver IO decodes structured outputs by querying the latent array with output-specific features that specify each output’s semantics.
  • Its read-process-write architecture encodes inputs, refines a latent representation, and decodes outputs while decoupling computation from input and output sizes.
  • The decoder maps latent representations to arbitrarily sized and structured outputs, supporting dense and multitask settings as a replacement for specialist networks.
  • Perceiver IO spans language, optical flow, multimodal autoencoding, classification, and symbolic tasks, including BERT and AlphaStar replacement settings.

2 RELATED WORK

Related work has pursued structured outputs, multimodal or multitask learning, and attention-based bottlenecks, but existing approaches often retain fixed modalities or domain-specific components.

  • Early neural architectures explored encoding and reproducing high-dimensional inputs, including images and structured outputs such as semantic segmentation.
  • Multi-domain approaches typically cover a fixed, predefined set of modalities using domain-specific networks.
  • Multi-task learning has become popular, while individual models generally achieve generality only within restricted settings.
  • Set Transformers and related models use learned inducing points and decoder queries to create bottlenecks and map sets to outputs with fewer elements.

3 THE PERCEIVER IO ARCHITECTURE

Perceiver IO maps arbitrary input arrays through a smaller latent space and uses output queries to produce arrays of the desired shape and semantics. Query construction adapts to positional, modality, task, and heterogeneous output structure.

  • The architecture builds on the Perceiver, which represents inputs as feature-vector elements and compresses them into a smaller latent array for processing.
  • A query array with one feature vector per desired output element attends to the latent array and yields an output array of the requested shape.
  • The latent size is separated from input and output sizes, avoiding the quadratic compute and memory scaling of homogeneous full-input Transformer attention.
  • Encoding, processing, and decoding: Perceiver IO encodes x ∈ R^(M×C) into z ∈ R^(N×D), processes the latent array, and decodes it into y ∈ R^(O×E).M, C, O, and E depend on task data, whereas N and D are hyperparameters chosen for tractable computation.
  • Query construction: Queries encode output structure through information such as spatial position or modality, allowing inputs and outputs with different shapes or spatial layouts.
  • Query construction: Outputs depend only on their query and the latent array, enabling parallel decoding and training-time subsampling for very large output arrays.

4 EXPERIMENTS

Perceiver IO is evaluated across language, optical flow, multimodal autoencoding, and additional domains to test its generality. Across these settings, it matches or improves on specialist and Transformer-based baselines while supporting byte-level and multitask processing.

  • Language: At a fixed FLOPs budget, Perceiver IO without tokenization matches a strong Transformer-based model trained with SentencePiece tokenization on GLUE.FLOPs are emphasized because they relate directly to training time, while Transformer compute grows quadratically with sequence length.
  • Language: 81.2 vs 81.1: Perceiver IO on SentencePiece inputs slightly outperforms a strong BERT baseline on GLUE.Its reduced latent size permits 26 processing layers versus BERT Base’s 12 layers at a similar FLOPs budget.
  • Language: 81.8 on GLUE: byte-level Perceiver IO scales with more FLOPs and remains on par with BERT using SentencePiece tokens.The model consumes and predicts raw UTF-8 bytes rather than using a fixed handcrafted vocabulary.
  • Language: Multitask Perceiver IO outperforms single-task approaches and matches using eight task-specific input tokens across all eight GLUE tasks.Its output queries decouple task outputs from the input array, which is useful when tasks are numerous or inhomogeneous.
  • Optical flow: Perceiver IO slightly improves on RAFT on Sintel and outperforms PWCNet on KITTI, achieving state-of-the-art performance on Sintel.final under the reported setting.It uses no cost volumes or explicit warping, is not explicitly hierarchical, and its latent representation does not maintain the inputs’ 2D layout.
  • Multimodal autoencoding: 45% top-1 accuracy with 20.7 PSNR for video: multimodal autoencoding shows a tradeoff between classification accuracy and reconstruction quality.Latent variables are shared across modalities, and changing loss weights shifts the balance among video, audio, and classification objectives.

5 CONCLUSION

Perceiver IO is presented as a general-purpose architecture for arbitrary inputs and outputs that scales linearly with input and output sizes. The paper evaluates this generality across additional domains and input configurations.

  • 5 CONCLUSION: Perceiver IO handles general-purpose inputs and outputs while scaling linearly in both input and output sizes.Its interface uses latent attention and learned queries for multimodal and multitask settings.
  • 5 CONCLUSION: Experiments cover ImageNet, StarCraft II, and AudioSet, extending evaluation across multiple domains.The paper also reports input and output sizes and processing choices for cross-domain comparison.
  • 5 CONCLUSION: Input key/value, positional encodings, and output queries are specified for each experiment to support comparison across domains.These details are organized in Tables 5 and 6.
  • 5 CONCLUSION: The experiments compare configurations ranging from no domain adaptation to moderate domain adaptation on most evaluated domains.Examples include tokenizer-free language, raw-pixel flow, and ImageNet without convolutional or patch-based preprocessing.

A IMAGE CLASSIFICATION

Perceiver IO evaluates its general decoder on ImageNet classification, where it consistently improves over the original Perceiver and remains competitive with Vision Transformer models. The experiments also test classification without architectural or feature-based 2D image information.

  • A IMAGE CLASSIFICATION: Perceiver IO consistently outperforms the original Perceiver on ImageNet classification.The comparison differs in the decoder, and neither model uses convolutional preprocessing by default.
  • A IMAGE CLASSIFICATION: Perceiver IO is competitive with Vision Transformer models even without relying on 2D convolutions.After JFT pretraining, it performs in the ballpark of models designed primarily for image classification.
  • A IMAGE CLASSIFICATION: Adding 2D convolutional preprocessing produces a moderate efficiency improvement and a performance bump.Perceiver IO remains compatible with convolutional preprocessing rather than requiring it architecturally.
  • A IMAGE CLASSIFICATION: A fully learned position encoding lets the model classify ImageNet without privileged information about 2D image structure.The experiment uses a 1D convolutional network to project RGB values before concatenation with the learned positional encoding.

A.1 DETAILS OF IMAGENET TRAINING

The ImageNet experiments use regularization, large-scale training, and optional preprocessing or pretraining configurations. Training speed and model settings are reported alongside these choices.

  • A.1 DETAILS OF IMAGENET TRAINING: ImageNet training uses RandAugment, CutMix, and MixUp regularization, with weight decay and gradient clipping but no dropout.The reported settings include weight decay of 0.1, maximum global gradient norm 10, and CutMix ratio 0.2.
  • A.1 DETAILS OF IMAGENET TRAINING: The reported ImageNet training-speed table includes Perceiver and Perceiver IO models, while baseline speeds were not reimplemented.The pretraining model is faster because it uses only 16 process modules.
  • A.1 DETAILS OF IMAGENET TRAINING: ImageNet models train for 110 epochs with batch size 1024 on 64 TPUs using LAMB and cosine learning-rate decay.The learning rate is 2 × 10^-3 for 55 epochs before decaying to zero over the remaining 55 epochs.
  • A.1 DETAILS OF IMAGENET TRAINING: JFT pretraining uses 300 million images spanning approximately 18,000 classes before ImageNet transfer.The pretraining schedule decays the learning rate to zero over 14 epochs.
  • A.1 DETAILS OF IMAGENET TRAINING: Perceiver IO matches the original Transformer on StarCraft II while using fewer FLOPs and parameters and requiring essentially no tuning.Overall system training speed does not change because the entity encoder is not the bottleneck.
  • A.1 DETAILS OF IMAGENET TRAINING: A single convolution plus max-pooling stage moderately improves ImageNet performance.Full images can otherwise be processed without convolutional pre- and post-processing because ImageNet inputs are relatively small.

B STARCRAFT II

Perceiver IO replaces AlphaStar’s Transformer entity encoder with a latent bottleneck while preserving the 512-entity input and output sizes. The resulting agent matches the original agent’s performance with fewer FLOPs.

  • Architecture replacement: The entity encoder processes a discrete, unordered set of game units represented by properties such as type, position, and health.Its outputs provide both a pooled state summary and per-unit representations used by a pointer network for unit selection.
  • Architecture replacement: Perceiver IO replaces AlphaStar’s Transformer entity encoder with a latent index dimension of 32 while retaining 512-unit inputs and outputs.Only the latent index dimension was swept, using values of 32 and 64; other hyperparameters were not tuned beyond this sweep.
  • Results: 87% win-rate versus the Elite bot was achieved after behavioral cloning on human data, matching the original AlphaStar agent’s performance.The comparison is reported for the resulting Perceiver IO agent and the original AlphaStar agent.
  • Results: 3× decrease in FLOPs was obtained relative to the original AlphaStar agent while reaching the same performance level.The paper reports theoretical FLOPs using separate counts for multiplies and accumulates; fused multiply-accumulate conventions would halve reported figures.
  • Attention and complexity: Encode attention maps inputs to a typically smaller latent index dimension, while decode attention maps latents to an often larger output index dimension.Process attention preserves the input index dimension and is implemented as self-attention.
  • Attention and complexity: QKV attention uses two matrix multiplications, and an attention module has overall time and memory complexity O(MNF) for input and latent index dimensions M and N.Perceiver IO uses cross-attention for encoder and decoder modules and self-attention for latent processing.

F.2 ARCHITECTURE DETAILS

The language-experiment architecture details are specified in Table 11, which documents Perceiver IO’s model configuration and training-speed setup.

  • Architecture details: The language experiments’ architecture hyperparameters and training speed are reported in Table 11.The passage points to the table rather than reproducing the settings.
  • Architecture details: Table 11 provides the Perceiver IO architecture details used in the language experiments.The supplied passage identifies the table’s purpose but does not enumerate its individual hyperparameter values.

F.3 MLM PRETRAINING

The MLM pretraining setup combines C4 and English Wikipedia with word-level masking, while byte-level BERT is narrowed to match the Perceiver IO FLOPs budget. Learned queries predict masked tokens at every position.

  • MLM pretraining: Pretraining uses 70% C4 and 30% English Wikipedia, concatenating 10 documents before splitting into crops.Each word is masked independently with probability 15%, with boundaries defined using whitespace.
  • MLM pretraining: The FLOPs-matched byte-level BERT reduces width, feed-forward size, layers, and heads to offset its 2048-byte sequence length.The resulting model has approximately the inference FLOPs of BERT Base on 512 tokens.
  • MLM decoding: Learned queries have the same dimension as the input array, with one output query per input position to predict every masked token.This setup uses M=O for masked language modeling.
  • MLM decoding: The first cross-attention layer learns both position-based and content-based look-ups.Position-based look-ups can be sparse and precise or more distributed and periodic.
  • Evaluation: GLUE evaluation excludes WNLI and uses task-specific metrics, including accuracy, Pearson correlation for STS-B, and Matthews correlation for CoLA.The average first averages MNLI-matched and MNLI-mismatched results before combining tasks.

F.5 ABLATION ON THE NUMBER OF LATENTS

The latent-count ablation studies the trade-off between latent count and latent width under a fixed FLOPs budget, comparing 128, 256, and 512 latents.

  • Latent-count ablation: Under a fixed FLOPs budget, increasing the number of latents requires adapting latent width to maintain comparable computation.The ablation varies the number of latents while matching the FLOPs budget through corresponding dimension changes.
  • Latent-count ablation: The experiment compares latent counts of 128, 256, and 512, with 256 identified as the best setting.The latent dimension is adapted for each count to match the computation budget.

G POSITIONAL ENCODINGS FOR IMAGE AND AUDIO EXPERIMENTS

Image experiments use 2D Fourier feature positional encodings based on sine and cosine bands, with frequencies linearly spaced between minimum and maximum values.

  • Image experiments use 2D Fourier feature positional encodings composed of sine and cosine frequency bands.The bands are linearly spaced from a minimum to a maximum frequency.
  • The encoding uses 64 sine/cosine bands per dimension in all image settings.
  • The minimum frequency is set to the minimum frequency of the input signal.

H OPTICAL FLOW: ADDITIONAL DETAILS AND RESULTS

Perceiver IO optical-flow experiments examine preprocessing, context, efficiency, and robustness across synthetic and real-world data. The results show benefits from frame concatenation and spatial context, while tiled inference remains necessary for varying input resolutions.

  • Optical-flow ablations: Convolutional downsampling and RAFT upsampling provide additional spatial context for both optical-flow inputs and queries.A lightweight configuration uses a 7×7 convolution, max pooling, Perceiver IO processing, and a RAFT upsampling layer.
  • Optical-flow ablations: Concatenating frames improved performance on the difficult Sintel.final and KITTI Flow 2015 datasets.The authors attribute the gain to increased spatial context and suggest the model compares spatial and temporal gradients.
  • Efficiency: 0.8 frames/sec is achieved by the most expensive model and 3.3 frames/sec by the lightweight model on a 2017 TITAN Xp.Perceiver IO is slower than RAFT on traditional GPUs, although the reported trend reverses on TPUs, the target architecture.
  • Qualitative results: Qualitative Sintel.final examples show flow propagation through heavy occlusion and low-texture regions, alongside handling of large motions and small objects.The examples include a blurred, minimally textured foreground person and motion involving very small objects.
  • Out-of-domain evaluation: Real-world videos show generally strong out-of-domain performance, but shadows and compression artifacts can produce incorrect or hallucinated flow.The model was evaluated on roughly 10 Getty Images videos after training only on synthetic imagery.
  • Evaluation setup: Tiled inference averages overlapping predictions because evaluation datasets have different resolutions, while input-shape invariance remains future work.The evaluation uses six evenly spaced tiles and weights predictions by distance from each tile’s nearest edge.
Loading 2107.14795v3…