Source-linked AI summary
Pretrained Transformers as Universal Computation Engines
Kevin Lu, Aditya Grover, Pieter Abbeel, Igor Mordatch
TL;DR
The paper investigates whether natural-language pretraining transfers to non-language sequence tasks without finetuning self-attention or feedforward layers. It introduces FPT, which minimally finetunes GPT-2 across numerical, vision, and protein tasks, and finds comparable performance across all seven tasks, with faster convergence and advantages over LSTMs on some long-sequence tasks.
Problem
The paper asks whether a transformer pretrained on natural language can generalize to other modalities with minimal finetuning, addressing limited evidence on cross-modal downstream transfer.
Method
FPT finetunes GPT-2's linear input/output layers, positional embeddings, and layer-normalization parameters while evaluating seven classification tasks spanning numerical computation, vision, and protein fold prediction.
Results
FPT achieves comparable performance to fully trained transformer benchmarks across all seven tasks and converges faster during training.
Takeaways & Limitations
Language-pretrained transformers can perform computation across diverse non-language modalities while leaving self-attention and feedforward layers frozen.
Takeaways & Limitations
Fully trained transformer comparisons are constrained by instability, divergence, or overfitting on some relatively small datasets, and reported baselines use different model sizes or tuning procedures.
Abstract
from arXiv · showhide
We investigate the capability of a transformer pretrained on natural language to generalize to other modalities with minimal finetuning -- in particular, without finetuning of the self-attention and feedforward layers of the residual blocks. We consider such a model, which we call a Frozen Pretrained Transformer (FPT), and study finetuning it on a variety of sequence classification tasks spanning numerical computation, vision, and protein fold prediction. In contrast to prior works which investigate finetuning on the same modality as the pretraining dataset, we show that pretraining on natural language can improve performance and compute efficiency on non-language downstream tasks. Additionally, we perform an analysis of the architecture, comparing the performance of a random initialized transformer to a random LSTM. Combining the two insights, we find language-pretrained transformers can obtain strong performance on a variety of non-language tasks.
1 Introduction
The paper asks whether transformers pretrained on natural language can transfer useful representations across modalities. It hypothesizes that self-attention can learn features useful for arbitrary sequential data and tests this with minimally finetuned GPT-2.
- Transformers have succeeded across language, images, proteins, behaviors, and multimodal tasks, motivating study of cross-modality transfer.
- Prior pretrained-model studies generally evaluate downstream tasks in the same modality as pretraining, leaving cross-modal generalization less explored.
- The paper hypothesizes that self-attention pretrained on plentiful natural-language data can identify representations useful for arbitrary data sequences.
- FPT finetunes only input/output layers, positional embeddings, and layer-normalization parameters while evaluating numerical, vision, and protein-fold tasks.Across these tasks, FPT shows comparable performance to fully trained transformers or LSTMs, matches reported benchmarks, and converges faster.
2 Methodology
The methodology evaluates language-pretrained transformers on diverse sequence tasks by adapting only lightweight input, output, normalization, and positional components. The core transformer computations and token-to-token communication remain frozen.
- 2.1 Tasks: The evaluation covers classification across numerical computation, vision, and protein fold prediction to test representations across diverse modalities.
- 2.1 Tasks: Bit memory presents five length-1000 bitstrings, then requires reconstructing one partially masked string from 120 tokens of dimension 50.
- 2.1 Tasks: Bit XOR requires predicting the element-wise XOR of two length-5 bitstrings represented as 10 tokens of dimension 1.
- 2.1 Tasks: ListOps tests parsing and evaluation of long mathematical expressions using 512 tokens of dimension 15.
- 2.1 Tasks: MNIST and CIFAR-10 use 4 × 4 image patches, with models receiving 64 tokens of dimension 16.CIFAR-10 LRA instead converts images to grayscale and flattenes them into 1024 tokens of dimension 1, reducing spatial inductive bias.
- 2.2 Architecture: FPT adapts GPT-2 through input and output layers, layer norms, and positional embeddings while freezing self-attention and feedforward layers.For the 124M-parameter base CIFAR-10 model, these trainable components comprise approximately 0.086% of the network; the share falls to 0.029% for GPT-2 XL.
- 2.2 Architecture: Because input and output layers are not connected to multiple tokens, all token-to-token communication occurs through frozen attention connections.
3 Empirical Evaluations
Across seven tasks, FPT achieves performance comparable to fully trained transformer benchmarks while often converging faster and outperforming LSTMs on long-sequence tasks. The experiments also show benefits from language pretraining, interpretable frozen attention patterns, and increased model capacity, alongside underfitting and tuning limitations.
- FPT achieves comparable performance to fully trained transformer benchmarks across all seven evaluated tasks.
- 100% performance on Bit Memory and Bit XOR shows that the models recover the exact algorithms.FPT also recovers the XOR algorithm for sequence lengths greater than n = 256.
- Language pretraining improves performance and convergence compared with random transformer initialization, including a considerable accuracy gap on MNIST.All tested pretraining modes converge faster than randomly initialized transformers, while Bit Memory pretraining is 6× slower than FPT on Bit XOR.
- FPT produces interpretable frozen attention patterns, whereas a random transformer solves Bit XOR with a weaker, less semantically interpretable diagonal pattern.FPT learns to attend to the correct Bit Memory string by similarity to the inputs rather than relying solely on position.
- FPT models underfit, and increasing model capacity improves performance more stably than fully training larger transformers, which can overfit, diverge, or become unstable.The reported CIFAR-10 FPT result uses a 36-layer large model, while literature baselines use extensive hyperparameter sweeps and task-specific tuning.
4 Related Work and Discussion
Prior work established transformers across language, vision, proteins, and multimodal tasks, while this paper focuses on transferring a pretrained transformer between more distant modalities. It reframes frozen-model evaluation around generalizable computation and relates FPT to adapters, reservoir computing, and studies of self-attention computation.
- Transformers have been applied successfully to language, vision, protein, behavior, and multimodal tasks.
- Most related transformer transfer work studies in-modality adaptation or uses separate transformers for different modalities.
- Prior LSTM work found transfer from non-linguistic data with latent structure to language, whereas this paper examines transfer in the opposite direction and across more distant modalities.
- Adapters commonly add a task-specific fully connected residual block and finetune layer-normalization parameters during transformer adaptation.
- Self-attention has been interpreted as supporting optimization steps, Hopfield-network updates, and storage or retrieval of large amounts of information.
- For cross-modality transfer, the paper finetunes input and output layers while evaluating whether frozen intermediate layers perform generalizable computation.
- FPT resembles reservoir and echo-state computing because a recurrent computation is frozen and trainable readout parameters are used for downstream tasks.
5 Conclusion
The paper concludes that pretrained transformer language models can transfer to non-language tasks while leaving self-attention and feedforward layers frozen. It identifies future opportunities alongside limits from restricted evaluation and potential bias transfer in high-stakes applications.
- Pretrained transformer language models achieved competitive performance on non-language downstream tasks without finetuning self-attention or feedforward layers.
- Future work could investigate other pretraining modalities, hybrid domains, and frozen pretrained models for reinforcement learning.
- The analysis is limited to specific models and a restricted set of tasks, so behavior may differ for other architectures or tasks.
- High-stakes applications raise concerns about transferring harmful biases from pretrained models trained on vast unlabeled, uncurated datasets.
Parameter ablations for pretrained models
The pretrained-model ablations separately evaluate individual parameter types and progressively add parameter groups to the finetuning set.
- Table 18 reports ablations that finetune individual parameter types in pretrained frozen transformers.
- The reported comparison identifies the most important parameter for each task by highest test accuracy.
- Table 19 reports ablations that successively add parameter groups to pretrained frozen transformers.
Parameter ablations for random models
The random-model ablation section reports individual and cumulative parameter-freezing comparisons, alongside descriptions of the GPT-2 transformer components used in the experiments.
- Parameter ablations: Table 20 reports finetuning individual parameter types for random frozen transformers.
- Parameter ablations: Table 21 reports ablations that successively add parameter groups for random frozen transformers.
- Transformer architecture: The experimental architecture uses GPT-2, whose self-attention maps input tokens into queries, keys, and values before weighted aggregation.
- Transformer architecture: GPT-2 uses multi-head attention, residual connections, a two-layer MLP, stacked transformer blocks, positional embeddings, and layer normalization.
- Transformer architecture: Causal masking restricts output token i to attend only to input tokens j ≤ i, supporting autoregressive language modeling.
- Transformer architecture: Layer normalization applies learned affine scale and shift parameters and is used twice per self-attention block.
B.5 Model Sizes
This section provides model-size information and notes that other architectures use the base model size’s hyperparameters.
- Table 22 lists hyperparameters for architectures at larger model sizes.
- Experiments with Vision Transformer, BERT, Longformer, and T5 use the base model size’s hyperparameters.
C Experimental Details
The experiments use Hugging Face implementations, Adam optimization, and mostly fixed learning-rate settings, while comparing FPT, fully trained transformer, and LSTM configurations.
- All models use Adam with PyTorch defaults, and transformer models generally use a learning rate of 10^-3 without scheduling.Remote homology instead uses 10^-4 because it performed better than 10^-3.
- The FPT configuration is a 12-layer base-size model finetuning input, output, position, and layernorm parameters.
- The Full configuration is a fully trained GPT-2 or reported vanilla-transformer baseline, depending on the comparison.
- The LSTM baseline uses 3 layers and a 768-dimensional hidden state while training all parameters.
All tasks
Across tasks, the study compares FPT with randomly initialized, Bit Memory-pretrained, and vision-pretrained transformer models under specified finetuning protocols.
- FPT uses a 12-layer base-size model finetuning input, output, position, and layernorm parameters.Its CIFAR-10 model size differs from the Table 1 configuration.
- Random uses a 12-layer randomly initialized base-size GPT-2 model with selected input, output, position, and layernorm parameters trained.
- Bit uses a 12-layer base-size GPT-2 model after full finetuning on Bit Memory, followed by finetuning selected parameters.
- ViT is a 12-layer, 768-dimensional ImageNet-21k-pretrained model with 16-pixel patches and a task-specific reinitialized input layer.It omits a CLS token and MLP output network, using a linear layer from the last token.
D.3 How important is the transformer architecture compared to LSTM architecture?
This section compares transformer and LSTM architectures using matched or modified depth, residual-connection, and positional-embedding configurations.
- D.3 How important is the transformer architecture compared to LSTM architecture?: The transformer comparison uses a 12-layer randomly initialized base-size GPT-2 model, while the standard LSTM has 3 layers and no residual connections or positional embeddings.
- D.3 How important is the transformer architecture compared to LSTM architecture?: LSTM* is a 12-layer, 768-dimensional model that trains input, output, position, and layernorm parameters.
- D.3 How important is the transformer architecture compared to LSTM architecture?: The depth comparison evaluates standard LSTMs with 12 and 3 layers, both using 768-dimensional hidden states.
- D.3 How important is the transformer architecture compared to LSTM architecture?: The ablation adds residual connections and then positional embeddings to a 12-layer, 768-dimensional standard LSTM.The final configuration matches LSTM* from Table 3.
D.4 Does language pretraining improve compute efficiency over random initialization?
The section refers to Table 6 and identifies the FPT and Random model configurations used for comparison.
- Table 6 is referenced as the source for this section’s results.
- FPT is a 12-layer base-size model with input, output, positional, and layer-norm parameters finetuned.
- Random is a 12-layer randomly initialized base-size GPT-2 model trained on input, output, positional, and layer-norm parameters.