PaperlayerView exact revision on arXiv

Source-linked AI summary

FAAST: Forward-Only Associative Learning via Closed-Form Fast Weights for Test-Time Supervised Adaptation

Guangsheng Bao, Hongbo Zhang, Han Cui, Ke Sun, Yanbin Zhao, Juncai He, Yue Zhang

arXiv:2605.04651v2cs.LGcs.CL

TL;DR

Adapting pretrained models is costly with backpropagation and memory- or context-based inference. FAAST instead compiles labeled examples into fast weights using forward-only associative learning, achieving efficient adaptation and competitive performance across supervised benchmarks.

  • Problem

    Pretrained-model adaptation faces high training costs from backpropagation and heavy inference overhead from memory- or context-based approaches.

  • Method

    FAAST separates frozen representation learning from associative learning and analytically compiles labeled key-value pairs into task-specific fast weights in a single pass.

  • Results

    Across classification and sequence modeling tasks, FAAST matches or exceeds backprop-trained and in-context-learning baselines while substantially reducing computation.

  • Takeaways & Limitations

    FAAST offers a computationally efficient, memory-light adaptation paradigm for deploying pretrained models across many downstream tasks and resource-constrained settings.

  • Takeaways & Limitations

    FAAST relies on task-relevant features in frozen pretrained representations and may be less effective for tasks requiring compositional reasoning or hierarchical dependencies.

Abstract

from arXiv · show

Adapting pretrained models typically involves a trade-off between the high training costs of backpropagation and the heavy inference overhead of memory-based or in-context learning. We propose FAAST, a forward-only associative adaptation method that analytically compiles labeled examples into fast weights in a single pass. By eliminating memory or context dependence, FAAST achieves constant-time inference and decouples task adaptation from pretrained representation. Across image classification and language modeling benchmarks, FAAST matches or exceeds backprop-based adaptation while reducing adaptation time by over 90% and is competitive to memory/context-based adaptation while saving memory usage by up to 95%. These results demonstrate FAAST as a highly efficient, scalable solution for supervised task adaptation, particularly for resource-constrained models. We release the code and models at https://github.com/baoguangsheng/faast.

1. Introduction

FAAST addresses the cost of backpropagation and the memory or context requirements of existing adaptation methods by compiling labeled examples into fast weights. It enables single-pass adaptation and constant-cost inference while performing competitively across image, language, and sequence tasks.

  • Backpropagation-based adaptation becomes expensive when repeated gradient computation, optimizer-state maintenance, and iterative updates are required across many downstream or online tasks.
  • FAAST separates fixed representation learning from task-specific associative learning between pretrained input and output embeddings.
  • FAAST compiles labeled key-value pairs from frozen encoders into fast weights through a single-pass, gradient-free procedure with constant-cost inference.
  • Unlike memory- or context-based adaptation, FAAST discards stored key-value pairs after compilation and retains only the computed fast weights.
  • On image classification benchmarks, FAAST matches backprop-based accuracy while saving 95% learning time, and on language modeling it saves more than 93% training and inference cost versus memory/context adaptation.
  • FAAST achieves consistently better full-set performance than LLM zero-shot and few-shot in-context-learning baselines on sequence-to-sequence machine translation tasks.

2. Related Work

FAAST builds on associative learning, fast weights, frozen representations, and forward-only learning while distinguishing itself through analytical adaptation on fixed pretrained representations. Its design avoids iterative optimization and context- or memory-dependent inference.

  • Prior work combines associative memory, frozen representations, forward-only rules, or pseudoinverse solutions, but these components are generally explored separately.
  • FAAST performs associative learning analytically on fixed pretrained representations, enabling single-pass, optimizer-free adaptation.
  • Unlike forward-forward and feedback-alignment methods, FAAST targets downstream task adaptation rather than representation training from scratch and does not require multiple forward passes or specialized objectives.
  • Unlike adapters, LoRA, and prefix tuning, FAAST does not depend on gradient-based optimization; unlike in-context and memory methods, it avoids conditioning on or querying stored examples during inference.

3. Preliminaries

The paper formulates supervised adaptation as learning an input-to-output association between embeddings produced by frozen encoders. It contrasts gradient-trained projections and explicit memory retrieval with associative mappings that can be computed from key-value pairs.

  • Each supervised example consists of an input xi and supervision signal yi, such as an image or text input and its label or next token.
  • Pretrained frozen encoders map inputs and outputs into fixed-dimensional embeddings, producing a key-value pair for each labeled example.
  • A linear projection W maps input embeddings to output embeddings, with normalized embeddings allowing the bias term to be omitted.
  • For classification, an attention head converts the projected output into class probabilities, while sequence modeling uses the same projection and softmax formulation for next-token prediction.
  • The conventional projection is learned by minimizing cross-entropy with gradient-based optimization, requiring iterative backpropagation for each downstream task.
  • Explicit memory and in-context methods store labeled key-value examples and retrieve relevant values for a query using similarity matching or self-attention.

4. Method

FAAST separates representation learning from associative adaptation, computing task-specific fast weights analytically from frozen representations. Its incremental and spectral mechanisms support scalable integration while controlling generalization.

  • FAAST Module: FAAST computes task-specific fast weights in closed form, providing a deterministic single-pass alternative to gradient-based adaptation.The solution uses the Moore–Penrose pseudoinverse and requires only one forward pass over the data.
  • FAAST Module: Once representations are fixed, FAAST solves the associative input-output mapping directly rather than approximating it through stochastic gradient descent.The mapping is formulated as a linear regression problem over stored key-value pairs.
  • Incremental Updates: Incremental updates aggregate new associative evidence without retaining all past key-value pairs, addressing adaptation at classification and language-model scales.The stated scale ranges from up to 10^6 classification pairs to approximately 10^10 language-model tokens.
  • Spectral Filtering: Spectral filtering suppresses unstable small-singular-value directions, balancing underfitting and overfitting across data regimes.Large singular values capture dominant, generalizable directions, whereas small values can encode noise or idiosyncratic examples.
  • Integration: In pretrained classifiers and language models, FAAST adds fast-weight outputs through interpolation or residual readouts while keeping pretrained components largely fixed.For classifiers, increasing memory size shifts predictions from prior-dominated toward task-specific adaptation; language-model readouts are trained once and folded at inference.
  • Integration: A lightweight key-value importance scorer modulates individual associations so informative pairs contribute more during fast-weight construction.The scorer is a sigmoid-activated linear classifier over concatenated keys and values.

5. Experiments on Supervised Classification Tasks

The classification experiments compare associative adaptation with gradient, zero-shot, and memory-based baselines using frozen pretrained features. FAAST remains competitive without gradients and is especially robust in low-data and arbitrary-label settings.

  • Experimental Setup: FAAST is evaluated against gradient-based, zero-shot, and memory baselines on identical pretrained features across CIFAR-10 and mini-ImageNet.The experiments cover 5-shot episodic and full-data regimes with standardized hyperparameters.
  • Experimental Setup: The evaluation reports classification accuracy, inference computation, and memory usage while isolating the projection layer.This setup tests adaptation without gradients, optimizer state, or multiple training epochs.
  • Results: FAAST consistently outperforms CLIP zero-shot baselines and improves smoothly from few-shot to full-data regimes.The result is reported across the image-classification settings examined.
  • Results: FAAST is more robust in low-data settings, where linear probing and full finetuning tend to overfit, while remaining competitive at scale.This comparison concerns backprop-based adaptations.
  • Results: 86.8% accuracy on mini-ImageNet with WordNet IDs shows FAAST can generalize under arbitrary label assignments where zero-shot transfer fails.The result indicates adaptation beyond pretrained semantic priors.

6. Experiments on Sequence Modeling Tasks

Sequence-modeling experiments test FAAST for autoregressive language modeling and conditional translation. Across WikiText-103 and IWSLT2017, the method improves or matches baselines while avoiding context-dependent adaptation costs.

  • Experimental Scope: The sequence experiments assess test-time adaptation, temporal credit assignment, long-range dependencies, and cross-task generalization.Language modeling uses autoregressive models, while translation tests structured input-output associations.
  • Language Modeling: FAAST is evaluated across GPT-2 sizes from 117M to 1.5B against zero-shot, gradient-based, in-context, and kNN-LM baselines on WikiText-103.Perplexity is measured on the held-out test split after adapting on the training split.
  • Language Modeling: 13.23 perplexity with an in-domain-trained readout matches or exceeds backprop-based adaptation, while FAAST uses far less memory than memory baselines.The method improves over zero-shot GPT2-XL with modest inference overhead.
  • Machine Translation: FAAST improves translation across all evaluated language pairs and adaptation settings on IWSLT2017.With Qwen2.5-3B, full-data adaptation gains more than 3 BLEU points for De-En, En-Fr, and Fr-En, and at least 2 BLEU points for the others.

7. Discussion and Limitations

FAAST offers efficient supervised adaptation through associative mappings over frozen representations, but its scope depends on representation quality and task structure.

  • Limitations: FAAST may fail when frozen encoders do not capture task-relevant features.The method therefore relies on the quality of pretrained representations.
  • Limitations: FAAST may be less effective for compositional reasoning, hierarchical dependencies, or long-range planning than iterative or gradient-based refinement.These task types are identified as settings where the method's associative approach is less suitable.
  • Limitations: Current evaluations focus on classification and sequence modeling, leaving more complex structured prediction and multimodal tasks for future investigation.The stated evaluation scope limits the demonstrated generality of the approach.

8. Conclusion

FAAST enables rapid, forward-only task adaptation without backpropagation or context-dependent inference. Across classification and sequence modeling, it matches or exceeds baseline performance while reducing computation and memory usage.

  • FAAST performs single-pass, forward-only associative adaptation without backpropagation or context-dependent inference.It provides constant inference cost for task-specific learning.
  • FAAST matches or exceeds backprop-trained and in-context learning baselines across classification and sequence modeling tasks.
  • FAAST substantially reduces computation and memory usage during adaptation and inference.
  • Associative, gradient-free adaptation is presented as a practical alternative for multi-task, online, and test-time deployment scenarios.
  • Future work will extend FAAST to structured prediction, compositional reasoning, and multimodal tasks.

Impact Statement

FAAST targets deployment bottlenecks by combining forward-only associative adaptation, compressed non-parametric storage, and inference without memory access. Its supported scope is downstream adaptation of pretrained representations, with extensions to other task types left for future work.

  • FAAST eliminates gradient-based optimization and context-length-dependent inference for downstream adaptation.
  • The paper identifies structured prediction, compositional reasoning, and multimodal tasks as future extensions.
  • FAAST combines forward-only learning, closed-form associative memory, non-parametric storage, and inference without memory access.
  • Unlike adapters, LoRA, and prefix tuning, FAAST computes task-specific mappings analytically without downstream parameter training.
  • Unlike memory-augmented and retrieval-based methods, FAAST removes the need for inference-time memory access.
  • FAAST targets downstream task adaptation on pretrained representations rather than training representations from scratch.

B.1. Fast Weights as Optimal Solutions to Regression

FAAST formulates fast weights as closed-form least-squares solutions over key-value pairs, connecting them to pseudoinverse attention and exact incremental updates. The framework also explains linear interpolation and supports deterministic online adaptation.

  • B.1. Fast Weights as Optimal Solutions to Regression: FAAST represents key-value pairs with matrices K and V and learns a linear predictor W through a least-squares objective.
  • B.1. Fast Weights as Optimal Solutions to Regression: The unique minimum-norm global minimizer is W⋆ = K†V.
  • B.1. Fast Weights as Optimal Solutions to Regression: Fast weights exactly match gradient-trained linear prediction while avoiding gradients, stochasticity, and iterative updates.
  • B.2. Fast Weights as Pseudoinverse Attention: The closed-form solution can be interpreted as pseudoinverse attention, retrieving values through a least-squares criterion.
  • B.2. Fast Weights as Pseudoinverse Attention: Pseudoinverse attention permits negative weights, enabling additive and subtractive interactions beyond standard nonnegative attention.
  • B.3. Incremental Update Rule for Fast Weights: Sufficient statistics update additively, and Sherman–Morrison–Woodbury enables exact fast-weight updates without recomputing from scratch.
  • B.3. Incremental Update Rule for Fast Weights: Under stated conditions, combining task contributions linearly induces linear interpolation of fast weights and supports smooth task mixtures.
  • B.3. Incremental Update Rule for Fast Weights: The batch-wise formulation supports continual, multi-task, and test-time adaptation through one deterministic update rule.

C. Experimental Setup

Experiments evaluate FAAST on frozen visual and language representations against zero-shot, gradient-based, memory, and in-context baselines. The setup also tests negative attention weights and reports substantial learning-cost reductions.

  • Image classification: Image experiments use frozen CLIP ResNet-50 encoders, with image embeddings as keys and class-prompt text embeddings as values.
  • Image classification: Image baselines include CLIP zero-shot, backprop-trained linear projection and full finetuning, k-NN memory, and softmax memory.
  • Image classification: Image evaluation covers CIFAR-10 and mini-ImageNet under few-shot episodic and full-data settings.
  • Language modeling: Language experiments use frozen GPT-2 and Qwen2.5 models, comparing FAAST with zero-shot, gradient-based, and in-context adaptation.
  • Learning cost: 93% and 96%: FAAST reduces learning time by over 93% on CIFAR-10 and over 96% on mini-ImageNet.
  • Learning cost: FAAST uses a single-pass closed-form update without gradient computation, optimizer state, or repeated epochs.
  • Negative attention weights: Centralization strengthens attention contrast, while negative weights suppress irrelevant memory items and improve retrieval performance.

D.1.3. ANALYSIS OF GENERALIZATION

FAAST generalizes across few-shot image classification and language modeling settings while exposing explicit controls for memorization, memory updates, and architectural extensions. Its gains include lower variance, improved perplexity, and substantial computational savings relative to backpropagation and memory-based methods.

  • Few-Shot Generalization: FAAST matches or outperforms backpropagation-trained linear projections under 256-shot settings with lower accuracy variance across runs.The comparison is summarized in Figure 4, whose standard deviation represents accuracy variance across episodes.
  • Arbitrarily Defined Labels: 6.4% accuracy with WordNet IDs is near the 5% random baseline for 20-way mini-ImageNet classification, while FAAST does not rely on prior semantic alignment.The result uses arbitrary class names and evaluates FAAST without prior, N0 = 0.
  • Overfitting–Underfitting Trade-off: Singular-value filtering through a relative tolerance threshold controls the trade-off between memorization and generalization.The pseudoinverse uses an SVD of the key matrix, where large singular values represent shared components and small singular values capture sample-specific variation.
  • Computational Cost: 0.2 GPU hours reduces adaptation cost by 93.3% versus backpropagation while matching the fastest inference speed and reducing memory-based inference latency by 96.5%.Table 10 compares FAAST with backpropagation-based methods such as LoRA and memory-based methods such as kNN-LM.
  • Language Modeling Generalization: 5.8% to 11.8% relative perplexity reductions are observed across GPT-2 sizes, with additional memory layers helping until gains plateau.The results indicate diminishing returns beyond moderate memory depth.
  • Architectural Variants: A right-to-left encoder reduces GPT2-XL WikiText-103 perplexity from 15.35 to 15.15, while attention-based readout lowers perplexity by roughly 0.08.The right-to-left encoder adds parameters and is therefore excluded from the main experiments; attention-based readout offers modest improvements.
Loading 2605.04651v2…