PaperlayerView exact revision on arXiv

Source-linked AI summary

A Study of Autoregressive Decoders for Multi-Tasking in Computer Vision

Lucas Beyer, Bo Wan, Gagan Madan, Filip Pavetic, Andreas Steiner, Alexander Kolesnikov, André Susano Pinto, Emanuele Bugliarello, Xiao Wang, Qihang Yu, Liang-Chieh Chen, Xiaohua Zhai

arXiv:2303.17376v1cs.CVcs.AIcs.LG

TL;DR

Most work on multi-task computer-vision models presents a system without systematically answering design and trade-off questions. This paper studies autoregressive decoders with frozen pretrained vision encoders, finding that conditioned multi-task decoders can match or sometimes surpass heavily tuned single-task baselines.

  • Problem

    Most prior work presents multi-task computer-vision systems without systematically examining the benefits and pitfalls of autoregressive-decoder design choices.

    Source: §1 · p. 1

  • Method

    The paper systematically studies a multi-task autoregressive decoder trained on a frozen pretrained vision encoder across computer-vision tasks and design choices.

    Source: §1 · p. 1, §4 · p. 2

  • Results

    Conditioned multi-task decoders match heavily tuned single-task baselines and sometimes surpass them; unrelated tasks can be beneficial, and a two-layer decoder works well.

    Source: §1 · p. 1, §6 · pp. 8–9

  • Takeaways & Limitations

    A small decoder can read out answers from a broadly pretrained frozen encoder, although the approach still falls short on dense-output tasks.

    Source: §6 · p. 8

    Autoregressive output sequences make dense tasks such as panoptic segmentation computationally prohibitive at million-pixel output scale.

    Source: §5.10 · p. 8

Abstract

from arXiv · show

There has been a recent explosion of computer vision models which perform many tasks and are composed of an image encoder (usually a ViT) and an autoregressive decoder (usually a Transformer). However, most of this work simply presents one system and its results, leaving many questions regarding design decisions and trade-offs of such systems unanswered. In this work, we aim to provide such answers. We take a close look at autoregressive decoders for multi-task learning in multimodal computer vision, including classification, captioning, visual question answering, and optical character recognition. Through extensive systematic experiments, we study the effects of task and data mixture, training and regularization hyperparameters, conditioning type and specificity, modality combination, and more. Importantly, we compare these to well-tuned single-task baselines to highlight the cost incurred by multi-tasking. A key finding is that a small decoder learned on top of a frozen pretrained encoder works surprisingly well. We call this setup locked-image tuning with decoder (LiT-decoder). It can be seen as teaching a decoder to interact with a pretrained vision model via natural language.

1. Introduction

Autoregressive encoder–decoder models are converging as a way to unify vision tasks, but most prior work presents systems without systematically examining multi-tasking trade-offs. This paper studies those design choices and reports that multi-task LiT decoders can need less tuning while benefiting from unrelated tasks.

1. Introduction

  • Computer-vision models increasingly converge on Transformer architectures as interest grows in handling many tasks with little task-specific design.The introduction identifies autoregressive encoder–decoder Transformers as a model choice that reuses pretrained image encoders and models output sequences.Source: §1 · p. 1
  • Most prior work presents complete multi-task systems without explaining the benefits and pitfalls of autoregressive-decoder multi-tasking.The paper aims to fill this evidence gap through controlled study.Source: §1 · p. 1
  • A frozen pretrained vision encoder paired with a multi-task autoregressive decoder defines the paper’s LiT-decoder setup.The study examines conditioning, decoder capacity, cross-task transfer, and dataset-mixture strategies.Source: §1 · p. 1
  • Multi-task LiT decoders seem to require less hyperparameter tuning than single-task decoders, and even unrelated tasks can be beneficial.The introduction also reports that a two-layer decoder on a frozen backbone works well.Source: §1 · p. 1

2. Related work

The paper situates its study within broad multi-task-learning and computer-vision literatures, then focuses on autoregressive decoders as a promising unifying approach. It emphasizes understanding design effects rather than proposing a new method.

2. Related work

  • Prior multi-task-learning research is extensive, and the paper adopts a simple encoder–decoder setup to study design choices.The cited discussion notes that regularization and data augmentation may matter more for performance than more advanced methods [40].Source: §2 · p. 2
  • Computer vision spans classification, dense-output, and image–language subfields, with structured-output tasks increasingly cast as language.The passage cites examples across these areas [59] [2] [94] [23] [6] [54] [80] [12] [35] [53] [100].Source: §2 · p. 2
  • Rather than introduce a new method, the paper studies autoregressive decoders and excludes zero-shot transfer and individual-task fine-tuning.It highlights the apparent effectiveness of a frozen pretrained encoder with a very small decoder.Source: §2 · p. 2

3. Background

Autoregressive decoders model complex outputs as a sequence of conditional predictions, avoiding explicit scoring of every possible output. Transformer decoders also support cross-attention to image representations and prefix-based conditioning or auxiliary inputs.

3. Background

  • A probabilistic generative model learns p(y | x) and can score candidate outputs before decoding one instance.The section notes that this is straightforward for single-variable outputs such as classification.Source: §3 · p. 2
  • Complex outputs can be too numerous to score explicitly, and their components may need to be modeled jointly rather than independently.The passage motivates an implicit joint output distribution for structured outputs.Source: §3 · p. 2
  • Autoregressive decoders factor the output probability into successive predictions conditioned on the image and preceding output values.Causal self-attention Transformers run these predictors in parallel during training, while inference remains sequential.Source: §3 · p. 2
  • Transformer inputs can use cross-attention to encoded images or output prefixes for task conditioning and extra inputs such as VQA questions.The paper uses both mechanisms.Source: §3 · p. 2

4. Experimental setup

The experiments use a frozen pretrained image encoder with an autoregressive decoder across representative vision tasks, alongside heavily tuned single-task baselines. The setup specifies task prompting, training choices, and evaluation procedures.

4. Experimental setup

  • The study uses a WebLI-pretrained ViT-B/16 encoder, ten tasks across classification, captioning, OCR, and question answering, and a Base Transformer decoder.The encoder is frozen in most experiments; task images near-duplicates of pretraining data are removed, and sequence length is generally capped at 64 tokens.Source: §4 · pp. 2–3
  • A task-prompted 12-layer decoder on frozen image representations performs multiple tasks about as well as separate decoders, and task conditioning outperforms task-type or no conditioning.The main differences are reported for INet-1k and Pet.Source: Table 1 · p. 3
  • Classification, OCR, and QA use exact-match accuracy, while captioning uses CIDEr; decoding is greedy for classification and QA and four-beam for captioning and OCR.The decoder can output any string even for classification tasks.Source: §4 · p. 3

4.1. Single-task baselines

  • Single-task baselines are heavily tuned across learning rate, weight decay, epochs, model size, dropout, label smoothing, and augmentation choices.The best validation model is retrained with three seeds for average test performance; classification also includes linear, MLP, and attention-based classifiers.Source: §4.1 · p. 3

4.2. Multi-task LiT decoder training

  • Multi-task training samples examples from tasks according to task size and prompts the decoder with a task token and separator before the target.Training usually runs for 50 epochs; OCR-VQA avoids random cropping, and label smoothing is fixed at 0.1.Source: §4.2 · p. 3

5. Results and insights

The experiments compare multi-task LiT decoders with tuned single-task models and examine how task load, mixing, conditioning, and encoder choices affect performance. Results vary by task: task conditioning supports mostly on-par performance, while smaller decoders and frozen encoders have specific limitations on textual and dense-output tasks.

5.1. Single- vs multi-task

  • With task conditioning, one multi-task decoder performs mostly on par with separately tuned single-task decoders across the evaluated tasks.Category conditioning improves task performance over an unconditioned decoder, while task-specific conditioning can exploit each dataset’s characteristics.Source: §5.1 · p. 4

5.2. Scaling number of tasks and decoder depth

  • Classification-style tasks are largely unaffected by task count or decoder depth, while smaller decoders gradually lose performance on textual tasks as load increases.The 12-layer decoder stays mostly constant; exceptions include an ImageNet-1k-related dip and overfitting on the small starting task set.Source: Figure 1 · p. 4, §5.2 · p. 4

5.3. Task mixing strategies

  • Mixing strategy matters beyond classification, and visiting datasets with multiple labels per image more often can help if overfitting is controlled.Mixing tasks within a batch performs slightly better, while homogeneous batches offer flexibility to vary encoders or sequence lengths by task.Source: §5.3 · p. 5

5.4. On transferring skills across tasks

  • Across the transfer experiments, auxiliary OCR gains are at best comparable to an Alt-Text control, meaningless class tokens change performance only slightly, and adding languages improves performance by reducing overfitting.The language effect also holds for languages with non-Latin scripts; the authors report no apparent transfer of skills, only a regularizing effect from adding meaningful tasks.Source: §5.4 · p. 5, §5.4.1 · p. 5, §5.4.2 · p. 6

5.4.4 Multi-tasking reduces need for regularization; 5.5. Importance of image encoder pretraining

  • Multi-task training reduces sensitivity to regularization settings, and image/text-pretrained encoders outperform ImageNet-21k-pretrained alternatives on all tasks except ImageNet-1k, where DeiT-III is on par.The encoder comparison finds larger differences on captioning, OCR, and QA than on classification.Source: §5.4.4 · p. 6, §5.5 · p. 7

5.6. Task conditioning with prompt; 5.7. Frozen vs finetune image encoder

  • Task prompts improve task-specific output, but do not ensure task compliance for out-of-distribution images; distinct task position embeddings make no significant difference.Fine-tuning the encoder also works, but costs more than freezing it; the authors recommend a frozen encoder unless a task shows a clear quality drop.Source: §5.6 · p. 7, §5.7 · p. 7

5.8. Using autoregressive decoders at test-time; 5.9. Attending to the encoder tokens

  • For classification, decoding strategies have practically identical performance despite large compute-time differences; OCR and QA can benefit modestly from more expensive decoding, while encoder-output compression degrades quality.Captioning decoding trends are less clear; simple compression methods reduce performance, with larger degradation on non-classification tasks as compression increases.Source: §5.8 · p. 8, §5.9 · p. 8, Figure 9 · p. 8

5.10. Towards multi-tasking with dense targets

  • Dense panoptic segmentation exposes a limitation of frozen encoders: the model scores 5.0% PQ frozen and 32.1% after fine-tuning, with slight deterioration on other tasks.Autoregressive decoders also face sequence-length scalability limits for dense outputs; this experiment uses compressed mask targets and pseudo-labels.Source: §5.10 · p. 8

6. Conclusion

The study finds that a task-prompted multi-task decoder can match heavily tuned single-task baselines and sometimes surpass them. A small decoder on a frozen pretrained encoder works best when pretraining is broad, though dense-output tasks remain difficult.

  • A small decoder on a frozen encoder works surprisingly well when the encoder has broad pretraining such as CLIP-style image-text data, but performance falls short on dense-output tasks.Source: §6 · p. 8
  • A task-prompted multi-task decoder matches heavily tuned single-task baselines and sometimes surpasses them through positive transfer despite task conditioning.Source: §6 · pp. 8–9
  • Multi-task decoders seem less prone to overfitting and less sensitive to regularization hyperparameters, including when languages are treated as separate tasks.Source: §6 · p. 9

A. Datasets for multitask

The paper documents its task datasets and builds single-task baselines through validation-guided sweeps and repeated runs. For feedforward classification baselines, it compares classifier and pooling choices and reports better results with softmax cross-entropy and ScalingViT AdaFactor.

A. Datasets for multitask

B. Settings for single-task baselines

B.1. Feedforward classifiers

  • Across five classification tasks, the full cross-product sweep compared linear classifiers with MLPs using 1024- or 2048-dimensional hidden layers.Source: §B.1 · p. 13
  • The feedforward sweeps considered pooling and image augmentation settings, including no pooling and random crops with flips, with gradient clipping at 1.0 and 3- or 10-epoch runs.Source: §B.1 · p. 13
  • Softmax cross-entropy outperformed sigmoid cross-entropy, and ScalingViT AdaFactor outperformed SGD with momentum.Source: §B.1 · p. 13

B.2. Single-task decoder

Single-task decoder settings were explored with sweeps, while early experiments identified several augmentations and regularization choices that did not help consistently. Multi-task runs used shared defaults after initial settings proved to regularize too aggressively.

B.2. Single-task decoder

  • The single-task decoder sweep included resize-only, random, or inception crops and label smoothing values of 0.0 or 0.1.Source: §B.2 · p. 13
  • RandAugment hurt across the board, encoder dropout and horizontal flips on non-classification tasks had no effect or hurt, and β2 values of 0.999 versus 0.95 showed no noticeable difference.Source: §B.2 · p. 13

C. Settings for multi-task

  • Initial multi-task settings underfit through excessive regularization, so the study used learning rate 0.001, decoupled weight decay 0.0001, and dropout 0.1 as common defaults.Source: C. Settings for multi-task · pp. 13–14

D. Results for varying decoder depth

Reducing decoder depth to one or two layers preserves much of the reported performance. Single-task classification and captioning show no drop with one layer, while multi-task performance declines slightly but remains close enough for lightweight prototyping.

E. Results for various pretrained models

The section compares multi-task results across pretrained image encoders. A DeiT-III backbone improves ImageNet performance but performs worse on the other tasks than the less specialized ViT-AugReg backbone.

E. Results for various pretrained models

F. Results for frozen/finetune/from-scratch image encoder

This section presents single-task and multi-task results for frozen, fine-tuned, and from-scratch image encoders. It uses hyperparameter sweeps for the fine-tuned and from-scratch setups, with a default encoder learning-rate multiplier for fine-tuning.

F. Results for frozen/finetune/from-scratch image encoder

  • Table 7 compares single-task and multi-task results with frozen, fine-tuned, and from-scratch image encoders.Source: F. Results for… · p. 14
  • The fine-tuned and from-scratch setups use hyperparameter sweep ranges similar to those in Appendices B and C.Source: F. Results for… · p. 14
  • Fine-tuning applies a default 0.1 learning-rate multiplier to the encoder weights.Source: F. Results for… · p. 14

G. Results for panoptic segmentation task

Adding COCO panoptic segmentation yields very low panoptic performance with a frozen encoder while leaving other tasks roughly unchanged. Stronger encoder fine-tuning improves panoptic performance but deteriorates other tasks, and standard fine-tuning performs similarly to the frozen setup.

G. Results for panoptic segmentation task

Loading 2303.17376v1…