Source-linked AI summary
A Unified Sequence Interface for Vision Tasks
Ting Chen, Saurabh Saxena, Lala Li, Tsung-Yi Lin, David J. Fleet, Geoffrey Hinton
TL;DR
Core vision tasks have diverse outputs and consequently use distinct architectures and loss functions. This paper unifies object detection, instance segmentation, keypoint detection, and image captioning through a prompt-conditioned pixel-to-sequence interface with shared architecture and objective. The resulting model achieves competitive performance compared with established task-specific models, while autoregressive inference can be slower for longer sequences.
Problem
Core computer vision tasks have diverse outputs that are not readily expressed in natural language, leading to specialized architectures and loss functions for different tasks.
Method
The method tokenizes task descriptions and outputs into discrete sequences, then uses prompts with one shared neural network architecture and objective function across four vision tasks.
Results
The unified model simultaneously solves all four tasks and achieves competitive performance compared with well-established task-specific models.
Takeaways & Limitations
A shared pixel-to-sequence interface can support multiple core vision tasks without specialized architectures or loss functions.
Takeaways & Limitations
Autoregressive inference can be slower than specialized systems, particularly for longer sequences.
Abstract
from arXiv · showhide
While language tasks are naturally expressed in a single, unified, modeling framework, i.e., generating sequences of tokens, this has not been the case in computer vision. As a result, there is a proliferation of distinct architectures and loss functions for different vision tasks. In this work we show that a diverse set of "core" computer vision tasks can also be unified if formulated in terms of a shared pixel-to-sequence interface. We focus on four tasks, namely, object detection, instance segmentation, keypoint detection, and image captioning, all with diverse types of outputs, e.g., bounding boxes or dense masks. Despite that, by formulating the output of each task as a sequence of discrete tokens with a unified interface, we show that one can train a neural network with a single model architecture and loss function on all these tasks, with no task-specific customization. To solve a specific task, we use a short prompt as task description, and the sequence output adapts to the prompt so it can produce task-specific output. We show that such a model can achieve competitive performance compared to well-established task-specific models.
1 Introduction
Core vision tasks have heterogeneous outputs that have traditionally required specialized architectures and loss functions. The paper proposes unifying four such tasks through a shared pixel-to-sequence interface driven by task prompts.
- Motivation: Unlike language tasks, core vision tasks produce heterogeneous outputs such as boxes, masks, and keypoints that are not readily expressed in natural language.Object detection outputs bounding boxes and labels, instance segmentation outputs masks, and keypoint detection outputs image landmarks.
- Motivation: These diverse outputs have led existing methods to develop specialized architectures and sophisticated loss functions for individual vision tasks.
- Motivation: A unified interface could simplify architecture and loss design while increasing feature sharing across tasks.The motivation is framed as a step toward expressing seemingly disparate vision tasks within one framework.
- Approach: The proposed approach represents four tasks—object detection, instance segmentation, human keypoint detection, and image captioning—as pixel-to-sequence problems.It extends Pix2Seq from object detection to a broader set of tasks.
- Approach: A task prompt specifies the requested task, and one shared neural network architecture and objective function generate the corresponding task-specific sequence output.This replaces task-specific heads with prompt-conditioned output adaptation.
2 Approach
The approach represents diverse vision tasks as pixel-to-sequence problems using shared tokenization, architecture, and training objectives. Task prompts condition autoregressive decoding, while task-specific encoding and decoding translate between outputs and discrete sequences.
- 2.1 A unified interface with tokenization: The unified interface converts task descriptions and outputs into sequences of discrete tokens for object detection, instance segmentation, keypoint detection, and captioning.Bounding boxes, polygons, keypoints, and text are represented through task-specific tokenization schemes within a shared vocabulary.
- 2.1 A unified interface with tokenization: Object detection serializes each object as five tokens—[ymin, xmin, ymax, xmax, c]—after quantizing continuous image coordinates.Multiple objects are randomly ordered and concatenated into one sequence.
- 2.1 A unified interface with tokenization: Instance segmentation predicts quantized polygon coordinates rather than per-pixel masks, with random polygon starts and separator tokens for multiple polygons per instance.Each instance therefore has a single corresponding sequence.
- 2.1 A unified interface with tokenization: Keypoint sequences encode quantized y,x coordinates for body landmarks, using an occlusion token when keypoints are hidden; captioning directly predicts text tokens.The keypoint ordering is fixed for the 14 person keypoints considered in COCO.
- 2.2 Unified architecture and objective function: An encoder-decoder model maps image pixels to hidden representations and autoregressively generates tokens conditioned on preceding tokens and a task prompt.This shared design removes per-task heads or necks from the architecture.
- 2.2 Unified architecture and objective function: Training concatenates prompt and output tokens, but assigns zero weight to prompt-token targets so the decoder learns to predict only desired outputs.At inference, the prompt is fixed and the model generates the remaining sequence.
- 2.3 Training: Joint training can mix image-sequence pairs or task-specific batches, with task weights controlling their contributions; this work uses batch mixing.Batch mixing simplifies task-appropriate image augmentations, while data mixing may simplify future task additions.
- 2.4 Inference: During inference, nucleus sampling generates tokens from the prompt-conditioned likelihood, after which task-specific decoding converts sequences into required outputs.Bounding-box decoding groups predictions into five-token tuples and dequantizes coordinate tokens.
3 Experiments
The experiments evaluate the unified model on COCO across four tasks, comparing task-specific baselines, single-task variants, and one multi-task model. Results show competitive task performance, with inference and weighting choices affecting outcomes.
- Experimental setup: The study evaluates a 132M-parameter ViT-B encoder and Transformer autoregressive decoder on MS-COCO 2017, using 118k training and 5k validation images.The dataset spans object detection, instance segmentation, human keypoint detection, and image captioning.
- Implementation details: Eight segmentation samples improve performance by approximately 6 AP over one sample, while more than 12 samples provide no further boost.The method averages independently generated masks and thresholds the result at 50%.
- Implementation details: A crop twice the person bounding-box size improves keypoint detection by approximately 9 AP over an extremely large crop.The extremely large crop is approximately 20 times the box size and approximates using the entire image.
- Task weighting: The four-task model uses weights of 0.1782, 0.7128, 0.099, and 0.01 for detection, segmentation, captioning, and keypoint detection, respectively.Weights are selected greedily by adding tasks while preserving the relative weighting of existing tasks.
- Quantitative results: The unified model achieves competitive results against specialized baselines despite using no task-specific architectural priors or loss functions, while keeping one set of weights across tasks.The comparison includes Faster R-CNN and DETR for object detection.
- Quantitative results: Larger image sizes generally improve performance, except for keypoint detection, where an existing cropped region makes scaling less helpful and may cause overfitting with limited labels.The keypoint task already operates on a cropped region of interest.
- Qualitative results: The multi-task model detects objects in cluttered, occluded scenes and produces well-localized instance masks and keypoint predictions on selected COCO validation images.The paper also presents generated captions, noting that large-scale image-text pretraining could improve captioning performance.
4 Related work
Related work connects this approach to sequence-based decoding, shared-backbone multi-task learning, and generalist vision models. The paper extends unified sequential prediction to several core visual localization tasks.
- Decoding visual concepts: Vision tasks encode concepts in different formats, including words for captioning, labels and boxes for detection, and pixels or keypoints for localization.This diversity motivates interfaces that can represent multiple visual outputs.
- Decoding visual concepts: DETR, MaskFormer, and Pix2Seq progressively support shared decoding schemes, with Pix2Seq treating boxes and labels as discrete token sequences.These methods provide precedents for reducing task-specific decoding differences.
- Novelty: The paper extends shared-interface, architecture, and training-objective ideas beyond object detection to language and multiple visual localization tasks.This positions the work as a broader unification of previously distinct task formulations.
- Generalist vision models: Earlier generalist vision approaches commonly shared a backbone while retaining independent task-specific heads, whereas this work targets a more unified interface.Transformer-based systems also explored unified backbones and multimodal decoding architectures.
- Generalist vision models: The proposed direction expresses core vision tasks through a universal interface, with tokens grounded to visual concepts; related work includes Gato, UViM, and Unified-IO.These concurrent approaches unify vision, control, or vision tasks using sequential prediction or learned discrete codes.
5 Conclusion
The paper presents a unified sequence interface for diverse vision tasks and reports competitive performance against specialized models. It acknowledges remaining architectural, scaling, and inference-speed limitations.
- The model expresses diverse vision tasks through discrete token sequences while sharing architecture and loss functions across tasks.The approach covers object detection, instance segmentation, keypoint detection, and image captioning.
- Competitive performance against well-established task-specific models supports the feasibility of the unified approach.
- Further architectural and training improvements, including larger pretraining datasets or model sizes, may be needed to challenge specialized systems.
- Autoregressive inference can be slower than specialized systems, particularly for longer sequences.The paper identifies non-autoregressive sequence modeling as a possible efficiency improvement.
- The optimal implementation of a unified interface remains an open research question, and the proposed sequence interface is only one possible implementation.
Checklist
The checklist reports that the paper provides code, training details, compute information, and a limitations discussion, but does not report repeated-run error bars or asset licenses.
- The authors state that code will be open-sourced at the Pix2Seq GitHub repository.
- Training details, including data splits and hyperparameters, are reported.
- Error bars from repeated experiments are not reported because some experiments are expensive to repeat and standard errors are usually small.
- The experiments use 32–128 Cloud TPUs and generally require 4–12 hours, depending on architecture and task.
- The paper cites existing assets and describes the work’s limitations, but does not state asset licenses or discuss potential negative societal impacts.