PaperlayerView exact revision on arXiv

Source-linked AI summary

Pix2seq: A Language Modeling Framework for Object Detection

Ting Chen, Saurabh Saxena, Lala Li, David J. Fleet, Geoffrey Hinton

arXiv:2109.10852v2cs.CVcs.AIcs.CLcs.LG

TL;DR

Object detection is typically built from specialized architectures and losses, while Pix2Seq asks whether object descriptions can instead be generated as language-like token sequences from pixels. It discretizes boxes and labels, trains an encoder-decoder with autoregressive likelihood, and reports competitive COCO performance while retaining a generic framework. The main scope boundary is that autoregressive inference is expensive for long sequences and remains insufficiently fast for real-time detection.

  • Problem

    Existing detection systems are specialized and complex, limiting their ease of integration and generalization to broader tasks.

    Source: §1 · p. 2

  • Method

    Pix2Seq casts object detection as language modeling by converting boxes and labels into discrete tokens generated by an encoder-decoder conditioned on pixels.

    Source: §1 · p. 2, §2.1 · p. 3

  • Results

    50 AP is achieved by the best Objects365-pretrained Pix2Seq model on COCO, 5% higher than the best model trained from scratch.

    Source: §3.2 · p. 7

  • Takeaways & Limitations

    Pix2Seq provides a simple, generic detection framework that may extend to vision tasks whose outputs are concise sequences of discrete tokens.

    Source: §5 · p. 9

    Autoregressive modeling is expensive for long sequences, and further work is needed to make Pix2Seq fast enough for real-time object detection.

    Source: §5 · p. 9

Abstract

from arXiv · show

We present Pix2Seq, a simple and generic framework for object detection. Unlike existing approaches that explicitly integrate prior knowledge about the task, we cast object detection as a language modeling task conditioned on the observed pixel inputs. Object descriptions (e.g., bounding boxes and class labels) are expressed as sequences of discrete tokens, and we train a neural network to perceive the image and generate the desired sequence. Our approach is based mainly on the intuition that if a neural network knows about where and what the objects are, we just need to teach it how to read them out. Beyond the use of task-specific data augmentations, our approach makes minimal assumptions about the task, yet it achieves competitive results on the challenging COCO dataset, compared to highly specialized and well optimized detection algorithms.

1 INTRODUCTION

Pix2Seq reframes object detection as generic language modeling over pixel inputs, addressing the complexity and task-specific specialization of existing detectors. It represents object descriptions as discrete token sequences generated by an encoder-decoder model.

1 INTRODUCTION

  • Existing object detectors are carefully designed and highly customized, making them difficult to integrate into larger systems or generalize across tasks.Their specialization includes task-specific architectural and loss-function choices.Source: §1 · pp. 1–2
  • Pix2Seq casts object detection as language modeling conditioned on pixel inputs.The model generates sequences corresponding to object descriptions such as bounding boxes and class labels.Source: §1 · p. 2
  • Object descriptions are converted into discrete token sequences that a neural network learns to generate from images.The framework is reminiscent of image captioning, but its output describes detected objects.Source: §1 · p. 2
  • The framework uses generic architecture and loss choices rather than engineering them specifically for object detection.This design is intended to make the framework extensible to different domains and applications.Source: §1 · p. 2

2 THE PIX2SEQ FRAMEWORK

Pix2Seq builds object detection as a sequence-generation pipeline: annotations become discrete tokens, an encoder-decoder predicts them autoregressively, and sequence augmentation adds task-specific robustness. Quantization, ordering, termination, and inference choices define how object descriptions are represented and generated.

2 THE PIX2SEQ FRAMEWORK

  • Pix2Seq comprises image augmentation, sequence construction and augmentation, an encoder-decoder architecture, and a token likelihood objective.The model maximizes the log likelihood of tokens conditioned on the image and preceding tokens.Source: §2 · p. 2

2.1 SEQUENCE CONSTRUCTION FROM OBJECT DESCRIPTIONS

  • Each object becomes five tokens—[ymin, xmin, ymax, xmax, c]—with coordinates uniformly quantized into [1, nbins].A shared vocabulary contains coordinate bins and class labels; 600 bins suffice for zero quantization error on a 600×600 image.Source: §2.1 · p. 3
  • Multiple object descriptions are serialized in random order, and an EOS token marks the end of variable-length sequences.Random ordering avoids imposing a semantic order on objects whose order does not matter for detection.Source: §2.1 · p. 3

2.2 ARCHITECTURE, OBJECTIVE AND INFERENCE

  • The encoder processes pixels while the Transformer decoder generates one target token at a time using a maximum-likelihood objective.The objective can assign token-specific weights, although the standard setting uses w_j = 1 for every token.Source: §2.2 · p. 4
  • At inference, tokens are sampled until EOS, then converted back into object descriptions; nucleus sampling yields higher recall than arg max sampling.The autoregressive decoder can terminate before predicting all objects, creating a precision–recall trade-off when EOS likelihood is adjusted.Source: §2.2 · p. 4, §2.3 · p. 4

2.3 SEQUENCE AUGMENTATION TO INTEGRATE TASK PRIORS

  • Sequence augmentation trains on real and synthetic noise objects, labels noise as “noise,” and assigns zero loss weight to their “n/a” coordinates.Synthetic objects come from perturbing ground-truth boxes or generating random boxes, improving robustness to noisy and duplicated predictions.Source: §2.3 · pp. 4–5, Figure 5 · p. 5
  • With sequence augmentation, Pix2Seq can delay EOS to improve recall without increasing noisy and duplicated predictions.The model instead predicts to a maximum length and replaces generated “noise” labels with the highest-likelihood real class.Source: §2.3 · pp. 4–5

3 EXPERIMENTS

Pix2Seq is evaluated on COCO against established detectors, with competitive performance from scratch and stronger results after Objects365 pretraining. Ablations show that sequence design and augmentation materially affect detection behavior, while decoder attention concentrates on objects during generation.

Main comparisons

  • Pix2Seq achieves competitive COCO results against Faster R-CNN and DETR baselines.The comparison uses average precision across multiple thresholds and object sizes on the COCO validation set.Source: §3.2 · p. 6, Table 1 · p. 6
  • 4-5 AP better on small objects than DETR, while performance is comparable or slightly worse on medium and large objects.Source: §3.2 · p. 7
  • 50 AP is achieved by the best Objects365-pretrained model, 5% higher than the best model trained from scratch.The pretraining uses 640×640 images, while fine-tuning can use larger image sizes.Source: §3.2 · p. 7, Table 2 · p. 7

Ablation on sequence construction

  • 500 quantization bins or more are sufficient for performance on images whose longest side is 640 pixels.At 500 bins, each bin represents approximately 1.3 pixels, introducing no significant coordinate approximation error.Source: §3.3 · p. 7
  • Random object ordering yields the best average precision and average recall among the evaluated ordering strategies.The authors conjecture that random ordering lets the model recover objects missed earlier in the sequence.Source: §3.3 · p. 7, Figure 7 · p. 7

Ablation on sequence augmentation

  • Sequence augmentation mainly improves recall, with its strongest effect during fine-tuning and a smaller influence on average precision.Without sequence augmentation, recall is significantly worse for the optimal AP in both training strategies.Source: §3.4 · p. 8, Table 3 · p. 8, Figure 8 · p. 8

Visualization of decoder’s cross attention map

  • Decoder cross attention is diverse for the first coordinate token, then quickly concentrates on the predicted object.The visualization averages attention over layers and heads while generating the first few object-description tokens.Source: §3.5 · p. 8, Figure 9 · p. 8

4 RELATED WORK

Related work spans specialized object-detection architectures and losses, recurrent one-object-at-a-time predictors, and language-modeling methods for sequential data. Pix2Seq is positioned at their intersection by applying language modeling to object detection.

Object detection

  • Existing detectors encode task knowledge through proposals, anchors, window centers, and specialized suppression mechanisms.Source: §4 · p. 9
  • Detection losses are tailored to bounding-box regression, intersection over union, or set-based matching, whereas Pix2Seq uses maximum likelihood.Source: §4 · p. 9
  • Earlier recurrent detection systems predict one object at a time but are not Transformer-based and lack evaluation against modern baselines on larger datasets.Source: §4 · p. 9

Language modeling

  • Pix2Seq draws on language modeling methods that have also modeled sequential data such as translation and image captioning.Source: §4 · p. 9

5 CONCLUSION AND FUTURE WORK

Pix2Seq casts object detection as language modeling over discrete object-description tokens, simplifying the detection pipeline and suggesting broader vision-task applicability. Its main practical limitation is expensive autoregressive inference for long sequences.

5 CONCLUSION AND FUTURE WORK

  • Pix2Seq casts object detection as language modeling over sequences of discrete tokens.The framework is intended as a simple, generic approach to detection and potentially other vision tasks with concise discrete-token outputs.Source: §5 · p. 9
  • The framework largely simplifies detection by removing most specialization in modern detection algorithms.Source: §5 · p. 9
  • Pix2Seq may extend to keypoint detection, image captioning, and visual question answering.Source: §5 · p. 9
  • Autoregressive modeling is expensive for long sequences, mainly during inference.For COCO, images average about 7 objects and approximately 35 tokens, but faster real-time detection remains future work.Source: §5 · p. 9

B TRAINING DETAILS

Training uses transformer-based encoder-decoder architectures, extensive image and sequence augmentation, and distinct settings for COCO training, Objects365 pretraining, and COCO fine-tuning.

B TRAINING DETAILS

  • COCO baseline architectures use a ResNet backbone followed by six encoder and six causal decoder transformer layers.The transformer dimension is 256, with 8 attention heads and a feed-forward dimension of 1024.Source: B TRAINING DETAILS · p. 14
  • Training uses scale jittering, random crops, color distortion, 2000 quantization bins, and randomized object order.Source: B TRAINING DETAILS · p. 14
  • The network is trained from scratch for 300 epochs with batch size 128 and two independent augmentations per image.This produces an effective batch size of 256 and is reported to help reduce overfitting.Source: B TRAINING DETAILS · p. 14
  • Objects365 pretraining uses broader architecture variants, weaker augmentation, 640×640 images, and 400K training steps.Source: B TRAINING DETAILS · p. 14
  • COCO fine-tuning uses batch size 128 for ResNet backbones and 64 for ViT backbones, typically for 60 epochs.Source: B TRAINING DETAILS · p. 14

C ABLATION ON INFERENCE (arg max VS NUCLEUS SAMPLING)

The inference ablation varies nucleus-sampling probability p. Sampling with p > 0 improves recall and also average precision, with p = 0.4 selected as the default.

C ABLATION ON INFERENCE (arg max VS NUCLEUS SAMPLING)

D VISUALIZATION OF SIMILARITY AMONG COORDINATE TOKENS

The coordinate-token embedding space preserves spatial locality: nearby coordinates have higher cosine similarity than distant coordinates. The paper attributes this emergent structure to uncertainty in bounding-box annotations.

D VISUALIZATION OF SIMILARITY AMONG COORDINATE TOKENS

E THE ABILITY TO DIRECT THE ATTENTION WITH GIVEN COORDINATES

Pix2Seq can use coordinate tokens to direct decoder attention toward specified image regions, including across different grid scales.

E THE ABILITY TO DIRECT THE ATTENTION WITH GIVEN COORDINATES

F MORE VISUALIZATION ON DECODER’S CROSS ATTENTION

Decoder cross attention is examined during class-token prediction to determine whether the model focuses on the detected object.

F MORE VISUALIZATION ON DECODER’S CROSS ATTENTION

G VISUALIZATION OF DETECTION RESULTS

Detection predictions from Pix2Seq are visualized on crowded COCO validation images, illustrating model outputs at a specified score threshold.

G VISUALIZATION OF DETECTION RESULTS

Related papers

Loading 2109.10852v2…