PaperlayerView exact revision on arXiv

Source-linked AI summary

Pix2seq: A Language Modeling Framework for Object Detection

Ting Chen, Saurabh Saxena, Lala Li, David J. Fleet, Geoffrey Hinton

arXiv:2109.10852v2cs.CVcs.AIcs.CLcs.LG

TL;DR

Object detection is typically built from specialized architectures and losses, while Pix2Seq asks whether object descriptions can instead be generated as language-like token sequences from pixels. It discretizes boxes and labels, trains an encoder-decoder with autoregressive likelihood, and reports competitive COCO performance while retaining a generic framework. The main scope boundary is that autoregressive inference is expensive for long sequences and remains insufficiently fast for real-time detection.

  • Problem

    Existing detection systems are specialized and complex, limiting their ease of integration and generalization to broader tasks.

  • Method

    Pix2Seq casts object detection as language modeling by converting boxes and labels into discrete tokens generated by an encoder-decoder conditioned on pixels.

  • Results

    50 AP is achieved by the best Objects365-pretrained Pix2Seq model on COCO, 5% higher than the best model trained from scratch.

  • Takeaways & Limitations

    Pix2Seq provides a simple, generic detection framework that may extend to vision tasks whose outputs are concise sequences of discrete tokens.

  • Takeaways & Limitations

    Autoregressive modeling is expensive for long sequences, and further work is needed to make Pix2Seq fast enough for real-time object detection.

Abstract

from arXiv · show

We present Pix2Seq, a simple and generic framework for object detection. Unlike existing approaches that explicitly integrate prior knowledge about the task, we cast object detection as a language modeling task conditioned on the observed pixel inputs. Object descriptions (e.g., bounding boxes and class labels) are expressed as sequences of discrete tokens, and we train a neural network to perceive the image and generate the desired sequence. Our approach is based mainly on the intuition that if a neural network knows about where and what the objects are, we just need to teach it how to read them out. Beyond the use of task-specific data augmentations, our approach makes minimal assumptions about the task, yet it achieves competitive results on the challenging COCO dataset, compared to highly specialized and well optimized detection algorithms.

1 INTRODUCTION

Pix2Seq reframes object detection as generic language modeling over pixel inputs, addressing the complexity and task-specific specialization of existing detectors. It represents object descriptions as discrete token sequences generated by an encoder-decoder model.

  • 1 INTRODUCTION: Existing object detectors are carefully designed and highly customized, making them difficult to integrate into larger systems or generalize across tasks.Their specialization includes task-specific architectural and loss-function choices.
  • 1 INTRODUCTION: Pix2Seq casts object detection as language modeling conditioned on pixel inputs.The model generates sequences corresponding to object descriptions such as bounding boxes and class labels.
  • 1 INTRODUCTION: Object descriptions are converted into discrete token sequences that a neural network learns to generate from images.The framework is reminiscent of image captioning, but its output describes detected objects.
  • 1 INTRODUCTION: The framework uses generic architecture and loss choices rather than engineering them specifically for object detection.This design is intended to make the framework extensible to different domains and applications.

2 THE PIX2SEQ FRAMEWORK

Pix2Seq builds object detection as a sequence-generation pipeline: annotations become discrete tokens, an encoder-decoder predicts them autoregressively, and sequence augmentation adds task-specific robustness. Quantization, ordering, termination, and inference choices define how object descriptions are represented and generated.

  • 2 THE PIX2SEQ FRAMEWORK: Pix2Seq comprises image augmentation, sequence construction and augmentation, an encoder-decoder architecture, and a token likelihood objective.The model maximizes the log likelihood of tokens conditioned on the image and preceding tokens.
  • 2.1 SEQUENCE CONSTRUCTION FROM OBJECT DESCRIPTIONS: Each object becomes five tokens—[ymin, xmin, ymax, xmax, c]—with coordinates uniformly quantized into [1, nbins].A shared vocabulary contains coordinate bins and class labels; 600 bins suffice for zero quantization error on a 600×600 image.
  • 2.1 SEQUENCE CONSTRUCTION FROM OBJECT DESCRIPTIONS: Multiple object descriptions are serialized in random order, and an EOS token marks the end of variable-length sequences.Random ordering avoids imposing a semantic order on objects whose order does not matter for detection.
  • 2.2 ARCHITECTURE, OBJECTIVE AND INFERENCE: The encoder processes pixels while the Transformer decoder generates one target token at a time using a maximum-likelihood objective.The objective can assign token-specific weights, although the standard setting uses w_j = 1 for every token.
  • 2.2 ARCHITECTURE, OBJECTIVE AND INFERENCE: At inference, tokens are sampled until EOS, then converted back into object descriptions; nucleus sampling yields higher recall than arg max sampling.The autoregressive decoder can terminate before predicting all objects, creating a precision–recall trade-off when EOS likelihood is adjusted.
  • 2.3 SEQUENCE AUGMENTATION TO INTEGRATE TASK PRIORS: Sequence augmentation trains on real and synthetic noise objects, labels noise as “noise,” and assigns zero loss weight to their “n/a” coordinates.Synthetic objects come from perturbing ground-truth boxes or generating random boxes, improving robustness to noisy and duplicated predictions.
  • 2.3 SEQUENCE AUGMENTATION TO INTEGRATE TASK PRIORS: With sequence augmentation, Pix2Seq can delay EOS to improve recall without increasing noisy and duplicated predictions.The model instead predicts to a maximum length and replaces generated “noise” labels with the highest-likelihood real class.

3 EXPERIMENTS

Pix2Seq is evaluated on COCO against established detectors, with competitive performance from scratch and stronger results after Objects365 pretraining. Ablations show that sequence design and augmentation materially affect detection behavior, while decoder attention concentrates on objects during generation.

  • Main comparisons: Pix2Seq achieves competitive COCO results against Faster R-CNN and DETR baselines.The comparison uses average precision across multiple thresholds and object sizes on the COCO validation set.
  • Main comparisons: 4-5 AP better on small objects than DETR, while performance is comparable or slightly worse on medium and large objects.
  • Main comparisons: 50 AP is achieved by the best Objects365-pretrained model, 5% higher than the best model trained from scratch.The pretraining uses 640×640 images, while fine-tuning can use larger image sizes.
  • Ablation on sequence construction: 500 quantization bins or more are sufficient for performance on images whose longest side is 640 pixels.At 500 bins, each bin represents approximately 1.3 pixels, introducing no significant coordinate approximation error.
  • Ablation on sequence construction: Random object ordering yields the best average precision and average recall among the evaluated ordering strategies.The authors conjecture that random ordering lets the model recover objects missed earlier in the sequence.
  • Ablation on sequence augmentation: Sequence augmentation mainly improves recall, with its strongest effect during fine-tuning and a smaller influence on average precision.Without sequence augmentation, recall is significantly worse for the optimal AP in both training strategies.
  • Visualization of decoder’s cross attention map: Decoder cross attention is diverse for the first coordinate token, then quickly concentrates on the predicted object.The visualization averages attention over layers and heads while generating the first few object-description tokens.

4 RELATED WORK

Related work spans specialized object-detection architectures and losses, recurrent one-object-at-a-time predictors, and language-modeling methods for sequential data. Pix2Seq is positioned at their intersection by applying language modeling to object detection.

  • Object detection: Existing detectors encode task knowledge through proposals, anchors, window centers, and specialized suppression mechanisms.
  • Object detection: Detection losses are tailored to bounding-box regression, intersection over union, or set-based matching, whereas Pix2Seq uses maximum likelihood.
  • Object detection: Earlier recurrent detection systems predict one object at a time but are not Transformer-based and lack evaluation against modern baselines on larger datasets.
  • Language modeling: Pix2Seq draws on language modeling methods that have also modeled sequential data such as translation and image captioning.

5 CONCLUSION AND FUTURE WORK

Pix2Seq casts object detection as language modeling over discrete object-description tokens, simplifying the detection pipeline and suggesting broader vision-task applicability. Its main practical limitation is expensive autoregressive inference for long sequences.

  • 5 CONCLUSION AND FUTURE WORK: Pix2Seq casts object detection as language modeling over sequences of discrete tokens.The framework is intended as a simple, generic approach to detection and potentially other vision tasks with concise discrete-token outputs.
  • 5 CONCLUSION AND FUTURE WORK: The framework largely simplifies detection by removing most specialization in modern detection algorithms.
  • 5 CONCLUSION AND FUTURE WORK: Pix2Seq may extend to keypoint detection, image captioning, and visual question answering.
  • 5 CONCLUSION AND FUTURE WORK: Autoregressive modeling is expensive for long sequences, mainly during inference.For COCO, images average about 7 objects and approximately 35 tokens, but faster real-time detection remains future work.

B TRAINING DETAILS

Training uses transformer-based encoder-decoder architectures, extensive image and sequence augmentation, and distinct settings for COCO training, Objects365 pretraining, and COCO fine-tuning.

  • B TRAINING DETAILS: COCO baseline architectures use a ResNet backbone followed by six encoder and six causal decoder transformer layers.The transformer dimension is 256, with 8 attention heads and a feed-forward dimension of 1024.
  • B TRAINING DETAILS: Training uses scale jittering, random crops, color distortion, 2000 quantization bins, and randomized object order.
  • B TRAINING DETAILS: The network is trained from scratch for 300 epochs with batch size 128 and two independent augmentations per image.This produces an effective batch size of 256 and is reported to help reduce overfitting.
  • B TRAINING DETAILS: Objects365 pretraining uses broader architecture variants, weaker augmentation, 640×640 images, and 400K training steps.
  • B TRAINING DETAILS: COCO fine-tuning uses batch size 128 for ResNet backbones and 64 for ViT backbones, typically for 60 epochs.

C ABLATION ON INFERENCE (arg max VS NUCLEUS SAMPLING)

The inference ablation varies nucleus-sampling probability p. Sampling with p > 0 improves recall and also average precision, with p = 0.4 selected as the default.

  • C ABLATION ON INFERENCE (arg max VS NUCLEUS SAMPLING): Nucleus sampling with p > 0 improves object recall and average precision over argmax sampling.At p = 0, nucleus sampling is equivalent to argmax sampling.
  • C ABLATION ON INFERENCE (arg max VS NUCLEUS SAMPLING): The average precision curve is relatively flat between p = 0.2 and p = 0.5.
  • C ABLATION ON INFERENCE (arg max VS NUCLEUS SAMPLING): The authors select p = 0.4 as the default value for other experiments.
  • C ABLATION ON INFERENCE (arg max VS NUCLEUS SAMPLING): Nucleus sampling draws from a truncated ranked token list whose cumulative probability is at least p.

D VISUALIZATION OF SIMILARITY AMONG COORDINATE TOKENS

The coordinate-token embedding space preserves spatial locality: nearby coordinates have higher cosine similarity than distant coordinates. The paper attributes this emergent structure to uncertainty in bounding-box annotations.

  • D VISUALIZATION OF SIMILARITY AMONG COORDINATE TOKENS: The analysis computes cosine similarity among embeddings for discrete coordinate tokens.Coordinate tokens and class-name tokens share the same embedding matrix, from which the coordinate-token subset is selected.
  • D VISUALIZATION OF SIMILARITY AMONG COORDINATE TOKENS: Nearby coordinates have higher embedding similarity than far-away coordinates.
  • D VISUALIZATION OF SIMILARITY AMONG COORDINATE TOKENS: The authors suggest annotation noise and uncertainty produce this locality in coordinate embeddings.Bounding-box annotations are described as samples from a distribution over potential boxes encoding coordinate locality.

E THE ABILITY TO DIRECT THE ATTENTION WITH GIVEN COORDINATES

Pix2Seq can use coordinate tokens to direct decoder attention toward specified image regions, including across different grid scales.

  • E THE ABILITY TO DIRECT THE ATTENTION WITH GIVEN COORDINATES: The experiment divides an image into an N × N grid of rectangular regions specified by bounding-box coordinates.Each region is represented by [ymin, xmin, ymax, xmax].
  • E THE ABILITY TO DIRECT THE ATTENTION WITH GIVEN COORDINATES: Nearby coordinate tokens have higher cosine similarity in their embeddings.The similarity visualization includes the full coordinate-token matrix and selected rows or submatrices.
  • E THE ABILITY TO DIRECT THE ATTENTION WITH GIVEN COORDINATES: After reading coordinate sequences, the decoder’s cross attention is visualized on the corresponding feature-map regions.Pixels are shuffled to remove distraction from existing objects, and the top 2% of attentions are removed for clarity.
  • E THE ABILITY TO DIRECT THE ATTENTION WITH GIVEN COORDINATES: The model appears able to attend to the specified region at different scales.The visualizations show attention focused on the pointed region across grids of different sizes.

F MORE VISUALIZATION ON DECODER’S CROSS ATTENTION

Decoder cross attention is examined during class-token prediction to determine whether the model focuses on the detected object.

  • F MORE VISUALIZATION ON DECODER’S CROSS ATTENTION: The decoder pays the most attention to the object when predicting its class token.This behavior is shown for several images using attention overlaid on the original images.
  • F MORE VISUALIZATION ON DECODER’S CROSS ATTENTION: Figure 13 visualizes Transformer decoder cross attention when predicting class tokens from given bounding boxes.The attention is over visual features conditioned on the supplied boxes.

G VISUALIZATION OF DETECTION RESULTS

Detection predictions from Pix2Seq are visualized on crowded COCO validation images, illustrating model outputs at a specified score threshold.

  • G VISUALIZATION OF DETECTION RESULTS: Figure 14 shows Pix2Seq detection results on a subset of crowded images from the COCO validation set.The images contain a crowded set of objects.
  • G VISUALIZATION OF DETECTION RESULTS: The visualized predictions come from one Pix2Seq model with 46 AP.The examples use a score threshold of 0.5.
Loading 2109.10852v2…