PaperlayerView exact revision on arXiv

Source-linked AI summary

Toward Transformer-Based Object Detection

Josh Beal, Eric Kim, Eric Tzeng, Dong Huk Park, Andrew Zhai, Dmitry Kislyuk

arXiv:2012.09958v1cs.CVcs.AIcs.LG

TL;DR

The paper asks whether Vision Transformers can support spatially sensitive object detection despite attention’s resolution challenges. It introduces ViT-FRCNN, which uses a ViT backbone with detection components, and reports competitive COCO results plus improved large-object detection and reduced sensitivity to NMS.

  • Problem

    Vision Transformers’ suitability for object detection is uncertain because detection requires high-resolution spatial detail while attention is computationally costly.

  • Method

    ViT-FRCNN repurposes Vision Transformer patch outputs as spatial features for a detection architecture with region proposal and box-regression components.

  • Results

    ViT-FRCNN achieves competitive COCO results, with +6.9 AP from reducing patch size from 32×32 to 16×16 and a smaller NMS-related drop than ResNet baselines.

  • Takeaways & Limitations

    The findings suggest transformer backbones can support large-scale pretraining and rapid fine-tuning for complex vision tasks.

  • Takeaways & Limitations

    The proposed explanation for fewer spurious overdetections is speculative, and the underlying mechanism requires further study.

Abstract

from arXiv · show

Transformers have become the dominant model in natural language processing, owing to their ability to pretrain on massive amounts of data, then transfer to smaller, more specific tasks via fine-tuning. The Vision Transformer was the first major attempt to apply a pure transformer model directly to images as input, demonstrating that as compared to convolutional networks, transformer-based architectures can achieve competitive results on benchmark classification tasks. However, the computational complexity of the attention operator means that we are limited to low-resolution inputs. For more complex tasks such as detection or segmentation, maintaining a high input resolution is crucial to ensure that models can properly identify and reflect fine details in their output. This naturally raises the question of whether or not transformer-based architectures such as the Vision Transformer are capable of performing tasks other than classification. In this paper, we determine that Vision Transformers can be used as a backbone by a common detection task head to produce competitive COCO results. The model that we propose, ViT-FRCNN, demonstrates several known properties associated with transformers, including large pretraining capacity and fast fine-tuning performance. We also investigate improvements over a standard detection backbone, including superior performance on out-of-domain images, better performance on large objects, and a lessened reliance on non-maximum suppression. We view ViT-FRCNN as an important stepping stone toward a pure-transformer solution of complex vision tasks such as object detection.

1. Introduction

The paper asks whether Vision Transformers can retain enough spatial information for locally sensitive tasks such as object detection. It proposes ViT-FRCNN and reports competitive COCO performance alongside transformer-associated benefits.

  • 1. Introduction: Vision Transformers process images as patch sequences and retain capacity for large-scale pretraining, despite trailing convolutional models on mid-sized classification datasets.This suggests attention-based components may supplement or replace standard convolution in vision modeling.
  • 1. Introduction: Global attention may weaken spatial correspondence between input regions and intermediate features, complicating the extension from classification to detection.Convolutional feature maps naturally preserve spatial layout, whereas transformer features attend globally at every layer.
  • 1. Introduction: ViT-FRCNN augments a Vision Transformer with detection-specific heads and retains sufficient spatial information for object detection.The model detects and localizes objects using a transformer-based backbone.
  • 1. Introduction: ViT-FRCNN achieves competitive COCO detection results while exhibiting desirable transformer properties, including benefits from massive pretraining.The experiments also report improved detection of large objects and fewer spurious overdetections.
  • 1. Introduction: The authors argue that large-scale pretraining followed by rapid task-specific fine-tuning can scale further in computer vision.They attribute this prospect to transformer model capacity and flexible features learned in such backbones.

2. Related work

Related work establishes transformers in language and image recognition, then contrasts detectors that use transformers for decoding with this paper’s use of a transformer to encode visual features.

  • 2. Related work: DETR introduced transformer-based object detection by adding transformer encoder-decoder modules above CNN features and using set matching without NMS.Its decoder learns to self-suppress duplicate bounding-box predictions.
  • 2. Related work: Deformable DETR addresses DETR’s long convergence and weak small-object performance by attending to a small set of learned sampling locations.The approach targets the complexity of processing high-resolution feature maps.
  • 2. Related work: This paper differs from DETR and Deformable DETR by using a Transformer to encode visual features and a traditional region proposal network to produce detections.Both prior detectors rely on CNNs for visual feature encoding.
  • 2. Related work: ViT showed that a pure Transformer can match state-of-the-art convolutional models on image recognition after large-scale pretraining and transfer.Its generalization to spatial tasks such as detection or segmentation had not yet been established.
  • 2. Related work: Other work explored fully Transformer-based visual generation, attention-convolution relationships, CNN limitations, and large-scale image pretraining for transfer.Prior findings on pretraining benefits for detection were mixed across dataset scales.

3. Method

ViT-FRCNN repurposes spatially arranged ViT patch outputs as a detection feature map and processes proposals with Faster R-CNN-style heads. The method adapts resolution and positional embeddings for detection inputs.

  • 3. Method: ViT-FRCNN augments a Vision Transformer backbone with a detection network that produces bounding-box classifications and coordinates.The goal is to transfer classification representations to object detection.
  • 3. Method: The model reinterprets unused patch-token outputs from ViT’s final layer as local features for detection.ViT normally feeds only the class-token state to its classification head.
  • 3. Method: The spatial feature map feeds an RPN, whose top proposals are RoI-pooled and passed to heads that classify regions and regress precise box coordinates.This detection network is modeled after Faster R-CNN.
  • 3. Method: RPN predictions classify object versus no object and regress offsets from anchor boxes to box coordinates.The box loss is a Huber loss between predicted and ground-truth offsets, with x, y, w, and h denoting box center, width, and height.
  • 3. Method: The RPN and detection heads are trained jointly end-to-end after top proposals are RoI-pooled into region features.The heads produce object-category or background predictions and bounding-box regressions.
  • 3. Method: Higher-resolution, aspect-ratio-preserving inputs support detection detail, while runtime bilinear interpolation adapts ViT position embeddings to input size and shape.These changes address ViT’s fixed-size square-image configuration.
  • 3. Method: The implementation follows Faster R-CNN conventions, including 15 anchors per feature-map location and NMS at overlap threshold 0.7.Training uses the top 2,000 proposals, while inference uses the top 1,000.
  • 3. Method: The authors compare using final versus all intermediate encoder outputs, reduce spatial features to 512 channels, and add intermediate residual blocks.The residual structure is visualized in Figure 2.

4. Experiments

Experiments show that ViT-FRCNN achieves competitive COCO detection performance, with results improving through larger pretraining datasets and higher encoder spatial resolution. Ablations also identify reduced overdetections and a useful role for residual blocks and overlapping patches.

  • COCO detection: ViT-FRCNN variants achieve respectable COCO val2017 detection performance, with 16×16 models substantially outperforming 32×32 models.The result indicates that preserving finer spatial detail is important for detection.
  • COCO detection: +1.2 AP improves ViT-B/16 from 36.6 to 37.8 AP when pretraining expands from ImageNet-21k to Annotations-1.3B.For ViT-B/32, the same dataset change yields a +1.6 AP improvement, from 29.3 to 30.9 AP.
  • Encoder spatial resolution: +6.9 AP results from decreasing the Annotations-1.3B ViT patch size from 32×32 to 16×16, reaching 37.8 AP overall.Small-object AP increases from 9.7 to 17.8 AP; overlapping patches also improve ViT-B/32 from 29.3 to 34.5 overall AP and from 9.0 to 15.6 small-object AP.
  • Ablations: Four connecting residual blocks increase performance from 24.1 AP to 28.8 AP, with diminishing returns beyond four blocks.The authors interpret this as evidence that encoder outputs benefit from learned post-processing before detection.
  • Overdetections: -2.8 AP is the ViT-B/16* drop as the NMS threshold rises from 0.5 to 0.9, versus -9.7 AP and -8.1 AP for ResNet50 and ResNet101.At an NMS threshold of 0.95, ViT-B/16*-FRCNN outperforms ResNet101-FRCNN-FPN by 8.1 AP, consistent with fewer overdetections.
  • Out-of-distribution generalization: +7 AP gives ViT-B/16*-FRCNN 22.9 AP over the ResNet101-FRCNN-FPN baseline on the out-of-distribution dataset.Increasing pretraining dataset size adds a further +2.7 AP boost, while Open Images V6 pretraining improves ViT-B/32 by +1.1 AP.

5. Conclusion

ViT-FRCNN is a competitive transformer-backbone object detector, supporting the use of transformer architectures for complex vision tasks. It retains large-scale pretraining and rapid fine-tuning properties observed in transformer models.

  • ViT-FRCNN is a competitive object detection solution using a transformer backbone.
  • The model suggests architectures distinct from established CNN backbones can advance complex vision tasks.
  • ViT-FRCNN exhibits the ability to pretrain on massive datasets without saturation and fine-tune quickly to new tasks.
  • The authors view ViT-FRCNN as an early example among transformer-based architectures addressing a broad range of vision problems.

A.1. ResNet baselines

The paper compares ViT-FRCNN with ResNet-based baselines and acknowledges that the baselines could perform substantially better with additional training and augmentation techniques.

  • The ResNet-FRCNN-FPN baselines are compared with results from the literature in Table 8.
  • Additional data augmentation and longer training schedules could substantially improve the ResNet-based baselines.
  • The authors avoid comparing ViT-FRCNN with ResNet results obtained using substantially more performance-enhancing tweaks.

A.2. ImageNet performance

The paper reports that models pretrained on Annotations-1.3B achieve competitive performance after fine-tuning on ILSVRC-2012, with detailed ViT-B/32 and ViT-B/16 results provided in Table 9.

  • Models pretrained on Annotations-1.3B achieve competitive performance when fine-tuned on ILSVRC-2012.
  • Table 9 reports ViT-B/32 and ViT-B/16 performance across different pretraining datasets.
  • The reported evaluation concerns transfer from pretraining datasets to ILSVRC-2012 fine-tuning.

A.3. Curriculum pretraining

The simplified detection architecture attaches box- and class-prediction MLP heads to Vision Transformer encoder outputs and uses a DETR-style set prediction loss.

  • The simplified detector uses a Vision Transformer backbone with box prediction and class prediction MLP heads attached to each encoder output.
  • The detection heads and set prediction loss follow the DETR implementation.
  • A box prediction offset at each output uses the corresponding patch center to improve early training convergence.

A.4. Object distributions

The analysis examines visual-object category frequencies across detection datasets and finds an approximate Zipfian pattern as dataset scale increases.

  • Datasets: The analysis compares COCO 2017, Open Images V6, and Object Index, which contains 248 million images, 646 million bounding boxes, and 220 categories.Object Index is described as a semi-supervised internal dataset.
  • Distribution: Visual-object class frequencies approximately follow Zipf’s law as detection datasets increase in scale.The paper connects this pattern to the longer-tailed category distributions expected in larger pretraining datasets.
  • Implications: The analysis identifies natural-language methods for handling class imbalance as a potential direction for increasingly long-tailed visual datasets.The passage frames this as worthy of further investigation rather than an established solution for object detection.

A.5. ObjectNet-D categories

The ObjectNet-D evaluation uses 23 categories corresponding to COCO 2017, while the broader ObjectNet overlap contains 29 categories shared with ILSVRC-2012 and COCO 2017.

  • ObjectNet overlap: ObjectNet contains 29 categories overlapping with both ILSVRC-2012 and COCO 2017.Examples include alarm clock, backpack, banana, bicycle, laptop, toaster, TV, umbrella, and wine bottle.
  • OBJECTNET-D: OBJECTNET-D restricts the evaluation to 23 corresponding COCO 2017 categories.Its listed categories include backpack, banana, bench, bicycle, bottle, bowl, chair, laptop, microwave, and umbrella.
  • Category distributions: Object category distributions converge toward a Zipfian distribution as object-detection datasets increase in scale.The figure presents category distributions for datasets of various sizes.
Loading 2012.09958v1…