A Comprehensive Study of Vision Transformers on Dense Prediction Tasks

1 INTRODUCTION

• The study addresses how VTs and CNNs differ in generalizability, robustness, reliability, and texture reliance when used for detection and segmentation.CNNs use convolutional layers with local receptive fields, while VTs process non-overlapping tokens with global self-attention.Source: §1 · p. 1
• VTs are better calibrated than CNNs, making their predictions more reliable for safety-critical applications.Source: §1 · p. 2
• At higher inference resolutions, performance degrades for both architectures, but CNNs outperform VTs in object detection.The introduction flags this resolution-dependent reversal as a qualification to the broader comparison.Source: §1 · p. 2
• VTs converge to wider minima than CNNs, which the authors attribute to their greater generalizability.Source: §1 · p. 2
• VTs generalize better to distribution shifts and are more robust to natural corruptions and targeted and untargeted adversarial attacks than CNNs.The introduction also reports that VTs are more accurate but slower on in-distribution data.Source: §1 · p. 2
• VT predictions are less dependent on texture cues than CNN predictions.Source: §1 · p. 2

2 RELATED WORK

• Vision Transformers such as ViT and DeiT were adapted as feature extractors for dense tasks including detection, segmentation, and depth prediction.Source: §2 · p. 2
• Image-classification studies compared VTs and CNNs on robustness, texture bias, and calibration, including the role of architecture in calibration.Source: §2 · p. 2
• The paper addresses the reported absence of comparisons of generalizability, robustness, calibration, and texture bias for VTs replacing CNN feature extractors in detection and segmentation.Source: §2 · p. 2

3 METHODOLOGY

• The methodology compares CNN and VT backbones of different sizes on object detection and semantic segmentation.Source: §3 · p. 3

3.1 Transformer as a Feature Extractor

• DeiT divides images into 16 × 16 patches, embeds them with position information, and processes them through repeated self-attention and feedforward blocks.For dense prediction, the classification and distillation embeddings are removed, and the remaining patch embeddings are reshaped into a feature map and expanded into five spatial scales.Source: §3.1 · p. 3

3.2 Detection Head

• The detection head uses FCOS, an anchor-free, key-point-based detector that predicts class scores, bounding boxes, and centerness across multiple feature-map scales.Source: §3.2 · p. 3

3.3 Segmentation Head

• The segmentation head progressively interpolates and concatenates feature maps, then predicts class probabilities for each input pixel.Source: §3.3 · p. 3

4 EXPERIMENTAL SETUP

• Experiments evaluate detection and segmentation with DeiT-T, DeiT-S, and DeiT-B vision-transformer backbones and CNN counterparts ResNet-18, ResNet-50, and ResNeXt-101, which have the same range of parameters.Detection uses COCO and segmentation uses COCO-Stuff; the datasets consist of 118K training images and 5K validation images.Source: §4 · p. 3
• Models are trained at 512×512 resolution with AdamW, cosine learning-rate scheduling, and task-specific schedules of 55 detection epochs and 45 segmentation epochs.The passage also specifies data augmentation and batch sizes by backbone.Source: §4 · p. 3
• Table 1 compares VT and CNN backbones on COCO detection and COCO-Stuff segmentation, with the best score for each metric marked in bold.Source: Table 1 · p. 4
• Evaluation reports mAP for detection and mIoU for segmentation, alongside parameter count, GMAC, inference time, and energy consumption.The metrics are calculated at training resolution.Source: §4 · p. 4

5.2 Out-of-Distribution Evaluation

• On OOD evaluation, VTs outperform CNN counterparts in both detection and semantic segmentation.Models were trained on COCO or COCO-Stuff and evaluated on BDD100K or BDD10K using 8 or 14 matching classes.Source: Figure 2 · p. 4, §5.2 · pp. 4–5

5.1 In-Distribution Evaluation

• On in-distribution detection and segmentation, VTs are more accurate, while CNNs are faster and consume less energy.Source: §5.1 · p. 4

5.3 Inference Resolution Study

• At higher inference resolutions, CNNs degrade more gradually and surpass VTs in detection, whereas VTs remain ahead in segmentation; positional-embedding interpolation is proposed as a possible explanation.Source: Figure 3 · p. 5, §5.3 · p. 5

5.4 Convergence to Flatter Minima Analysis

• With increasing Gaussian weight noise, VTs outperform CNNs; CNN performance drops more sharply, with RNX-101 showing the steepest decline.Noise standard deviations range from 0.0 to 0.013, and perturbed models are evaluated on 20% of the training data.Source: §5.4 · pp. 5–6, Figure 4 · p. 6

• ECE averages confidence-accuracy gaps across bins, while MCE measures the maximum such gap.Source: §6 · p. 6
• Segmentation calibration uses 15-bin ECE and MCE; detection uses D-ECE and weighted D-ECE with 15 bins, confidence threshold 0.3, and IoU threshold 0.6.Source: §6 · p. 6
• VTs are better calibrated than CNN counterparts in both tasks, with no observed relationship between model size and calibration within either architecture.Source: §6 · p. 6

7.2 Adversarial Robustness

• Across natural corruptions, VTs lead in both tasks; under untargeted attacks they lead detection, but CNNs are more robust in high-strength segmentation attacks.Source: Figure 6 · p. 7, §7.1 · p. 7, §7.2 · p. 7

7.1 Natural Corruption

• Under 15 common corruptions at severity 3, VTs are more robust in detection and segmentation, which the authors attribute to self-attention attending to salient, distant regions.Source: §7.1 · p. 7

7.2 Adversarial Robustness

• For targeted attacks, VTs generally retain a higher percentage of original AP and F1 scores; the evaluation generates attacks with PGD on classification loss.Source: §7.2 · p. 7

8 Texture Bias

• The texture-conflict dataset applies rich object textures to validation images, and texture bias is defined as T/(T + S), where T and S count texture and shape predictions.A prediction matching neither the applied texture nor the original class explains why the basic measure can be low for detection and segmentation.Source: §8 · pp. 7–8
• The basic texture-bias measure is low in detection and segmentation because models often predict neither the original class nor the applied texture.The study therefore uses Texture bias-SC, based on super-categories, to better reflect texture bias.Source: §8 · p. 8
• Vision Transformers show lower texture bias than CNNs under the super-category measure, which captures incorrect predictions when applied-texture and predicted classes share a super-category.The authors suggest global receptive fields let VTs rely more on global shape cues than local texture cues.Source: §8 · pp. 8–9, Table 4 · p. 9
• With cup texture applied, RN-50 predicts “bottle” and “wine glass,” classes in the same super-category as “cup.”This qualitative example illustrates why the super-category measure is used.Source: §8 · p. 8, Figure 9 · p. 9

9 Conclusion

• VTs outperform CNNs on in-distribution data while having lower inference speed but less computational complexity.The authors say Transformer-optimized GPUs could make VTs potential contenders for dominance in computer vision.Source: §9 · p. 9
• VTs generalize better to out-of-distribution datasets, and their flatter loss minima may explain this advantage.The loss-landscape analysis finds flatter minima for VTs than for CNNs.Source: §9 · p. 9
• VTs are better calibrated than CNNs, and architecture plays the primary role in determining model calibration.The authors associate better calibration with more reliable predictions for real-world deployment.Source: §9 · p. 9
• VT performance degrades at higher image resolutions, with interpolated positional embeddings proposed as a possible reason.The authors state this as a proposed explanation, not a confirmed cause.Source: §9 · p. 9
• VTs are more robust than CNNs to natural corruptions and adversarial attacks.The authors suggest global receptive fields and dynamic self-attention may contribute to this robustness.Source: §9 · p. 9
• VTs are less texture-biased than CNNs, with global receptive fields suggested to support greater reliance on shape-based cues.The authors say these findings provide a holistic comparison to inform architecture choice by application.Source: §9 · p. 9

9.1 COCO Texture Stylization for Dense Prediction Tasks

• The study applies random object textures to images in the COCO validation set using AdaIN-style stylization at increasing strengths.A texture is applied only when the source object is absent from the target image.Source: §9.1 · p. 11
• VTs rely less on texture cues than CNNs at all stylization strengths, and their higher performance as strength increases indicates greater robustness to distribution shifts.Source: §9.1 · p. 11
• The example shows zebra texture stylized onto an image containing a person and skateboard at strength 0.4.The source texture is applied only if its object is absent from the target image.Source: Figure 11 · p. 11, §9.1 · p. 11