A Comprehensive Survey of Transformers for Computer Vision

I. INTRODUCTION

• ViTs divide an image into patches, linearly embed them as tokens, encode them with a transformer, and use an MLP head to predict the image class.The encoder uses multi-head attention and normalization.Source: §I · p. 1
• Earlier milestones include a pure transformer outperforming CNNs in image classification and a transformer backend added to ResNet to lower costs while enhancing accuracy.The cited work places these developments in 2021.Source: §I · p. 1
• CNN inductive biases can help models learn image characteristics with less data, whereas ViTs capture broader global relationships but are harder to train.The survey also notes that CNN biases can make global connections harder to recognize.Source: §I · pp. 1–2
• The survey describes approaches intended to train ViTs without massive vision datasets or precise labels, a motivation for their appeal.It frames near-autonomous learning as a possible way to infer problem characteristics without large datasets or precise labeling.Source: §I · p. 2
• The survey classifies ViT use across seven CV tasks, reviews available models and their pros and cons, and presents category-specific lessons and future research directions.The tasks are image classification, object detection, segmentation, compression, super-resolution, denoising, and anomaly detection.Source: §I · p. 3
• The survey situates its chapters from related work to CV applications, then open research challenges and future directions, and finally the conclusion.The organization is shown in Figure 3.Source: §I · p. 3

II. RELATED WORK

• Survey [23] covers transformer theory, foundations, and memory efficiency, but omits computer-vision tasks; survey covers several vision tasks but not denoising or compression.The vision tasks in include classification, segmentation, super-resolution, and object detection.Source: §II · p. 3
• Survey [26] describes transformer architectures for image segmentation, classification, and object detection, but omits super-resolution, denoising, and compression.Source: §II · p. 3
• Survey covers low- and high-level vision, including colorization, super-resolution, generation, segmentation, and detection, but not image compression or classification.The passage also notes that [25] surveys NLP architectures without covering transformer applications to CV.Source: §II · p. 3
• The survey identifies a need for insight across classification, detection, segmentation, compression, denoising, and super-resolution based on gaps in existing survey coverage.The passage cites Table II as the basis for this analysis.Source: §II · p. 3

III. APPLICATIONS OF VITS IN CV

• Modified classical ViTs are used for object detection, segmentation, compression, super-resolution, denoising, and anomaly detection.Source: §III · p. 3
• Table I is titled as a list of acronyms.Source: Table I · p. 4

A. ViTs for Image Classification

• Image patches pass through a transformer encoder and MLP head to predict the class in the classical ViT pipeline.The encoder applies MLP, normalization, and multi-head attention to create embedded patches.Source: A. ViTs for Image Classification · p. 4
• CrossViT variants report 77.1%, 82.3%, and 82.8% accuracy on ImageNet1K, and CrossViT-18 reports 99.11% accuracy on CIFAR10.CrossViT-15 and CrossViT-18 report 99.0% and 99.11% accuracy on CIFAR10, respectively; CrossViT-18 reports 91.36% accuracy on CIFAR100.Source: A. ViTs for Image Classification · p. 4
• For remote-sensing classification, CTNet achieved 97.70% accuracy on AID and 95.49% on NWPU-RESISC45, while ES-GSNet reached 99.29% on UCM.DHViT reported 99.58%, 99.55%, and 96.40% accuracy on Trento, Houston 2013, and Houston 2018, respectively; ForestViT reported 94.28% on PUAS.Source: A. ViTs for Image Classification · p. 4
• MIL-ViT achieved 97.9% accuracy on APTOS2019 and 95.9% on RFMiD2020 for fundus image classification.Source: A. ViTs for Image Classification · p. 4
• LeViT reported 91.56% accuracy on Chinese asphalt pavement and 99.17% on German asphalt pavement, while a ViT achieved 98.3% on a malicious-drone dataset.Source: A. ViTs for Image Classification · pp. 4–5
• SeedViT outperformed CNNs with 96.70% accuracy on a custom maize-seed dataset, while DOViT reported 90.32% and 92.78% accuracy on GAOs-1 and GAOs-2.For femur-fracture classification on real X-rays, the surveyed model achieved 83% accuracy, 77% precision, 76% recall, and 77% F1-score.Source: A. ViTs for Image Classification · p. 5

• Beal et al. [45] paired supervised pre-trained ViT with faster R-CNN, YOLOS used a pre-trained encoder sequence-to-sequence, and Li et al. [47] studied MIM-pre-trained vanilla ViT at scale.Source: B. ViTs for Object Detection · p. 5
• 0.354 F1-score and 0.275 JI were reported for ViT-PP on satellite manipulation dataset 2.Source: B. ViTs for Object Detection · p. 5
• 52.8 mAP@0.5 on ScanNet-V2 was reported by both BrT and the point-cloud framework, while SUN RGB-D results were 55.2 and 45.2, respectively.Source: B. ViTs for Object Detection · p. 6

• 84.36% average dice was reported by TransUNet [55] at image resolution 512, compared with 77.48% at resolution 224.Source: C. ViTs for Image Segmentation · p. 6
• ViTBIS averaged 80.45% dice and 21.24% Hausdorff distance on the MICCAI 2015 dataset.Source: C. ViTs for Image Segmentation · p. 6
• UNetFormer [57] scored dice 96.03% for liver and 59.16% for tumor on MSD, with HD 7.21% and 8.49%, and 91.54% average dice on BraTS 2021.Source: C. ViTs for Image Segmentation · p. 6
• 72.73% IoU on RefCOCO, 62.14% on RefCOCO+, 61.24% on G-Ref UMD, and 60.50% on G-Ref Google were reported for LAVT [60].Source: C. ViTs for Image Segmentation · pp. 6–7

• Entroformer modified the Ballé 2018 model's entropy module with a transformer to capture long-range dependencies in probability estimation.Source: D. ViTs for Image Compression · p. 7
• 27.63 dB PSNR and 0.90132 MS-SSIM were reported on Kodak when optimizing Entroformer for MSE loss.Source: D. ViTs for Image Compression · p. 7

• 34.74 dB PSNR for static video decreased to 30.15 dB, 26.04 dB, and 22.95 dB for medium, fast, and extreme motion, respectively.Source: E. ViTs for Image Super Resolution · p. 7
• The spatio-temporal ViT [67] was developed for microscopic-image super-resolution and also addressed video super-resolution.Source: E. ViTs for Image Super Resolution · p. 7

F. ViTs for Image Denoising

• TED-Net uses transformers in both encoder and decoder for low-dose CT denoising, reporting SSIM 0.9144 and RMSE 8.7681 on the AAPM-Mayo clinic LDCT Grand Challenge dataset.Source: F. ViTs for Image Denoising · p. 7

G. ViTs for Anamoly Detection

• VT-ADL reports mean PRO scores of 0.984 on MNIST, 0.807 on MVTec, and 0.89 on BTAD.Source: G. ViTs for Anamoly Detection · p. 7
• AnoViT reports mean AUROC values of 92.4, 60.1, and 78 on MNIST, CIFAR, and MVTecAD, while TransAnomaly reports AUC values with and without sliding windows on Pred1, Pred2, and Avenue.TransAnomaly's AUC values without sliding windows are 84.0%, 96.10%, and 85.80%; with sliding windows they are 86.70%, 96.40%, and 87.00%, respectively.Source: G. ViTs for Anamoly Detection · p. 7

IV. OPEN RESEARCH CHALLENGES AND FUTURE DIRECTIONS

• The survey identifies computational cost, large training datasets, neural architecture search, interpretability, and hardware efficiency as implementation challenges for ViTs.Source: §IV · p. 7

A. High computational cost

• ViT-based models have millions of parameters and require high-computational-power computers for training.Source: A. High computational cost · p. 7
• ViTs perform better than CNNs in the comparison described, but their computational cost is much higher, making cost reduction a major research challenge.Source: A. High computational cost · p. 8

B. Large training dataset

• ViT performs worse than ResNet when trained on ImageNet1K but better than ResNet when trained on ImageNet21K.Source: B. Large training dataset · p. 8

C. Neural architecture search (NAS)

• Neural architecture search has been explored extensively for CNNs but not yet for ViTs, which the survey presents as a future research direction.Source: C. Neural architecture search (NAS) · p. 8

D. Interpret-ability of the transformers

• ViT attention from each layer becomes intermixed in succeeding layers, making input-token contributions to predictions difficult to visualize.Source: D. Interpret-ability of the transformers · p. 8

E. Hardware efficient desings

• Power and processing requirements can make large-scale ViTs unsuitable for edge devices and resource-constrained settings such as IoT.Source: E. Hardware efficient desings · p. 8

V. CONCLUSION

• The survey presents future directions intended to give young researchers a new perspective on ViT research challenges.Source: §V · p. 8