Source-linked AI summary
A Comprehensive Survey on Applications of Transformers for Deep Learning Tasks
Saidul Islam, Hanae Elmekki, Ahmed Elsebai, Jamal Bentahar, Najat Drawel, Gaith Rjoub, Witold Pedrycz
TL;DR
Existing surveys often emphasize transformer architecture, efficiency, or individual fields rather than examining deep learning tasks across major application domains. This paper surveys transformer models from 2017 to 2022 across five fields and proposes a task-based taxonomy, providing a reference for understanding current research focus and progress.
Problem
Existing transformer surveys often focus on architecture, efficiency, or specific fields, leaving limited comprehensive analysis of deep learning tasks across multiple application domains.
Method
The paper surveys transformer-based models from 2017 to 2022, identifies five major application fields, and classifies models by field and task using a proposed taxonomy.
Results
The survey covers NLP, Computer Vision, Multi-Modality, Audio and Speech, and Signal Processing, and summarizes transformer applications and progress across their deep learning tasks.
Takeaways & Limitations
The resulting taxonomy and overview serve as a reference for researchers examining transformer applications, current research focus, and progress across these fields.
Takeaways & Limitations
Medical-image transformer research is constrained by scarce 3D studies, small private databases, and computational challenges for high-resolution imaging.
Abstract
from arXiv · showhide
Transformer is a deep neural network that employs a self-attention mechanism to comprehend the contextual relationships within sequential data. Unlike conventional neural networks or updated versions of Recurrent Neural Networks (RNNs) such as Long Short-Term Memory (LSTM), transformer models excel in handling long dependencies between input sequence elements and enable parallel processing. As a result, transformer-based models have attracted substantial interest among researchers in the field of artificial intelligence. This can be attributed to their immense potential and remarkable achievements, not only in Natural Language Processing (NLP) tasks but also in a wide range of domains, including computer vision, audio and speech processing, healthcare, and the Internet of Things (IoT). Although several survey papers have been published highlighting the transformer's contributions in specific fields, architectural differences, or performance evaluations, there is still a significant absence of a comprehensive survey paper encompassing its major applications across various domains. Therefore, we undertook the task of filling this gap by conducting an extensive survey of proposed transformer models from 2017 to 2022. Our survey encompasses the identification of the top five application domains for transformer-based models, namely: NLP, Computer Vision, Multi-Modality, Audio and Speech Processing, and Signal Processing. We analyze the impact of highly influential transformer-based models in these domains and subsequently classify them based on their respective tasks using a proposed taxonomy. Our aim is to shed light on the existing potential and future possibilities of transformers for enthusiastic researchers, thus contributing to the broader understanding of this groundbreaking technology.
1 INTRODUCTION
Transformers address key limitations of earlier sequence-to-sequence networks by capturing longer dependencies and enabling parallel processing. The survey is motivated by the lack of a comprehensive analysis of transformer applications across major fields and tasks.
- Transformers have become a major deep-learning approach alongside task-specific architectures such as multilayer perceptrons, convolutional networks, and recurrent networks.
- Transformers use self-attention to address short-term dependency and sequential-processing limitations in earlier sequence-to-sequence architectures.This design supports more effective contextual modeling while removing the constraint that input processing must proceed sequentially.
- Existing surveys often focus on architecture, efficiency, or individual fields rather than transformer execution across tasks and application domains.
- The paper proposes a high-level classification framework that categorizes prominent transformer models by application field and evaluates their task performance.
2 PRELIMINARIES
The transformer is an attention-based architecture for sequential data that combines multi-head attention with encoder–decoder processing. Its components transform embedded inputs into contextual representations and generate outputs while preventing access to future decoder predictions.
- ATTENTION MECHANISM: Attention emphasizes contextually important information, and self-attention helps transformers capture long-range sequence dependencies without relying on external information.
- ATTENTION MECHANISM: Scaled dot-product attention uses Query, Key, and Value matrices, where Q and K have dimension d_k and V contains the value vectors.
- MULTI-HEAD ATTENTION: Multi-head attention applies attention in parallel with separate learnable projections, then concatenates and linearly transforms the head outputs.
- TRANSFORMER ARCHITECTURE: The encoder stack combines embedding and positional encoding, multi-head attention, feed-forward layers, residual connections, and Add and Norm layers.
- TRANSFORMER ARCHITECTURE: The decoder adds masked multi-head attention so predictions exclude future outputs and attend only to previous outputs.
3 RESEARCH METHODOLOGY
The survey methodology combines broad literature searches with expert-informed domain identification, systematic paper selection, and taxonomy construction. It identifies five application domains and classifies selected models by tasks, architecture, characteristics, operation, and datasets.
- Domain identification: The authors identified NLP, computer vision, multi-modality, audio and speech, and signal processing as the five principal transformer application domains.
- Literature search: They searched journal and conference literature using domain-specific transformer, attention, self-attention, and task-related keywords.
- Model selection: Selected models were chosen for introducing new task directions, novel attention or positional-encoding approaches, scientific impact, real-world performance goals, or substantial community interest.
- Survey organization: Table 1 organizes application fields, search keywords, popular tasks, relevant-paper counts, and selected-model counts for taxonomy and discussion.
- Taxonomy construction: The analysis classified models by task execution and examined their structures, characteristics, operational methods, and datasets.
4 RELATED WORK
Existing transformer surveys commonly concentrate on particular domains, tasks, architectures, or model properties. This survey distinguishes itself by covering five application fields and classifying transformer models by tasks across them.
- Survey scope: More than 50 transformer surveys were examined, with 17 significant surveys selected for in-depth analysis.Selection considered publication in reputed conferences or journals and citation impact, while unpublished papers were discarded.
- Prior coverage: Existing surveys often focus on individual domains such as NLP, computer vision, text summarization, emotion detection, or biomedical language processing.Other surveys emphasize architecture, efficiency, visualization, computational power, memory efficiency, or pretraining rather than cross-domain task applications.
- Coverage distinction: This survey covers five application fields: NLP, computer vision, multi-modality, audio/speech, and signal processing.Several compared surveys are limited to language and vision, NLP alone, computer vision alone, or multimodal NLP–CV tasks.
- Taxonomy distinction: The survey compares transformer models by their task execution and summarizes downstream tasks and commonly used datasets.This task-oriented organization differs from taxonomies based primarily on architecture, recurrence, memory capacity, pretraining datasets, embeddings, or language.
- Contribution: Unlike surveys focused only on computer vision, this work also includes NLP, multi-modality, audio/speech, and signal processing.Within computer vision, it separates natural and medical image processing and discusses tasks including visual question answering, classification, and segmentation.
5 TRANSFORMER APPLICATIONS
Transformers emerged for long-range textual processing but expanded beyond NLP as researchers explored their global attention capability in computer vision and other fields.
- Application expansion: Transformers were originally developed for processing long-range textual sentences, but their applications expanded beyond NLP.The expansion followed successes in NLP and included exploration of global attention in computer vision.
6 APPLICATION-BASED CLASSIFICATION TAXONOMY OF TRANSFORMERS
The study found that existing categorizations did not fully represent transformers’ diverse practical applications. It therefore organizes impactful models into five application areas through a more comprehensive taxonomy.
- Motivation: Existing categorizations did not fully capture the wide range of transformer models and their diverse applications across fields.The proposed taxonomy was designed to reflect practical applications more comprehensively.
- Application distribution: Figure 4 presents the proportion of transformer applications across the top five fields.
- Application areas: The taxonomy organizes impactful transformer models into NLP, computer vision, multi-modality, audio and speech, and signal processing.Figure 5 presents this organization as a framework for understanding the models’ diverse applications.
- Interpretive framework: The taxonomy provides a more nuanced framework for understanding transformer applications across the five identified areas.The study states that this framework makes the diverse applications of transformer-based models more explicit.
6.1 NATURAL LANGUAGE PROCESSING (NLP)
This section surveys transformer models across major NLP tasks, organizing influential models by task and describing their architectures and applications. It also covers reasoning-oriented models, including approaches for natural-language inference and automated proof generation.
- NLP task taxonomy: The taxonomy organizes influential NLP transformers across translation, summarization, classification and segmentation, question answering, text generation, natural language reasoning, and automated symbolic reasoning.
- Language translation: The original 2017 Transformer uses encoder and decoder modules with multi-head and masked multi-head attention for language translation.Its parallel computation and positional information support efficient processing of large text volumes and long-range dependencies.
- Text generation: GPT models use decoder-only Transformer blocks; GPT-2 has 1.5 billion parameters, while GPT-3 has 175 billion and performs across tasks without gradient updates or fine-tuning.
- Natural language reasoning: Natural language reasoning derives conclusions from premises and rules represented in natural language, distinct from question answering.Transformers allow reasoning directly in natural language rather than requiring formal representations.
- Natural language reasoning: PRover is an interpretable joint Transformer that generates directed-graph proofs with 87% accuracy for premise-and-rule reasoning.Graph nodes represent statements and rules, while edges indicate statements derived by applying rules.
- Automated symbolic reasoning: GPT-f is a decoder-only Transformer prover trained on approximately 38,000 proofs that generated novel proofs accepted into mathematical proof libraries.
6.2 COMPUTER VISION
This section surveys Transformer applications in computer vision, especially natural-image processing. It groups models by classification, recognition and object detection, and segmentation, while describing architectures that combine local and global visual information.
- Application scope: The survey identifies image classification and segmentation, recognition and feature extraction, and related natural-image tasks as major computer-vision applications.
- Classification: Conformer combines CNN-based local features with Transformer-based global context and long-range dependencies through cross-attention.
- Classification: ViT represents images as sequences of patches and uses only the Transformer encoder, demonstrating an alternative to convolutional neural networks for classification.It can also be applied to image-recognition tasks.
- Recognition and object detection: DETR formulates object detection as direct set prediction, using an encoder, object queries, and self-attention to model relationships between queries and image objects.
- Segmentation: Image segmentation partitions images by objects and boundaries, with semantic segmentation labeling class-consistent regions and instance segmentation identifying object instances and boundaries.
6.3 MEDICAL IMAGE PROCESSING
The survey reviews transformer applications in medical image processing, emphasizing segmentation, classification, and image-to-image translation. These models combine attention with CNN or specialized transformer components to capture local, global, and long-range image information.
- Motivation: Medical image diagnosis is described as complicated, time-consuming, error-prone, and subjective, motivating deep learning alternatives.
- Image Segmentation: Medical image segmentation groups image regions by category and commonly uses encoder-decoder architectures, with segmentation sometimes preceding classification.
- Image Segmentation: nnFormer segments 3D medical images using volume-based and global volume-based multi-head self-attention to reduce computational complexity.
- Image Segmentation: TransConver processes 2D and 3D brain tumor images by combining CNN and Swin transformers in parallel and merging global and local features through cross-attention.
- Image Segmentation: MTPA Unet combines CNN and transformer architectures to capture local and global retinal image information, including long-term pixel dependencies and positional context.
- Image Translation: Image-to-image translation can synthesize medical images from noncorrupted sources and prepare images for registration or segmentation.
6.4 MULTI-MODALITY
The survey covers transformer applications across multi-modal tasks that combine text, images, video, or speech. It highlights visual question answering, cross-modal representation, image generation, video-language understanding, and cloud-computing applications.
- Overview: Multi-modal models process and combine different data types, commonly including natural language, vision, and speech.
- Visual Question Answering: Visual question answering takes an image or video and text as input and produces a textual answer by jointly understanding visual and textual information.
- Video-Text Understanding: VIOLET associates video with text using Video Swin Transformer, language embedding, and cross-modal transformer modules, with evaluation on 12 datasets.
- Text-to-Image Generation: GLIDE generates and edits images conditioned on textual information through a diffusion process that progressively adds and removes noise.
- Cloud Computing: VMD and R-Transformer combines multi-head attention for long-term dependencies with local techniques for nonlinear workload relationships, targeting more precise cloud workload forecasting than RNNs.
6.5 AUDIO & SPEECH
The survey examines transformer models for speech recognition, separation, and classification. The reviewed approaches use attention to model long-range context while some combine transformers with convolution, masking, or multilingual pretraining.
- Speech Recognition: Speech recognition converts human speech into textual or written form and involves acoustic and language models.
- Speech Recognition: Conformer combines transformer global-feature modeling with CNN local-feature modeling and has been reported to outperform each individually for automatic speech recognition.
- Speech Recognition: Speech Transformer replaces the conventional RNN-based speech-processing approach with attention, reducing training time and memory usage for scalable long-term-dependency tasks.
- Speech Recognition: XLSR-Wav2Vec2 recognizes speech across multiple languages without extensive labeled data for each language by learning shared cross-lingual characteristics.
- Speech Separation: Sepformer separates overlapping speech without recurrence by using self-attention and binary mask prediction to capture short- and long-term dependencies.
- Speech Classification: AST applies a transformer encoder directly to audio spectrograms without convolution and captures long-range frame context for audio classification.
6.6 SIGNAL PROCESSING
The survey presents transformer applications in wireless, cloud, and medical signal processing. These models address signal detection, estimation, denoising, classification, decoding, and biomedical prediction by combining attention with task-specific architectures.
- Overview: Transformer use in signal processing is presented as a recent approach that can outperform conventional methods, while wireless-network applications remain limited.
- Wireless Signal Processing: SigT uses a transformer encoder to process MIMO-OFDM signals in parallel while integrating signal detection, channel estimation, and data decoding.
- Wireless Signal Processing: ACNNT combines CNN layers for localized signal features with a transformer for global relationships in wireless interference identification.
- Medical Signal Processing: The three-tower transformer network performs binary EEG seizure classification using separate time, frequency, and channel encoders.
- Medical Signal Processing: TransHFO uses a BERT-based transformer to detect high-frequency oscillations in one-dimensional MEG data for more precise epileptic-area identification.
- Medical Signal Processing: A TCN and Transformer system classifies sleep stages from heart-rate signals by extracting features with TCN and modeling sequence dependencies with a transformer.
7 FUTURE PROSPECTS AND CHALLENGES
The survey identifies future opportunities and challenges for transformers across wireless communication, cloud computing, medical data, reinforcement learning, and other applications. It highlights promising uses while noting data scarcity, computational demands, limited model development, and trade-offs in sequence-modeling approaches.
- Wireless Signal Communication: Transformers may broaden wireless applications through interference detection, 5G/6G networking, and integration with advanced communication technologies.Proposed directions include reducing interference, supporting signal detection and channel estimation, and improving network capacity and spectrum efficiency.
- Mobile Edge Computing: Transformers could enhance mobile edge computing and caching by predicting popular content, managing resources, and reducing response times and transmission delays.Their parallel computational capabilities are presented as useful for faster processing of user requests in advanced 6G systems.
- Cloud Computing: Cloud applications include workload prediction, resource allocation, load balancing, threat detection, anomaly detection, and privacy protection.The survey also identifies task scheduling, memory allocation, and IoT-oriented cloud management as potential uses.
- Medical Image and Signal Processing: Medical transformer research mainly addresses image segmentation and classification, but remains nascent and faces limited 3D coverage, small private datasets, and high-resolution computational demands.Existing work commonly adapts models such as BERT, ViT, and SWIN for 2D, 3D, and multimodal medical images.
- Medical Image and Signal Processing: Biomedical signal applications remain less developed because databases are small and dedicated transformer models are scarce.The survey describes biomedical-signal transformer use as relatively limited compared with medical-image applications.
- Reinforcement Learning: In reinforcement learning, sequence-modeling transformers show promise, but combining them with traditional reinforcement-learning algorithms remains an open direction.The survey notes that sequence modeling may overlook advantages of traditional reinforcement-learning methods.
8 CONCLUSION
The survey reviews transformer applications across five major fields and organizes them into a task-based taxonomy. It summarizes research progress and identifies emerging areas for future investigation.
- The survey proposes a taxonomy covering transformer applications across NLP, computer vision, multi-modality, audio and speech, and signal processing.
- It provides a comprehensive overview of transformer models across different deep learning tasks and examines advancements in each field.
- The authors plan to extend the investigation to wireless networks, cloud computing, reinforcement learning, and other emerging fields.