PaperlayerView exact revision on arXiv

Source-linked AI summary

A Comprehensive Survey on Applications of Transformers for Deep Learning Tasks

Saidul Islam, Hanae Elmekki, Ahmed Elsebai, Jamal Bentahar, Najat Drawel, Gaith Rjoub, Witold Pedrycz

arXiv:2306.07303v1cs.LGcs.CL

TL;DR

Existing transformer surveys often focus on architecture, efficiency, or specific fields, leaving applications across domains without a comprehensive, methodical analysis. This survey reviews transformer applications across five fields and classifies models by task; it reports that medical image segmentation and classification are the most common medical applications.

  • Problem

    Existing surveys often focus on architecture, efficiency, or specific fields, leaving transformer execution across deep-learning tasks in multiple application domains without comprehensive, methodical analysis.

    Source: §1.1 · p. 2

  • Method

    The authors survey transformer applications across five fields—NLP, computer vision, multimodality, audio and speech, and signal processing—and propose a taxonomy organized by field and task.

    Source: §4 · p. 12, §8 · p. 48

  • Results

    In medical applications, segmentation and classification are the most common transformer-based tasks, followed by image translation.

    Source: §7.2 · p. 47

  • Takeaways & Limitations

    The survey serves as a reference for researchers by summarizing transformer-model advances and research focus across application fields.

    Source: §8 · p. 48

    Studies applying transformers to 3D medical images are scarce, with most medical-image research focused on 2D images.

    Source: §7.2 · p. 47

Abstract

from arXiv · show

Transformer is a deep neural network that employs a self-attention mechanism to comprehend the contextual relationships within sequential data. Unlike conventional neural networks or updated versions of Recurrent Neural Networks (RNNs) such as Long Short-Term Memory (LSTM), transformer models excel in handling long dependencies between input sequence elements and enable parallel processing. As a result, transformer-based models have attracted substantial interest among researchers in the field of artificial intelligence. This can be attributed to their immense potential and remarkable achievements, not only in Natural Language Processing (NLP) tasks but also in a wide range of domains, including computer vision, audio and speech processing, healthcare, and the Internet of Things (IoT). Although several survey papers have been published highlighting the transformer's contributions in specific fields, architectural differences, or performance evaluations, there is still a significant absence of a comprehensive survey paper encompassing its major applications across various domains. Therefore, we undertook the task of filling this gap by conducting an extensive survey of proposed transformer models from 2017 to 2022. Our survey encompasses the identification of the top five application domains for transformer-based models, namely: NLP, Computer Vision, Multi-Modality, Audio and Speech Processing, and Signal Processing. We analyze the impact of highly influential transformer-based models in these domains and subsequently classify them based on their respective tasks using a proposed taxonomy. Our aim is to shed light on the existing potential and future possibilities of transformers for enthusiastic researchers, thus contributing to the broader understanding of this groundbreaking technology.

1 INTRODUCTION

The paper surveys transformer-based models across five application domains and classifies significant models by their deep-learning tasks. It motivates a high-level framework to assess model effectiveness and support exploration of new domains.

1 INTRODUCTION

  • Transformers use multi-head self-attention to capture context across a sequence and enable parallel processing, addressing recurrent networks’ sequential-processing and long-dependency limitations.The survey notes that parallelism and GPU computation can speed tasks with large inputs (Niu et al., 2021, Vaswani et al., 2017, Zheng et al., 2020).Source: §1 · pp. 1–2

1.1 CONTRIBUTIONS AND MOTIVATIONS

  • Existing surveys often focus on specific fields, model performance, or architecture, leaving a need for systematic analysis of transformer contributions across application fields and deep-learning tasks.The paper frames this gap as a lack of comprehensive coverage of major application domains and their task-specific uses of transformer models.Source: §1 · p. 2, §1.1 · p. 2

1 INTRODUCTION

  • The survey identifies NLP, Computer Vision, Multi-Modality, Audio & Speech, and Signal Processing, then classifies significant models by task and discusses future prospects and challenges.It also analyzes top-performing and significant models within the application fields.Source: §1 · p. 2

1.1 CONTRIBUTIONS AND MOTIVATIONS

  • The authors propose a high-level classification framework by application field and task, with criteria for assessing model effectiveness and helping researchers extend transformers to new domains.They describe this as the first review paper to offer such a classification scheme and criteria, and also identify unresolved challenges and future applications.Source: §1.1 · p. 2

2 PRELIMINARIES

The preliminaries introduce transformers as attention-based models whose architectures vary by task, while establishing the encoder-decoder design and attention mechanisms used throughout the survey.

2.1 TRANSFORMER ARCHITECTURE

  • Proposed in 2017 for machine translation, transformer models may use vanilla, encoder-only, or decoder-only architectures, but all rely on self-attention and multi-head attention.Source: §2.1 · p. 3

2.1.1 ATTENTION MECHANISM

  • Scaled dot-product attention uses Query, Key, and Value representations and SoftMax probability scores; self-attention captures local features while reducing reliance on external information.Source: §2.1.1 · p. 3

2.1.2 MULTI-HEAD ATTENTION

  • Multi-head attention runs scaled dot-product attention in parallel with head-specific learnable weights, then concatenates and linearly transforms outputs.It combines distinct attention features to capture diverse characteristics of sequential inputs and improve context representations.Source: §2.1.2 · pp. 3–4

2.2 ARCHITECTURE OF THE TRANSFORMER MODEL

  • The translation transformer stacks encoder and decoder modules as needed, with attention mechanisms executed repeatedly in parallel across multiple heads.Source: §2.2 · p. 4

2.2.1 ENCODER MODULE

  • The encoder combines feed-forward and multi-head attention layers with residual connections and Add-and-Norm, processing embeddings with positional information through Q, K, and V attention.The passage says the feed-forward layer addresses rank collapse, normalization reduces dependencies between layers in gradient computation, and residual connections address vanishing gradients.Source: §2.2.1 · pp. 4–5

2.2.2 DECODER MODULE

  • The decoder masks future positions, attends both among target outputs and between encoder inputs and targets, then uses linear and SoftMax layers to generate tokens until the sequence ends.Source: §2.2.2 · p. 5

3 RESEARCH METHODOLOGY

The survey systematically searched and evaluated transformer research, identified five application domains, and selected influential models for task-based classification and analysis.

3 RESEARCH METHODOLOGY

  • The authors searched established engines for relevant surveys, screened them for relevance, and then systematically searched journal and conference papers across five application domains.The domains were NLP, computer vision, multi-modality, audio/speech, and signal processing.Source: §3 · p. 5
  • After examining more than 600 models, the authors selected a smaller set for deeper analysis based on novelty, attention innovations, impact, real-world performance aims, and community interest.The criteria included models opening new research paths, introducing attention or position-encoding approaches, earning citations or recognition, and targeting real-world applications.Source: §3 · pp. 5–6
  • The search and selection process organized the reviewed papers by application field, popular tasks, relevant-paper counts, and the models selected for taxonomy discussion.Table 1 summarizes these categories and counts.Source: §3 · p. 5, Table 1 · p. 7
  • The authors classified selected models by task and developed a taxonomy, examining model structures, characteristics, operations, and datasets to discuss future applications.They consulted prominent research repositories, including AAAI, ACM, ACL, CVPR, ICML, ICLR, IEEE, SCOPUS, and arXiv.Source: §3 · p. 7

4 RELATED WORK

The section compares this survey with existing transformer reviews, many of which focus on architecture, efficiency, or individual application domains. It presents a task-oriented survey spanning five application fields as the gap this work addresses.

Comparison with existing surveys

  • Among more than 50 located surveys, the authors selected 17 for analysis and found that many emphasize architecture or efficiency, while others focus on NLP or computer vision.The section says only a few examined text-image multimodal combinations; this survey instead organizes models around tasks and applications rather than architectural design and experimental value.Source: §4 · pp. 7, 9, 11

NLP surveys

  • NLP reviews address pretraining and language-model taxonomies or specific tasks such as summarization, BERT word embedding, emotion detection, and organizational text mining (Subramanyam et al., 2021b; Wang et al., 2020a; Kaliyar, 2020; Acheampong et al., 2021; Gruetzemacher & Paradice, 2022).The section contrasts these task- or domain-focused reviews with its coverage of five application fields.Source: §4 · pp. 8–9

Computer-vision surveys

  • Computer-vision surveys classify models by recognition tasks and examine datasets, performance, or vision-task groupings; this survey also emphasizes medical images and distinguishes medical image tasks from medical signals (Khan et al., 2022).The comparison notes that some medical-image reviews do not cover medical signals.Source: §4 · pp. 8, 10, 11

Multimodal and other application domains

  • Multimodal reviews cover vision-language and video-language models or visual-linguistic tasks, while the survey’s scope additionally includes NLP, computer vision, audio and speech, and signal processing.The section also notes transformer applications in time series and reasoning tasks.Source: §4 · pp. 8, 11

Novelty and scope

  • This survey identifies NLP, computer vision, multi-modality, audio and speech, and signal processing as five application fields, classifies significant models by task, and discusses future prospects.It aims to provide a combined reference on models’ contributions, characteristics, and execution methods across these fields.Source: §4 · pp. 9–12

5 TRANSFORMER APPLICATIONS

The survey reviews more than 650 transformer-based models from 2017 onward across NLP, computer vision, multimodal applications, audio and speech processing, and signal processing. It summarizes their applications and impact and uses their distribution across fields to identify areas for further research.

5 TRANSFORMER APPLICATIONS

  • The survey reviewed more than 650 transformer-based models from 2017 onward across multiple fields.Transformers were initially developed for long-range text processing, then applied beyond NLP, including computer vision, where their global attention capability was explored alongside CNNs’ local-feature tracking.Source: §5 · p. 12
  • The analysis overviews models and applications in each major field and aims to help identify further research areas and tasks for transformers.The five fields identified are NLP, computer vision, multimodal applications, audio and speech processing, and signal processing.Source: §5 · p. 12
  • About 250 NLP models represented around 40% of the collection; computer vision accounted for about 200 models, including natural images (20%) and medical images (11%), or 31% combined.The survey separates natural and medical image processing because they involve different image processing and both fields have grown extensively.Source: §5 · p. 12
  • Multimodal applications included about 90 models (15%), audio and speech about 70 (11%), and signal processing 4% of the collection.Figure 4 presents the percentage breakdown across application fields.Source: §5 · p. 12

6 APPLICATION-BASED CLASSIFICATION TAXONOMY OF TRANSFORMERS

The study proposes a task-based taxonomy to address existing categorizations’ limited coverage of transformer applications, organizing influential models across five application areas.

6 APPLICATION-BASED CLASSIFICATION TAXONOMY OF TRANSFORMERS

  • Existing categorizations did not fully capture the breadth of transformer models and their applications, motivating a more comprehensive taxonomy.The authors reviewed transformer models and classified them by task within their fields of application.Source: §6 · p. 12
  • The taxonomy groups impactful transformer models into NLP, Computer Vision, Multi-modality, Audio and Speech, and Signal Processing.The authors present this organization as a more nuanced framework for understanding transformer applications.Source: §6 · p. 13
  • The proposed taxonomy is intended to help researchers and practitioners identify models and techniques relevant to their applications.Source: §6 · p. 13

6.1 NATURAL LANGUAGE PROCESSING (NLP)

This section surveys influential transformer models across a range of NLP tasks, describing their mechanisms and applications. Its scope spans translation, classification and segmentation, question answering, summarization, text generation, natural-language reasoning, and automated symbolic reasoning.

6.1 NATURAL LANGUAGE PROCESSING (NLP)

  • The NLP taxonomy groups influential transformer models into seven tasks and discusses each task’s models, significance, and working methods to guide model selection.The tasks are translation, summarization, classification and segmentation, question answering, text generation, natural language reasoning, and automated symbolic reasoning.Source: §6.1 · p. 13

6.1.1 LANGUAGE TRANSLATION

  • The Vanilla transformer uses encoder-decoder attention and parallel processing for translation; XLM combines supervised and unsupervised learning, while BART reconstructs text corrupted with noise (Lewis et al., 2020).The Vanilla transformer’s positional information supports simultaneous word processing and long-range dependencies; XLM’s unsupervised approach uses masked and causal language modeling.Source: §6.1.1 · p. 15

6.1.2 CLASSIFICATION & SEGMENTATION

  • Transformers capture long-range context for classification and segmentation, while Charformer learns latent subwords from characters and Switch scales through multiple feed-forward networks at comparable computational cost (Fedus et al., 2021).Switch Transformer has trillions of parameters and was evaluated on 11 tasks, with significant improvement reported for translation, question answering, classification, and summarization (Fedus et al., 2021).Source: §6.1.2 · pp. 16–17

6.1.3 QUESTION ANSWERING

  • GPT-3’s 175 billion parameters supported strong performance across tasks without gradient updates or fine-tuning, while BERT variants use bidirectional context and DistilBERT suits question answering (Brown et al., 2020; Devlin et al., 2019; Sanh et al., 2019).Electra uses masked language modeling with a discriminator network to learn language without memorizing training data (Clark et al., 2020a).Source: §6.1.3 · pp. 18–19

6.1.4 TEXT SUMMARIZATION

  • PEGASUS masks important whole sentences and generates them as output, while T5 uses one parameter set across NLP tasks and can be fine-tuned to perform comparably to task-specific models (Zhang et al., 2020a; Raffel et al., 2020).PEGASUS showed significant performance on unseen summarization datasets (Zhang et al., 2020a).Source: §6.1.4 · p. 20

6.1.5 TEXT GENERATION

  • Transformer-based models have advanced text generation, a task that derives meaning from trained text to produce connected output and supports applications such as long-form documentation.The survey directs readers to Table 7 for models used in text generation.Source: §6.1.5 · p. 20
  • ProphetNET’s n-stream future n-gram prediction supports summarization and question generation, while InstructGPT uses human feedback and reinforcement learning to produce more natural, realistic outputs (Ouyang et al., 2022).CTRL is a 1.63-billion-parameter multilingual text generator fine-tunable for tasks such as question answering and summarization (Keskar et al., 2019); BART can also perform translation.Source: §6.1.5 · p. 21

6.1.6 NATURAL LANGUAGE REASONING

  • Transformers can reason directly over natural-language premises and rules, addressing a task distinct from question answering that traditionally relied on formal representations (Mark A Musen, 1988).Natural-language reasoning derives conclusions from premises and rules rather than locating an answer in a passage.Source: §6.1.6 · p. 21
  • 98% accuracy was achieved by RoBERTa-large on a test set of natural-language premise-and-rule inference, but models trained on easier data reached only 57.7% for T5 and 59.6% for RoBERTa on harder instances (Clark et al., 2020b; Richardson & Sabharwal, 2022).The harder dataset was built by translating hard instances from SAT propositional formulas into natural language, exposing the limits of training on uniformly sampled, easier theories.Source: §6.1.6 · p. 22
  • PRover generates directed-graph proofs with 87% accuracy when determining whether conclusions follow from premises and rules.Graph nodes represent statements and rules, while edges show which statements follow from applying rules.Source: §6.1.6 · p. 22
  • A BERT neural unifier mimics backward chaining to generalize from shallow training queries to multi-step ones, while CLUTRR requires BERT to infer relationships and rules from statements.The neural unifier uses separate fact-checking and unification units; CLUTRR examples include inferring that Jim is Bob’s grandfather from family relations.Source: §6.1.6 · p. 22

6.1.7 AUTOMATED SYMBOLIC REASONING

  • SATformer learns clause interdependencies for end-to-end SAT solving, while TRSAT targets MaxSAT and a standard transformer generated satisfying traces or partial assignments for Boolean and temporal formulas (Shi et al., 2022b; Shi et al., 2021; Hahn et al., 2021).SATformer uses GNN clause embeddings and attention trained to emphasize clause groups that may make a formula unsatisfiable; Hahn et al.’s generated sequences included some not seen during training.Source: §6.1.7 · p. 23
  • GPT-f, a decoder-only prover trained on about 38,000 set.mm proofs, generated novel proofs accepted into mathematical proof libraries and communities (Polu & Sutskever, 2020).The largest investigated model had 36 layers and 774 million trainable parameters.Source: §6.1.7 · p. 23

6.2 COMPUTER VISION

The survey divides computer-vision transformers into natural- and medical-image applications, with natural-image processing receiving greater focus because data are more available. It covers classification, recognition and object detection, segmentation, and image generation.

6.2.1 NATURAL IMAGE PROCESSING

  • Natural-image transformer research is organized around four common tasks: classification and segmentation, recognition and feature extraction, mask modeling prediction, and image generation.The survey gives natural images greater emphasis than medical images, citing their wider data availability and applications.Source: §6.2.1 · p. 24

6.2.2 IMAGE CLASSIFICATION

  • ViT variants target different needs, while BEIT uses masked image modeling and Conformer combines local CNN features with transformer context for vision tasks.ConViT adds an inductive bias, SiT supports self-supervised tasks, and DeiT requires less training data; ViViT uses both encoder and decoder modules. BEIT recovers masked image tokens and performs competitively in classification and segmentation; Conformer uses cross-attention to combine local and global features for classification and object detection/recognition.Source: §6.2.2 · pp. 24–25

6.2.3 IMAGE RECOGNITION & OBJECT DETECTION

  • Recognition and detection models use varied transformer designs, from patch-based ViT and nested-patch TNT to DETR’s direct set prediction and HOTR’s human-object interaction prediction.LoFTR learns feature representations from raw images using encoder and decoder modules, leveraging global context and long-range dependencies for visual feature matching.Source: §6.2.3 · p. 26

6.2.4 IMAGE SEGMENTATION

  • Segmentation covers semantic and instance tasks, with SWIN processing image windows, CMT combining CNN and transformer strengths, and SETR using sequence-to-sequence prediction.SWIN uses shifted windows to reduce parameters and computation; SETR divides images into patch sequences and offers three variants with different sizes and performance levels.Source: §6.2.4 · pp. 26–27

6.2.5 IMAGE GENERATION

  • Image-generation models include autoregressive Image Transformer and I-GPT, while VideoGPT combines likelihood-based and VAE architectures to encode spatio-temporal video correlations.I-GPT predicts pixels autoregressively without using the 2D image and has four parameter-count variants; VideoGPT is described as resource-efficient and performs notably on BAIR Robot Pushing.Source: §6.2.5 · pp. 27–28

6.3 MEDICAL IMAGE PROCESSING

The survey reviews transformer models for medical-image processing, where their application is relatively new, across segmentation, classification, and image translation.

6.3.1 MEDICAL IMAGE SEGMENTATION

  • Medical image segmentation commonly uses encoder-decoder architectures and pixel-wise cross-entropy or Dice loss for tasks such as lesion detection and cancer classification.Segmentation may also precede classification to improve classification accuracy.Source: §6.3.1 · p. 28
  • For 3D medical-image segmentation, nnFormer reduces attention complexity with volume-based attention, while TransConver and SwinBTS combine transformer and convolutional components.nnFormer uses LV-MSA and GV-MSA; TransConver merges global and local features through cross-attention, and SwinBTS uses an encoder-decoder with ETrans.Source: §6.3.1 · p. 29
  • Other segmentation models target retinal, ultrasound, and skin-lesion images using combined CNN-transformer features or attention mechanisms adapted to image context.MTPA Unet uses multi-resolution features and position attention for retinal images; TFNet fuses features with transformer and channel attention, while FTN uses sliding-window tokenization and spatial-pyramid attention.Source: §6.3.1 · pp. 29–30

6.3.2 MEDICAL IMAGE CLASSIFICATION

  • Medical-image classification examples include a CCT model for malaria-parasite detection in red blood cell images, using sequence pooling instead of class tokens.The model also uses Grad-CAM to validate its learning process and is reported to perform well compared with other deep-learning models such as CNNs.Source: §6.3.2 · p. 30

6.3.3 MEDICAL IMAGE TRANSLATION

  • For medical image translation, MMTrans uses Swin-based registration and generation modules to translate MRI without requiring paired, pixel-aligned images, while TransCBCT generates synthetic CBCT images.TransCBCT combines transformer and convolutional modeling; the cited comparison notes that CCTR experiments used a 1,010-patient lung-image database versus 91 patients for TransCBCT.Source: §6.3.3 · p. 31

6.4 MULTI-MODALITY

The section surveys transformer models that combine modalities—especially text, images, video, and speech—across question answering, classification, captioning, reasoning, generation, and cloud-computing tasks.

6.4.1 VISUAL QUESTION ANSWERING

  • Visual question-answering models combine text and image or video inputs; VIOLET models temporal-spatial video information and uses masked visual-token, language, and visual-text pretraining.VIOLET was evaluated on 12 datasets and achieved state-of-the-art performance on various benchmarks (Fu et al., 2021); surveyed approaches also include single-stream and two-stream BERT variants and GIT image-to-text generation.Source: §6.4.1 · pp. 31–33

6.4.2 CLASSIFICATION & SEGMENTATION

  • Multimodal classification and segmentation use multiple input modalities, with surveyed models including CLIP, VATT, MBT, ALIGN, and Florence.MBT’s bottleneck reduces processed data, model size, training time, and computational cost; VATT covers video, audio, and text downstream tasks (Akbari et al., 2021), while ALIGN uses noisy image-text pairs and contrastive losses (Jia et al., 2021).Source: §6.4.2 · pp. 34–35

6.4.3 VISUAL CAPTIONING

  • Visual captioning generates textual image descriptions; BLIP filters noisy captions for downstream tasks, while SimVLM uses weak supervision and full-image patches.BLIP supports image captioning, question answering, and image-text retrieval; SimVLM is also noted for question answering.Source: §6.4.3 · pp. 35–36

6.4.4 VISUAL COMMONSENSE REASONING

  • Visual commonsense reasoning uses image features and object relationships to infer interactions; Unicoder-VL uses cross-modal pretraining, while ViLT processes whole images without convolutional visual embedders.Unicoder-VL uses masking and cross-modeling, and ViLT uses downstream data augmentation for classification and retrieval (Kim et al., 2021b).Source: §6.4.4 · pp. 36–37

6.4.5 IMAGE/VIDEO/SPEECH GENERATION

  • Multimodal generation spans text-to-image models and speech translation, with CogView and GLIDE also reported to outperform DALL-E in some cases.DALL-E uses a 12-billion-parameter GPT-3 architecture to generate images from text; Chimera proposes an attention-based Shared Semantic Projection Module to reduce speech-translation latency and errors. CogView uses a transformer-GPT architecture (Ding et al., 2021), while GLIDE generates and edits images through diffusion.Source: §6.4.5 · pp. 37–38

6.4.6 CLOUD COMPUTING

  • Cloud models address workload forecasting, edge caching, job scheduling, and live-streaming resource allocation using attention, vision-transformer, actor-critic, or communication-transformer mechanisms.VMD and R-Transformer capture long-term dependencies and local nonlinearities for workload forecasting; TEDGE uses ViT to predict popular content without preprocessing or added context. ACT4JS uses Actor and Critic networks to schedule jobs, while SACCT adapts resource use to network conditions and demand.Source: §6.4.6 · pp. 38–39

6.5 AUDIO & SPEECH

This section surveys transformer applications in speech recognition, speech separation, and speech classification. It highlights attention architectures and self- or semi-supervised speech representations for these tasks.

6.5.1 SPEECH RECOGNITION

  • Conformer combines Transformer global features with CNN local features and is reported to outperform either architecture alone in automatic speech recognition (Gulati et al., 2020).The section also describes Speech Transformer as an early attention-based alternative to recurrent speech processing, with lower training time and memory use.Source: §6.5.1 · p. 40
  • Wav2vec 2.0 performs well on speech recognition with a small amount of supervised data (Baevski et al., 2020b), whereas BigSSL improves recognition using less labeled and substantial unlabeled speech (Zhang et al., 2022).VQ-Wav2Vec learns by predicting missing parts of corrupted speech (Baevski et al., 2020a); HuBERT clusters features offline and restricts loss calculation to masked regions (Hsu et al., 2021).Source: §6.5.1 · p. 40
  • Transformer Transducer supports real-time transcript prediction, Whisper targets low-volume speech, and XLSR-Wav2Vec enables multilingual recognition with less language-specific labeled data (Zhang et al., 2020b; Radford et al., 2022; Conneau et al., 2021).Whisper uses weak supervision and a larger training corpus, while XLSR-Wav2Vec learns shared relationships among languages.Source: §6.5.1 · pp. 40–41

6.5.2 SPEECH SEPARATION

  • Speech separation models address overlapping speech and noise, with Sepformer using binary-mask self-attention, DPTNeT using end-to-end dual paths, and WavLM predicting clean speech from corrupted input (Subakan et al., 2021; Chen et al., 2020a; Chen et al., 2022a).Separation can serve as preprocessing for speech recognition, since interference can cause recognition errors.Source: §6.5.2 · p. 41

6.5.3 SPEECH CLASSIFICATION

  • Speech classification covers tasks such as language and speaker identification; AST captures long-range spectrogram context, while Mockingjay uses bidirectional context and can help when labels are scarce (Gong et al., 2021).The survey reports that transformer-based classifiers demonstrate superior accuracy to conventional models.Source: §6.5.3 · pp. 41–42
  • XLS-R uses multilingual pretraining for downstream speech tasks, while UniSpeech combines semi-supervised representation learning with multitask learning (Babu et al., 2022; Chen et al., 2022b).XLS-R maps raw speech through convolutional encoders into a Transformer; UniSpeech uses a quantizer to capture speech-recognition information.Source: §6.5.3 · p. 42

6.6 SIGNAL PROCESSING

The section surveys transformer applications in signal processing, focusing on wireless-network and medical signals, which have distinct processing methods and functionalities.

6.6.1 WIRELESS NETWORK & SIGNAL PROCESSING

  • Wireless models address signal reception, denoising, interference identification, modulation classification, and channel-feedback quantization.SigT unifies detection, channel estimation, and decoding for MIMO-OFDM signals; TSDN estimates signal AoA, especially in NLoS environments. ACNNT combines CNN local-feature extraction with transformer global relationships for interference identification; MCformer combines CNN and self-attention for modulation classification; Quan-Transformer performs quantization for channel feedback in RIS-aided networks.Source: §6.6.1 · pp. 43–44

6.6.2 MEDICAL SIGNAL PROCESSING

  • Biomedical-signal transformer research is still early, with most studies from 2021–2022 focusing on classification, including epilepsy prediction and detection.A three-encoder model processes EEG time, frequency, and channel features for seizure prediction and outperforms CNNs; TransHFO classifies MEG high-frequency oscillations, with a shallow transformer favored over a deeper one given the small dataset, even after ADASYN augmentation.Source: §6.6.2 · p. 44
  • ECG transformers classify arrhythmias and other heart diseases using temporal information and multiple signal features.The constrained transformer combines CNNs and transformers for arrhythmia classification; CAT used data from over ten thousand patients, while CRT-NET extracts waveform, morphological, and time-domain features to identify cardiovascular diseases.Source: §6.6.2 · pp. 44–45
  • A TCN-plus-transformer system classifies sleep stages from one-dimensional heart-rate signals and outperformed CNN and RNN alternatives on 5,000 participants.It extracts signal features with a temporal convolution network, then models sequence dependencies with transformer encoders and a simplified decoder.Source: §6.6.2 · p. 45

7 FUTURE PROSPECTS AND CHALLENGES

The survey identifies opportunities to extend transformers beyond NLP and computer vision, including speech recognition, recommendation, natural language generation, and multimodal audio–visual tasks.

7.1 TRANSFORMERS IN WIRELESS NETWORK AND CLOUD COMPUTING

  • Transformers could improve wireless-interference detection by exploring successive models that reduce complexity and increase recognition efficiency.The stated goal is more resilient, better-performing communication systems.Source: §7.1 · p. 46
  • For 5G and 6G, transformers could support RIS and MIMO-OFDM, signal-state feedback, and simultaneous signal detection, channel estimation, and data decoding.The survey also proposes combining transformers with Massive MIMO, millimeter-wave communication, and cognitive radio to improve network performance, capacity, and spectrum efficiency.Source: §7.1 · p. 46
  • Transformers could aid cloud and IoT security through threat and anomaly detection, while supporting task scheduling, memory allocation, workload prediction, resource allocation, and load balancing.Multi-head and long-range attention are identified as useful for resource allocation and cloud performance; transformers could also support data anonymization and privacy-preserving computation.Source: §7.1 · p. 46
  • In 6G systems, transformers could improve mobile edge computing and caching by predicting popular content, managing content and resources, and accelerating requests and data transmission.The passage links these applications to reduced communication latency and faster network response times.Source: §7.1 · p. 46

7.2 MEDICAL IMAGE & SIGNAL PROCESSING

  • Medical-image transformer research is nascent and concentrated on segmentation and classification; challenges include limited 3D work, small private datasets, high-resolution computation, and few mature models.The survey also reports image translation applications and reuse of BERT, ViT, and SWIN; CNN/GAN hybrids and knowledge distillation are discussed as responses to computational and resource constraints.Source: §7.2 · p. 47
  • Biomedical-signal applications remain comparatively limited, with small datasets and few dedicated transformer models constraining training, validation, and further development.One cited study included only 20 patients; virtual sample generation such as ADASYN is proposed to mitigate small-dataset limitations.Source: §7.2 · p. 47

7.3 REINFORCEMENT LEARNING

  • In reinforcement learning, transformers support architecture enhancement and trajectory optimization, but sequence-modeling approaches may overlook traditional RL strengths, motivating their integration.Offline RL can learn policies from static datasets where environment interaction is difficult or costly, while standard transformer structures may face stability issues; modified architectures such as GtrX are proposed.Source: §7.3 · pp. 47–48

7.4 OTHER PROSPECTS

  • Transformer prospects extend across scientific, industrial, and artistic applications, while artistic image generation remains relatively unexplored.Examples include protein modeling, JavaScript bug repair (Lajkó et al., 2022), battery state-of-charge prediction, mechanical-vibration classification, heterogeneous-graph meta-path construction, and AffectGAN for text- and emotion-conditioned images (Galanos et al., 2021).Source: §7.4 · p. 48

8 CONCLUSION

The survey reviews transformer applications across five deep-learning fields and proposes a taxonomy organized by fields and tasks. It summarizes research progress and identifies emerging fields for future investigation.

8 CONCLUSION

  • The survey proposes a taxonomy of transformer applications across NLP, Computer Vision, Multi-Modality, Audio & Speech, and Signal Processing, organized by their respective tasks.It examines advancements in each field to describe current research focus and progress.Source: §8 · p. 48
  • Transformers capture long-term dependencies and leverage parallel computation, supporting their use across fields including NLP and computer vision.The conclusion says they have demonstrated superior performance to traditional recurrence-based models in processing sequential data.Source: §8 · p. 48
  • The authors plan to extend their investigation to wireless networks, cloud computing, reinforcement learning, and other emerging fields.The aim is to uncover new possibilities for transformer utilization.Source: §8 · p. 48
  • The survey aims to serve as a reference for researchers and inspire further exploration of transformers across tasks.The conclusion describes the expanding range of applications as evidence of transformer versatility and potential for continued growth.Source: §8 · p. 48

Related papers

Loading 2306.07303v1…