The NLP Cookbook: Modern Recipes for Transformer based Deep Learning Architectures

I. INTRODUCTION

• Transformers addressed recurrent models’ context limitations with attention and increased throughput by processing inputs in parallel.The introduction describes RNNs as vulnerable to vanishing and exploding gradients on long sequences, and LSTMs as limited by a single context state.Source: §I · p. 1
• Transfer learning improved performance on several NLP tasks, but growing pretraining and fine-tuning data complicated comparisons between data scale and architecture.The introduction contrasts XLNet’s marginal improvement over BERT trained on about 10% as much data with RoBERTa’s later outperformance of XLNet after training on more data.Source: §I · p. 1
• NLG models trained with denoising objectives support sequence generation with zero- or few-shot learning, while large NLU and NLG models can be costly to run.The passage names T5, BART, mBART, and T-NLG as examples and describes distillation, pruning, and quantization as model-size reduction techniques.Source: §I · p. 2
• Knowledge retrievers expose explicit documents to language models, while modified attention architectures target input sequences longer than a fixed token span.The passage also notes other efficiency techniques, including hashing, sparse attention, factorized embeddings, replaced-token detection, and parameter sharing.Source: §I · p. 2

II. RELATED REVIEWS/TAXONOMY

• NLU models use pretraining and task-motivated fine-tuning for classification, structured prediction, and query-generation tasks.Source: §II · p. 2
• NLG models generate clean text from corrupted utterances and support sequence-to-sequence tasks through few- or single-shot learning.Source: §II · p. 2
• Model-size reduction uses distillation, pruning, and quantization to make large models more economical, including for real-time edge deployment.Source: §II · p. 2
• Information-retrieval systems extract documents from large corpora for open-domain question answering and outperform contemporary language models on multiple benchmarks.Source: §II · pp. 2–3
• Long-sequence models address the quadratic attention cost that often limits Transformers to 512 tokens, a constraint that can hinder reasoning across lengthy documents.The passage contrasts this need with coreference resolution, which it describes as benefiting from shorter inputs.Source: §II · p. 3
• Computationally efficient architectures seek memory efficiency and accuracies comparable to large language models, and taxonomy categories are not mutually exclusive.The authors characterize the taxonomy as a generalized categorization in which some models can serve dual purposes.Source: §II · p. 3

III. PRELIMANIRES TO MODERN NLP ARCHITECTURES

• An RNN encoder-decoder jointly encodes an input sequence and generates a different output sequence, maximizing the target’s conditional probability.The encoder produces the input’s encoded version, and the decoder generates its decoded version.Source: §III · p. 3
• The encoder-decoder system is reported to outperform vanilla RNNs, LSTMs, and GRUs on machine translation, sequence mapping, or text summarization tasks.Source: §III · p. 3
• The encoder’s final layer transmits contextual information to the decoder, which uses the combined word representation for the language task.Source: §III · pp. 3–4
• Attention removes reliance on a single final encoder node by weighting inputs individually for target-sequence generation.Alignment scores compare an input position with an output position and use encoder states and the previous decoded hidden state.Source: §III · p. 4
• Luong attention uses global attention over all encoder states or local attention over a subset of words, with the latter helping translation of longer sequences.Source: §III · p. 4

IV-A TRANSFORMERS

• The six-layer encoder-decoder Transformer uses self-attention for context, multi-head attention for parallel perspectives, and causal masking to prevent access to future target tokens.The decoder also attends to encoder outputs when mapping source tokens to target tokens.Source: §IV-A · pp. 4–5

IV-B EMBEDDINGS FROM LANGUAGE MODELS: ELMo

• ELMo combines forward and backward LSTM language models into task-specific deep contextualized word representations intended to capture syntax, semantics, and polysemy.Its contextual representations use token occurrences across the sequence rather than a predetermined embedding.Source: §IV-B · p. 5

IV-B GENERATIVE PRE-TRAINING MODEL: GPT-I

• GPT-I first pre-trains a decoder on unlabeled text, then fine-tunes it on smaller supervised datasets, succeeding on common-sense reasoning, semantic similarity, and reading comprehension.The paper describes this approach as more task-agnostic than ELMo and ULMFiT.Source: §IV-B · p. 6

IV-C BIDIRECTIONAL ENCODER REPRESENTATIONS FROM TRANSFORMER: BERT

• BERT trains bidirectional Transformer encoders with masked-token prediction and a next-sentence classification objective to represent token context and sentence relationships.It randomly selects 15% of input tokens for masking and uses [IsNext] or [NotNext] labels for sentence continuity.Source: §IV-C · p. 7

IV-D GENERALIZED AUTOREGRESSIVE PRETRAINING FOR LANGUAGE UNDERSTANDING: XLNeT

• XLNet shares parameters across token-factorization orders and uses target-aware query and content streams to predict without exposing the target content.Its permutation-based objective is described as building bidirectional context while retaining autoregressive modeling.Source: §IV-D · pp. 7–8

IV-E MEGATRON LANGUAGE MODEL (LM)

• Megatron split its 24 × BERT-sized model across about 512 GPUs, sustaining 15.1 Petaflops and 76% scaling efficiency.The paper attributes its scaling to layer-normalization and residual-connection reordering, and says the model could not fit on a single GPU.Source: §IV-E · p. 9

V-A LANGUAGE MODELS ARE UNSUPERVISED MULTITASK LEARNERS: GPT-II

• The 1.5-billion-parameter GPT-2 outperformed predecessors on most cited tasks in zero-shot settings, with performance improving as model size scaled up.It is described as an extension of the GPT-I decoder-only architecture, trained on more data and capable of unsupervised task learning.Source: §V-A · p. 10

V-B BIDIRECTIONAL AND AUTOREGRESSIVE TRANSFORMERS: BART

• BART reconstructs original text from randomly corrupted input, using flexible noising schemes including sentence shuffling and replacing spans with a single mask.For machine translation, a separately initialized source encoder is first trained with most BART parameters frozen, then all parameters are jointly trained; BART reports state-of-the-art text-generation results and comparative discriminative results to RoBERTa.Source: §V-B · p. 10

V-C MULTILINGUAL DENOISING PRE-TRAINING FOR NEURAL MACHINE TRANSLATION: mBART

• mBART denoising-pretrains on 25 Common Crawl languages and supports supervised or unsupervised language-pair fine-tuning; evaluation demonstrates transfer when source-language bitext is absent.The model is pretrained for all 25 languages and fine-tuned for the target language.Source: §V-C · pp. 10–11

V-D EXPLORING THE LIMITS OF TRANSFER LEARNING WITH A TEXT-TO-TEXT TRANSFORMER: T5

• T5 uses a unified text-to-text setup and C4; its survey finds encoder-decoder models outperform decoder-only models, while denoising suits fill-in-the-blank tasks.The authors also report in-domain data effectiveness, overfitting on small pretraining datasets, and the need to monitor task frequency in multitask training.Source: §V-D · p. 11

V-E TURING NATURAL LANGUAGE GENERATION: T-NLG

• T-NLG is a 17-billion-parameter, 78-layer model whose shared multi-task resources support generalized downstream tasks with fewer fine-tuning samples.The passage also reports greater speedup than Nvidia’s Megatron and question answering without prior context.Source: §V-E · p. 11

V-F LANGUAGE MODELS ARE FEW-SHOT LEARNERS: GPT-III

• GPT-3 uses 175 billion parameters and few-shot learning through a text interface without gradient updates or fine-tuning, achieving state-of-the-art results on several NLP tasks.Its few-shot learning falls short of reproducing similar results on other tasks.Source: §V-F · pp. 11–12

V-G SCALING GIANT MODELS WITH CONDITIONAL COMPUTATION AND AUTOMATIC SHARDING: GShard

• GShard scales multilingual translation beyond 600 billion parameters with sparsely gated experts and automated sharding; increased tasks or language pairs yield positive transfer for low-resource languages.Its strategy uses expert routing, and increased depth helps only when MoE capacity is also relaxed.Source: §V-G · p. 12
• A 3.3-point BLEU gain followed quadrupling experts from 128 to 512 across 100 languages; quadrupling width instead gained 1.3 BLEU, with further depth gains conditional on relaxing capacity.GShard combines increased depth and width with sparse learned token routing.Source: §V-G · p. 12

VI-A DISTILLATION

• Knowledge distillation trains a smaller student under a larger teacher using soft probability targets alongside hard-label supervision.Temperature shapes the soft targets, while the standard loss combines distillation and student losses.Source: §VI · pp. 12–13
• TinyBERT retained over 96% of BERT’s performance using 28% of its parameters and 31% of its inference time across BERT-based distillation models.Its transfer objectives cover embeddings, attention matrices, hidden states, and output logits.Source: §VI · pp. 13–14
• DistilBERT retained 97% of BERT’s language-understanding performance while reducing size by 40% and inference time by 60%.It halves layer depth and combines distillation with masked-language-modeling and embedding-cosine losses.Source: §VI · p. 13
• MobileBERT is 4.3 times smaller and 5.5 times faster than BERT, with a GLUE score 0.6 units lower and 62 ms latency on Pixel 4.The passage attributes the low latency to replacing Layer Normalization and gelu with a simpler Hadamard-product linear transformation.Source: §VI · pp. 13–14

VI-B PRUNING

• Pruning-robust training can use weight-drop subnetwork sampling, with the passage describing layer dropping as an effective strategy across NLP tasks.The passage compares this approach to Dropout and DropConnect,.Source: §VI · p. 14
• 40% pruning retained 98% of performance for BERT, RoBERTa, and XLNet, comparable to DistilBERT.Pruning zeros weights, biases, layers, or activations; iterative pruning removes low-importance weights and retrains the model.Source: §VI-B · p. 14

VI-B.1-A STRUCTURED DROPOUT

• Structured dropout randomly drops layers during training and testing, allowing selection of a desired-depth subnetwork from a pruning-robust network.Source: §VI-B · p. 14

VI-B.1-B POOR MAN’S BERT

• For span K=2, odd-alternate dropping outperformed top and even-alternate dropping; preserving the final layer was more important than preserving other top layers.In a 12-layer network, odd-alternate dropping removes layers 9 and 11, while top dropping removes layers 11 and 12.Source: §VI-B · p. 14
• Symmetric dropping was the second-best BERT strategy and remained robust after removing four layers, while it substantially degraded XLNet performance.XLNet was more pruning-robust overall; dropping 30% of DistilBERT’s layers caused minimal degradation, and a six-layer RoBERTa performed similarly to DistilRoBERTa.Source: §VI-B · pp. 14–15

VI-B.2-A STRUCTURED PRUNING

• Structured pruning trades some performance for hardware-friendly dense matrices, while FLOP combines low-rank factorization with an L0-pruned diagonal mask to retain dense structure.The mask is pruned during training, and an L0 regularizer controls the model’s final sparsity.Source: §VI-B · p. 15

VI-B.3 HEAD PRUNING

• Specialized heads were associated with positional, syntactic, and rare-word functions, while a small subset were key for translation.The reported roles included attending to adjacent tokens, syntactic dependencies, and infrequent tokens.Source: §VI-B · p. 15
• Only 8 of 96 attention heads significantly changed performance when removed, and half of those removals increased BLEU; a single head was sufficient in most layers.The final encoder-decoder attention layer was an exception: using one head reduced performance by 13.5 BLEU points.Source: §VI-B · p. 16
• Head importance is assessed by expected sensitivity to masking, with low-scoring heads iteratively pruned; earlier approaches based on averaged or maximum attention weights did not capture varying head significance.Source: §VI-B · pp. 15–16

VI-C QUANTIZATION

• BERT quantization to 16/8-bit weights achieved 4× model compression with minimal accuracy loss.Source: §VI-C · p. 16

VI-C.1 LQ-NETS

• LQ-Nets learns quantization levels and intervals for weights and activations, using bitwise-compatible operations to support quick inference.Its learnable quantization uses a floating-point basis and binary encodings.Source: §VI-C · p. 16

VI-C.2 QBERT

• QBERT’s Hessian-aware quantization assigns more bits to sensitive layers and quantizes less-sensitive layers more aggressively; fake quantization and straight-through estimation support inference-time quantization during training.Group-wise quantization treats each dense matrix as a group and partitions it by continuous output neuron.Source: §VI-C · pp. 16–17

VII. INFORMATION RETRIEVAL

• Large implicit knowledge storage can make language models costly and slow for knowledge-intensive tasks such as data updating and retrieval.The paper notes that storing additional knowledge in model parameters increases size and latency.Source: §VII · p. 17

VII-A GOLDEN RETRIEVER

• GoldEn Retriever iteratively generates queries from the question and accumulated context, retrieves documents, and appends them for subsequent multi-hop reasoning.Its lightweight RNN extracts text spans from context to form queries.Source: §VII-A · p. 17

VII-B ORQA

• ORQA jointly trains a BERT-based reader and retriever, scoring evidence blocks with a dense question–block inner product and retrieving from an open corpus.The retriever is pre-trained with an Inverse Cloze Task because training latent retrieval over complete Wikipedia from scratch is described as unfeasible.Source: §VII-B · p. 17

VII-C REALM

• REALM retrieves documents from a corpus using embedding inner products, then combines the input and retrieved document for Transformer fine-tuning and cross-attention.Its pre-training predicts masked tokens using retrieval relevance scores.Source: §VII-C · p. 17

VII-D RETRIEVAL AUGMENTED GENERATION: RAG

• RAG combines a parametric sequence-to-sequence model with non-parametric Wikipedia memory accessed by a neural retriever, and is described as outperforming current language models.Its retriever and generator are trained together in a semi-unsupervised manner.Source: §VII-D · p. 18

VII-D DENSE PASSAGE RETRIEVAL: DPR

• DPR uses separate dense encoders for queries and passages to retrieve relevant passages through vector similarity at run time.The method trains embeddings so relevant question–passage pairs have greater similarity than irrelevant pairs.Source: §VII-D · p. 18

VIII. LONG SEQUENCE MODELS

• Vanilla Transformers split inputs longer than 512 tokens into chunks, potentially losing context between related words in different chunks.Source: §VIII · p. 19

VIII-A DEEPER SELF-ATTENTION

• A 64-layer Transformer uses three supporting losses to accelerate convergence and enable deeper networks, including predictions across positions and intermediate layers.It also adds multiple future-target predictions with separate classifiers and half-weighted extra target losses.Source: §VIII-A · p. 19, Figure 19 · p. 19

VIII-B TRANSFORMER-XL

• Transformer-XL reuses and caches prior hidden states across consecutive segments, creating segment-level recurrence to capture longer dependencies.Source: §VIII-B · pp. 19–20

VIII-C LONGFORMER

• Longformer combines sliding-window, dilated-window, and global attention; sliding-window attention has O(n × w) complexity, while global attention enables chunk-less processing.Global attention can require greater space-time complexity than RoBERTa when sequence length exceeds the window size.Source: §VIII-C · p. 20

VIII-D EXTENDED TRANSFORMER CONSTRUCTION: ETC

• ETC combines global and long inputs through four global-local attention patterns and uses relative positional encodings.Global inputs and three attention variations provide broader attention to compensate for the fixed-radius long-to-long span.Source: §VIII-D · p. 20

VIII-E BIG BIRD

• Big Bird combines global tokens, random keys, and local-neighbor windows in a sparse attention pattern, with internal and extended construction options for global tokens.Source: §VIII-E · p. 20

IX-A SPARSE TRANSFORMER

• Sparse Transformer attention reduces complexity to O(N√N) from O(N^2) and deciphers sequences thirty times longer than its predecessors.Its strided pattern uses one head for nearby locations and another for every lth location, with stride l close to √n.Source: §IX-A · pp. 20–21, Figure 22 · p. 21

IX-B REFORMER

• Reformer uses locality-sensitive hashing to reduce attention complexity to O(L log L), grouping nearby vectors into hash buckets with high probability.Multiple hash functions reduce the probability that similar items fall into different buckets, while masking restricts queries to permitted positions.Source: §IX-B · p. 21
• Reformer’s reversible residual blocks reconstruct activations during backpropagation, avoiding the need to store them in memory.The blocks are recomputed from the next layer’s activations.Source: §IX-B · p. 22, Figure 24 · p. 22

IX-C A LITE BERT: ALBERT

• ALBERT combines embedding factorization and cross-layer parameter sharing to reach 12M parameters, nearly 90% fewer than BERT-base while maintaining competitive benchmark performance.Its shared attention parameters are reported to stabilize network parameters.Source: §IX-C · p. 22, Figure 25 · p. 22

IX-D ELECTRA

• ELECTRA’s replaced-token detection trains a discriminator to classify tokens, learning from all input tokens rather than BERT’s 15% masked subset.A small masked-language-model generator makes probabilistic, meaningful token substitutions.Source: §IX-D · p. 23, Figure 27 · p. 23

IX-E LINFORMER

• Linformer projects keys and values into lower-dimensional representations; when k<<n, this yields significant memory and space savings.The section also describes sharing projection parameters across heads, key-value pairs, and layers.Source: §IX-E · p. 23

IX-E PERFORMER

• Performer uses a kernel approximation to decompose attention before softmax, enabling a lower-cost computation when sequence length L is much greater than dimension d.The cited comparison gives O(L^2d) for the quadratic computation and O(d^2L) for the alternative when L>>d.Source: §IX-E · p. 23

• Autoregressive models predict future tokens from token history using causal masking and are generally fine-tuned for text generation.The section notes zero-shot learning in the GPT series.Source: §X · p. 23
• Auto-encoded models mask input tokens during pretraining and are fine-tuned to reproduce them, making them appropriate for sequence or token classification.These encoder models have full access to the input array during learning, without causal masking.Source: §X · p. 23
• Sequence-to-sequence models learn the joint distribution of input and target, often from corrupted inputs, and are used for tasks such as translation, summarization, and question answering.BART and T5 are named examples.Source: §X · p. 24

• T5 performs best in the comparison, which draws on GLUE and SuperGLUE benchmarks covering varied NLP tasks.The authors attribute its performance to multi-task text-to-text learning with a denoising objective and large amounts of unlabeled data.Source: §XI · p. 24, Figure 28 · p. 24
• GPT-3 is the largest model in the comparison and is absent from the older GLUE benchmark because it is newer.Source: §XI · p. 24
• GPT-3 matches state-of-the-art fine-tuned models on several NLP tasks in zero-, one-, and few-shot settings, although it does not rank highly on SuperGLUE.The passage describes GPT-3 as the quickest to learn a task at inference time and reports high-quality samples and qualitative performance on tasks defined on the fly.Source: §XI · p. 24

• The paper’s taxonomy covers language understanding and generation, model-size reduction, information retrieval, long-sequence modeling, and other model-reduction approaches.Source: §XII · p. 24
• The conclusion says model scaling and attention costs make large models infeasible to access globally, prompting efforts toward smaller models and more efficient attention.These efforts are intended to improve convergence speed and reduce latency.Source: §XII · p. 24
• Mixture-of-experts models activate only part of the network per input, but communication costs and GPUs suited to dense computation impede wide-scale deployment.The conclusion describes this approach as promising for computational efficiency.Source: §XII · p. 24
• The authors identify robust mixture-of-experts training and reinforcement learning for translation, summarization, and question answering as future research directions.The conclusion also notes that large models can memorize training data, and describes single-epoch training on deduplicated instances as a way to minimize overfitting.Source: §XII · p. 24