PaperlayerView exact revision on arXiv

Source-linked AI summary

Efficient Methods for Natural Language Processing: A Survey

Marcos Treviso, Ji-Ung Lee, Tianchu Ji, Betty van Aken, Qingqing Cao, Manuel R. Ciosici, Michael Hassid, Kenneth Heafield, Sara Hooker, Colin Raffel, Pedro H. Martins, André F. T. Martins, Jessica Zosa Forde, Peter Milder, Edwin Simpson, Noam Slonim, Jesse Dodge, Emma Strubell, Niranjan Balasubramanian, Leon Derczynski, Iryna Gurevych, Roy Schwartz

arXiv:2209.00099v2cs.CL

TL;DR

Scaling NLP models improves capabilities but increases demands for data, time, storage, energy, and computation, creating a need for efficient methods. This survey synthesizes efficiency techniques across the NLP pipeline and identifies progress, trade-offs, and open challenges. It concludes that efficiency has advanced especially through model design, but cannot be measured by a single metric and remains dependent on hardware and software choices.

  • Problem

    Scaling NLP performance increases consumption of limited and unevenly distributed resources, motivating methods that achieve similar results with fewer resources.

  • Method

    The survey synthesizes efficient NLP methods across data, model design, pre-training, fine-tuning, inference, hardware, evaluation, and model selection.

  • Results

    The survey finds that most progress has come from model design targeted to specific computational budgets and hardware paradigms, while highlighting alternative efficient architectures such as S4 and Mega.

  • Takeaways & Limitations

    Efficient NLP requires analyzing trade-offs between end-task performance and resource consumption across software and hardware, rather than relying on a single efficiency metric.

  • Takeaways & Limitations

    Efficiency comparisons remain uncertain because hardware operations, utilization, and carbon intensity vary across settings, and long-sequence behavior is not yet well understood.

Abstract

from arXiv · show

Recent work in natural language processing (NLP) has yielded appealing results from scaling model parameters and training data; however, using only scale to improve performance means that resource consumption also grows. Such resources include data, time, storage, or energy, all of which are naturally limited and unevenly distributed. This motivates research into efficient methods that require fewer resources to achieve similar results. This survey synthesizes and relates current methods and findings in efficient NLP. We aim to provide both guidance for conducting NLP under limited resources, and point towards promising research directions for developing more efficient methods.

1 Introduction

Scaling improves NLP performance but increases resource demands, motivating efficient methods that achieve comparable results with fewer resources. This survey organizes efficiency across the NLP pipeline and targets both resource-constrained practitioners and efficiency researchers.

  • Motivation: Scaling has enabled state-of-the-art NLP performance, but its resource demands hinder accessibility, environmental sustainability, and hardware compatibility.The cited discussion identifies growing parameter counts and challenges involving limited resources, environmental impact, and hardware constraints.
  • Definition: Efficiency is defined as producing the same output with fewer resources, with costs related to execution, dataset size, and hyperparameter-tuning runs.The survey relates efficiency to model execution per example, dataset size, and the number of training runs required for tuning.
  • Scope: The survey adopts a pipeline-wide view of efficiency rather than restricting it to a single cost formulation.Its coverage spans data collection, model design, pre-training, fine-tuning, inference, evaluation, and model selection.
  • Scope: The survey is intended to guide NLP researchers working with limited resources and researchers developing more efficient methods.Each section also discusses limitations, open challenges, and possible future directions.
  • Motivation: Figure 1 depicts exponential growth in the number of parameters in pretrained language models.The figure is adapted from Lakim et al. (2022).

2 Data

Data efficiency seeks to reduce training data or use available instances more effectively, while preserving target performance and lowering training effort. The survey covers filtering, active learning, curriculum learning, and data-quality assessment, alongside challenges in ordering and auditing data.

  • Data Efficiency: Data efficiency reduces the number of training instances or improves how available instances are used.Fixed compute budgets motivate balancing model size and training-data size, especially during pre-training.
  • Filtering: Removing duplicates can reduce pre-training computation while achieving equal or better model performance than using all data.The passage describes duplicate removal as especially useful when pre-training data are abundant but compute is limited.
  • Filtering: Adversarial filtering identified a subset of approximately 2% of SNLI that achieved performance comparable to the full corpus.Filtering can also help mitigate biases, but existing datasets may lack enough training data for such approaches.
  • Active Learning: Active learning selects the most useful instances for annotation during data collection, using uncertainty, representativeness, or both.Its goal is to reduce the number of training instances by prioritizing examples before labeling.
  • Curriculum Learning: Curriculum learning orders data to reduce the training steps needed to reach a target performance without reducing dataset size.It has been applied to transformer pre-training and fine-tuning tasks including question answering and machine translation.
  • Curriculum Learning: Curriculum learning depends on choosing an appropriate pace, because poorly timed progression can waste computation on easy instances.Self-paced learning adapts ordering to the current model state, but it can be expensive and difficult to disentangle from optimizer and batch-size effects.

3 Model Design

Efficient model design reduces computation, memory, or training cost through architectural changes, sparse computation, retrieval, and parameter reduction. The survey highlights gains on long sequences and resource use, while noting unresolved trade-offs in stability, retrieval quality, and generalization.

  • Improving Attention in Transformers: Efficient attention mechanisms reduce the transformer's quadratic sequence-length cost through recurrence, fixed patterns, sparsity, or low-rank approximations.Local and global attention can scale linearly with input length, while learned sparsity patterns may provide more accurate but more expensive approximations.
  • Improving Attention in Transformers: S4 and Mega outperform attention-based methods on Long Range Arena while increasing training speed by approximately 5x and reducing memory cost by about 15% versus a standard transformer.Their convolutional structures emphasize nearby tokens and use parameter counts that grow sub-linearly with sequence length.
  • Sparse Modeling: Mixture-of-experts models route inputs through small subnetworks, reducing overall resource consumption while maintaining strong performance across several NLP tasks.GLaM used only ∼1/3 of GPT-3's energy consumption with additional hardware-based optimization, while another system achieved a 5x training-cost reduction.
  • Retrieval-Augmented Models: Retrieval-augmented models trade model size for database entries and can match much larger models by retrieving text during inference.RETRO matched models 25 times larger by retrieving chunks from a 2 trillion token database.
  • Model Design Considerations: Retrieval-based adaptation can reduce domain-specific fine-tuning, but retrieval quality, datastore growth, and parameter-versus-external-memory trade-offs remain important concerns.Database construction can improve translation over fine-tuning and reduce total translation time, although retrieval may slow as the datastore scales.
  • Model Design Considerations: Long-sequence modeling remains difficult because real-world documents can exceed training and benchmark lengths, and the effects of efficient architectures are not yet well understood.Sustainability reports average 243.5 pages, exceeding the 16k-token maximum in Path-X from Long Range Arena.

4 Pre-training

Efficient pre-training improves how models use data, objectives, and computational resources rather than relying only on larger models. The survey describes masking and denoising objectives, while emphasizing data quality, model-size/data balancing, and the high cost of pre-training.

  • Pre-training Setup: Modern transfer learning typically pre-trains models self-supervised on large text collections before fine-tuning them for specific tasks.Improving pre-training can reduce hyperparameter-tuning costs and increase data efficiency during fine-tuning.
  • Pre-training Objectives: Causal language modeling predicts the next token from context, whereas masked language modeling fills randomly masked tokens.GPT and PaLM use causal language modeling, while BERT uses masked language modeling.
  • Pre-training Objectives: Content-aware or heavier masking improves task performance and data use, while replaced token detection converges more quickly to better performance.ELECTRA and DeBERTa use a small generator to replace input tokens before detection.
  • Pre-training Objectives: Denoising sequence-to-sequence objectives let encoder-decoder models predict masked token spans and train on shorter sequences without losing task performance.T5 and BART use this approach, which helps reduce training costs.
  • Pre-training Considerations: Pre-training efficiency depends strongly on model design, model selection, and data use, while pre-training itself remains expensive in computation, memory, and high-quality data.Chinchilla showed that reducing model size to match available data can improve performance, reduce computational cost, and improve downstream applicability.

5 Fine-tuning

Fine-tuning methods adapt pre-trained models to downstream tasks while reducing parameter updates, storage, or training effort. The survey covers broad adaptation paradigms including adapters, prompt-based methods, multi-task learning, and combinations of parameter-efficient techniques.

  • Fine-tuning Foundations: Fine-tuning applies a pre-trained model to a downstream task, with this survey using the term broadly for any such adaptation method.The broader definition includes approaches that do not explicitly optimize fine-tuning efficiency.
  • Fine-tuning Foundations: Full-model fine-tuning creates a new parameter set for each task, making storage onerous across many tasks, while feature extraction updates fewer parameters but can reduce performance.Feature extraction trains a new classification layer while keeping the rest of the model fixed.
  • Parameter-Efficient Fine-Tuning: Adapters and related methods can update or add up to four orders of magnitude fewer parameters than full-model fine-tuning without sacrificing, and sometimes improving, performance.Adapters insert trainable dense layers while keeping the original model parameters fixed.
  • Parameter-Efficient Fine-Tuning: Adapter-based methods reduce storage but can increase inference time because of added parameters, motivating layer selection that activates only task-relevant adapters.A learned selector is proposed to use only the adapter layers necessary for a given task.
  • Parameter-Efficient Fine-Tuning: Parameter-efficient adaptation can modify activations with learned vectors or use sparse, low-rank, or low-dimensional updates.Prefix-tuning and prompt-tuning learn continuous prompts, while low-rank approaches reduce inference-time overhead but require selecting an effective rank.
  • Parameter-Efficient Fine-Tuning: AdaMix combines parameter-efficient fine-tuning techniques through routing and can outperform full fine-tuning.The approach combines different parameter-efficient techniques rather than relying on a single update form.
  • Alternative Adaptation Paradigms: Multi-task learning trains one model across tasks and can improve fine-tuning performance or sometimes support new tasks without fine-tuning.The latter setting is referred to as zero-shot transfer.
  • Prompting: Prompting casts a task as a textual instruction, avoiding fine-tuning but making suitable prompt design difficult and potentially costly to automate.Prompts may be manually crafted or generated with fill-in templates, while automated creation adds training cost.

6 Inference and Compression

Inference efficiency targets latency and memory, while compression methods reduce model cost across the NLP pipeline. The survey covers pruning, distillation, quantization, and context-specific inference optimization, emphasizing trade-offs and hardware dependence.

  • Inference efficiency improves either latency or memory requirements through acceleration and model compression.The appropriate objective depends on the deployment context.
  • Pruning: Pruning removes irrelevant weights or larger network components to reduce computation, memory capacity, and bandwidth requirements.Structured pruning can remove attention heads, attention values, or hidden layers, while unstructured pruning often better preserves performance.
  • Pruning: Early pruning can reduce training costs but risks over-pruning essential nodes and may require regrowth or additional hyperparameter tuning.Regrowing weights mitigates the risk but increases training costs.
  • Knowledge distillation: Knowledge distillation trains a smaller student model using supervision signals from a larger teacher and can outperform similarly sized models trained without that supervision.
  • Quantization: 4x–24x lower memory consumption and 4.5x higher throughput have been reported for low-precision formats versus 32-bit floating point.Mixed-precision methods customize precision across components and can preserve accuracy, but benefits depend on compatible hardware and kernels.
  • Inference optimization has no one-size-fits-all solution because requirements vary by use case, hardware, and execution framework.The survey highlights combinations of compression, task-specific mechanisms, parallelism, and hardware-aware architecture search.

7 Hardware Utilization

Hardware utilization methods address GPU memory bottlenecks, edge-device constraints, and opportunities for hardware–software–algorithm co-design. However, transformer irregularities can limit the benefits of dense specialized hardware.

  • Hardware-specific efficiency methods target GPU memory consumption, specialized hardware, co-design, and edge-device deployment.Techniques can be combined across training and inference stages.
  • Gradient offloading and dynamic block-wise quantization reduce memory pressure during training.bitsandbytes reduces memory consumption by 75% through block-wise tensor quantization and reports faster training.
  • Specialized ASIC and FPGA designs support efficient quantization and pruning operations, including ultra-low-bit and mixed-precision computation.These designs are not yet broadly available.
  • Joint hardware, software, and algorithm co-design can improve parallelization and produce speedups through hardware-aware mixture-of-experts and attention mechanisms.
  • Edge devices: Edge-oriented architectures use lightweight attention, group convolutions, depth-wise separable convolutions, or long-short range attention to meet mobile constraints.
  • Hardware considerations: Transformer sparsity and mixed data types restrict the use of dense computational units in specialized hardware.The survey suggests hardware-optimized formats and sparsity methods adapted to existing platforms.

8 Evaluating Efficiency

Evaluating efficiency requires specifying the resource objective and accounting for trade-offs among quality, computation, energy, carbon, and memory. The survey stresses that measurement methods and efficiency gains remain context-dependent and incomplete.

  • Efficiency evaluation should identify the computational aspect being minimized, especially FLOP/s or power consumption.
  • Pareto optimality: Pareto-optimal solutions achieve no worse task performance with lower resource consumption than competing systems.NLP studies compare dimensions such as runtime, memory, model size, BLEU, perplexity, and power.
  • FLOP/s: FLOP/s measurements vary across hardware and omit non-floating-point operations, while utilization is rarely 100%.Therefore, theoretical FLOP/s multiplied by elapsed time does not directly yield total computing performed.
  • Power consumption: Power measurement tools can miss energy because of sampling limits, unreported cooling and system-management costs, and incomplete architecture support.Direct TPU power figures are unavailable, and macOS power consumption is difficult to manage.
  • Carbon emissions: Carbon emissions depend on power consumption and the carbon intensity of marginal energy generation, so low energy does not necessarily mean low carbon.Carbon intensity varies by region and time, and efficiency gains can be offset by increased demand through Jevons’ paradox.
  • Efficiency must be characterized separately for pre-training, fine-tuning, and inference because memory requirements and optimizer costs differ across stages.Cost indicators can also disagree: mixture-of-experts improves quality–FLOP trade-offs, while unstructured sparsity may increase memory requirements.
  • Other desiderata: Compression can amplify existing biases, and more benchmarking is needed for efficiency’s effects on fairness, robustness, transfer, and out-of-distribution generalization.

9 Model Selection

Efficient model selection uses surrogate-based search, successive halving, transferred hyperparameters, and budget controls to reduce the cost of finding strong configurations. Limited budgets and reporting practices remain important constraints.

  • Model selection methods aim to efficiently identify a well-performing model variant.
  • Bayesian optimization and graph-based semi-supervised learning use surrogate models to search hyperparameters more efficiently than grid search or manual tuning.Successive halving provides a complementary budget-allocation strategy.
  • With limited computational budgets, Bayesian optimization and ASHA may fail to identify good hyperparameter settings.Their usefulness for selecting random initial weights or ordering training samples remains unclear.
  • Transferring hyperparameters across datasets or tasks can reduce trials during expensive stages such as pre-training.µTransfer tunes a small model and transfers its hyperparameters to a larger model.
  • Improved comparability may require fixing the hyperparameter-tuning budget or reporting the full search space.

10 Conclusion

The survey identifies broad progress in efficient NLP while emphasizing that efficiency has multiple dimensions and cannot be reduced to a single metric. It highlights unresolved trade-offs involving performance, resources, and interactions among algorithms, software, and hardware.

  • Efficiency in NLP has produced immediate successes, but substantial challenges remain.
  • Efficiency cannot be measured by a single metric because it has multiple definitions and can be achieved in many ways.
  • Key open challenges include modelling trade-offs between end-task performance and resource consumption.
  • Efficiency depends on interactions between hardware choices and software implementations.
  • Promising directions include better data use, lower pre-training and fine-tuning costs, and greater attention to algorithm–software–hardware interactions.
  • Reducing computation is important for making NLP advances accessible to a broader portion of society.
Loading 2209.00099v2…