PaperlayerView exact revision on arXiv

Source-linked AI summary

Advancing Transformer Architecture in Long-Context Large Language Models: A Comprehensive Survey

Yunpeng Huang, Jingwei Xu, Junyu Lai, Zixu Jiang, Taolue Chen, Zenan Li, Yuan Yao, Xiaoxing Ma, Lijuan Yang, Hao Chen, Shupeng Li, Penghao Zhao

arXiv:2311.12351v2cs.CLcs.LG

TL;DR

Long-context prompts matter in practice, but Transformer costs and limited memory constrain context handling. This survey organizes architectural advances across model stages and examines their evaluation and optimization landscape. It also identifies challenges and future research directions for long-context LLMs.

  • Problem

    Transformer time and space costs constrain training input lengths and effective inference context, while LLMs rely on the KV cache without an explicit memory mechanism.

    Source: §1 · p. 2, §2.2 · p. 7

  • Method

    The survey maps Transformer architectural advances for long-context capability across pretraining, fine-tuning, and inference.

    Source: §1 · p. 3, §10 · p. 26

  • Results

    The survey organizes architectural advances and reviews long-context evaluation and optimization resources, while identifying challenges and future directions.

    Source: §8 · p. 24, §9 · p. 25, §10 · p. 26

  • Takeaways & Limitations

    The survey provides a landscape for considering long-context methods alongside evaluation and optimization toolkits for training and inference.

    Source: §1 · p. 3, §8 · p. 24, §9 · p. 25

    LLMs show noticeable performance degradation on inputs exceeding Lmax, and their in-context memory relies solely on the KV cache.

    Source: §2.2 · p. 7

Abstract

from arXiv · show

Transformer-based Large Language Models (LLMs) have been applied in diverse areas such as knowledge bases, human interfaces, and dynamic agents, and marking a stride towards achieving Artificial General Intelligence (AGI). However, current LLMs are predominantly pretrained on short text snippets, which compromises their effectiveness in processing the long-context prompts that are frequently encountered in practical scenarios. This article offers a comprehensive survey of the recent advancement in Transformer-based LLM architectures aimed at enhancing the long-context capabilities of LLMs throughout the entire model lifecycle, from pre-training through to inference. We first delineate and analyze the problems of handling long-context input and output with the current Transformer-based models. We then provide a taxonomy and the landscape of upgrades on Transformer architecture to solve these problems. Afterwards, we provide an investigation on wildly used evaluation necessities tailored for long-context LLMs, including datasets, metrics, and baseline models, as well as optimization toolkits such as libraries, frameworks, and compilers to boost the efficacy of LLMs across different stages in runtime. Finally, we discuss the challenges and potential avenues for future research. A curated repository of relevant literature, continuously updated, is available at https://github.com/Strivin0311/long-llms-learning.

1 INTRODUCTION

Long-context use is increasingly important, but Transformer costs and positional-embedding weaknesses constrain effective context handling. The survey maps architectural approaches across model stages and reviews evaluation, optimization tools, and open challenges.

1 INTRODUCTION

  • Quadratic time and space costs constrain training input lengths and inference context windows, while positional embeddings can degrade or fluctuate on longer or shifted sequences.Source: §1 · p. 2
  • The survey groups long-context approaches around attention efficiency, memory mechanisms, extrapolative positional embeddings, context pre/postprocessing, and other methods.Source: §1 · p. 2
  • Its panorama covers architectural advances across development stages and considers Transformer design from algorithmic and system-architecture perspectives.Source: §1 · p. 3
  • The survey reviews long-context evaluation datasets, metrics, and baselines, alongside optimization libraries, frameworks, and compilers for training and inference.Source: §1 · p. 3
  • It identifies challenges in revamping Transformer structure and outlines corresponding future research directions.Source: §1 · p. 3
  • The authors provide a continuously updated repository of relevant literature for this rapidly developing area.Source: §1 · p. 3

2 OVERVIEW

This section introduces language-modeling objectives, modeling stages, and Transformer components as a foundation for examining long-context limitations in Transformer LLMs. It outlines challenges including quadratic attention costs, call-bound working memory, and maximum-length constraints, then previews a taxonomy of architectural approaches across five classes.

2 OVERVIEW

  • The overview previews analysis of long-context limitations and a taxonomy of architectural methods for enhancing LLM capabilities.The taxonomy groups upgrades by target, including attention, KV cache, positional embeddings, context processing, and the decoder block or loss module.Source: §2 · p. 3, Figure 1 · p. 4

2.1 Preliminaries

  • Language modeling approximates text log-probability, while its probability-distribution support grows exponentially with sequence length; CLM predicts each token from preceding tokens.MLM instead predicts masked tokens from the remaining bidirectional context.Source: §2.1 · pp. 3–4
  • LLMs typically progress through preprocessing, pretraining, fine-tuning, and inference, with causal models generating responses autoregressively one token at a time.Causal models use the CLM objective for pretraining and fine-tuning, though on different corpora.Source: §2.1 · p. 4
  • Decoder-only LLMs use stacked blocks with masked multi-head attention and feed-forward layers, followed by a language-modeling head that produces vocabulary probabilities.The layers use normalization and residual connections; GPT-series models are decoder-only CLM examples, and this architecture is described as predominant among current LLMs.Source: §2.1 · p. 5
  • RoPE rotates queries and keys to encode relative positional patterns while preserving their magnitudes, providing a more stable scheme for longer sequences.Unlike sinusoidal embeddings added to word embeddings, RoPE is applied to queries and keys in each attention layer.Source: §2.1 · p. 6
  • During autoregressive generation, the KV cache stores prior keys and values, and its memory occupation grows linearly with generated tokens.Each new token attends to the accumulated cached keys and values, which are updated with the new key-value pair.Source: §2.1 · pp. 6–7

2.2 Limitations

  • MHA incurs O(L^2 d) time and O(L^2) space, so both computational costs grow quadratically with sequence length.The space cost includes O(L^2) buffers for attention weights and probabilities.Source: §2.2 · p. 7
  • LLMs have only in-context working memory: they retain prior-token representations in a call’s KV cache but do not recall states across calls unless history is reloaded.This statelessness enables parallelism but challenges applications such as chatbots that need long-term memory.Source: §2.2 · p. 7
  • During training, max-length L_max, the upper bound on sequence length per batch sample, is commonly 1K, 2K, or 4K based on computational resources to avoid GPU OOM; inference providers restrict or truncate prompts to L_max.Source: §2.2 · p. 7

2.3 Taxonomy

  • The survey’s taxonomy includes efficient attention to reduce computation, long-term memory mechanisms, and extrapolative positional encodings to improve length generalization.Efficient attention can reduce costs, in some cases to linear time, while supporting longer pretraining and effective inference contexts.Source: §2.3 · pp. 7–8
  • The other taxonomy classes are context processing, which wraps LLMs with pre/postprocessing and multiple calls to work around context-window limits, and miscellaneous methods.Context processing keeps each call’s input within the maximum-length requirement.Source: §2.3 · p. 8

3 EFFICIENT ATTENTION

This section surveys efficient-attention strategies for addressing Transformer attention’s computational and memory demands in long-context settings. It covers local, hierarchical, sparse—including irregular sparsity—approximate, and IO-aware methods, along with inference-time KV-cache memory management.

3.1 Local Attention

  • Block-wise attention costs O(LBd) time and O(LB) memory but limits the global receptive field; Bi-BloSAN [198] adds inter-block attention for long-range dependencies.BlockBERT [176] restricts tokens to their fixed-size block.Source: §3.1 · p. 8
  • Sliding windows limit attention to w≪L preceding tokens, while StreamLLM [240] retains initial-token KV as attention sinks to largely recover sliding-window performance.Adding a placeholder token during pretraining further improves streaming deployment.Source: §3.1 · pp. 8–9
  • LSH attention selects up to K≪L neighboring keys by hashing similar queries and keys into the same bucket.Reformer [110] uses k-nearest-neighbor and locality-sensitive hashing for this selection.Source: §3.1 · p. 9

3.2 Hierarchical Attention

  • Hierarchical attention ranges from HAN’s word-to-sentence-to-document representation to BPT’s fine-to-coarse span attention, supporting document classification and multi-scale context.HAN’s two-level approach improves efficiency and performance in document classification.Source: §3.2 · p. 10
  • H-Transformer-1D reduces runtime and memory complexity to O(Ld), while Combiner offers sub-quadratic computation and memory with full-attention expressiveness.H-Transformer-1D typically uses log(L/2) hierarchy levels.Source: §3.2 · p. 10

3.3 Sparse Attention

  • Sparse Transformer [43] combines row-based local attention with column-based information propagation, while LogSparse lets each position attend to log L tokens.These patterns aim to reduce computation while retaining global context awareness.Source: §3.3 · p. 11
  • LongNet [62] uses dilated attention with mixed rates for long- and short-range dependencies, reducing computation to O(Ld) and scaling to sequences of up to one billion tokens.Its attentive field expands exponentially with token distance.Source: §3.3 · p. 11
  • Expire-Span learns which prior tokens to retain or expire, while Routing Transformer uses k-means clustering to connect queries with keys in the same cluster.Expire-Span assigns each previous token a learnable scalar in [0, 1], retaining tokens with important information.Source: §3.3 · p. 11
  • Star-Transformer [79] uses local ring and virtual-relay connections, whereas BigBird combines random, sliding-window, and global attention for O((w + r + 1)Ld) complexity.These methods frame sparse attention as graph sparsification; BigBird’s design reduces quadratic dependence to linear complexity.Source: §3.3 · p. 11

3.4 Approximated Attention

  • Linformer [229] uses learnable low-rank projections for O(Lkd) attention with error ϵ, while Luna uses nested pack and unpack attention with a constant-length side sequence.Luna propagates packed context without future-information leakage and supports varying sequence lengths because its projection matrices do not depend on L.Source: §3.4 · p. 12
  • Linear Transformer reduces time and space complexity to O(Ld^2) and O(Ld), respectively, while Performer [45] reduces them to O(Lrd) and O((d + r)L).Performer uses orthogonal random features for unbiased, low-variance estimation; kernelized methods require iterative computation across queries and do not fully exploit parallelism.Source: §3.4 · pp. 12–13
  • Scatterbrain [31] combines LSH-based sparse matrices with randomized-feature low-rank kernel decomposition, achieving linear time O(Lrd) with higher precision and flexible component choices.The described example combines Reformer [110] and Performer [45].Source: §3.4 · p. 13

3.5 IO-Aware Attention

  • Memory-Efficient Attention [177] uses constant working memory with respect to sequence length, but its time complexity remains quadratic.It computes numerically stable attention by sequentially processing single or chunked queries while tracking the normalization factor.Source: §3.5 · p. 13
  • Flash Attention requires O(L^2d^2M^-1) HBM accesses versus standard O(Ld+L^2), with M≫d^2 denoting SRAM size, and achieves up to 7.6× speedup on GPT2 and up to 20× greater memory efficiency.Source: §3.5 · p. 14
  • SCFA [168] extends Flash Attention beyond block-sparse patterns to irregular masks, including key/query dropping and Reformer-style hashing [110], yielding 2.0 to 3.3 times training speedup without sacrificing perplexity.Flash Attention can support block-sparse structures but may lack flexibility for irregular sparse strategies and arbitrary attention masks.Source: §3.5 · p. 14
  • Paged Attention [119] reduces KV-cache memory waste from fragmentation and redundancy during batched inference by enabling flexible sharing, drawing on virtual-memory paging.The KV cache grows dynamically with batched requests.Source: §3.5 · p. 14

4 LONG-TERM MEMORY

Long-term context is addressed through internal recurrence and external memory banks that preserve or retrieve information beyond a single input segment. The survey distinguishes cache designs and retrieval strategies, including their constraints.

4.1 Internal MemoryCache

  • Internal MemoryCache divides long text into segments and stores previous segments in a cache for recurrence-based access.The memory cache is usually detached from the computation graph.Source: §4.1 · p. 14
  • Transformer-XL caches outputs from m prior segments, extending dependency distance to O(Nml); Compressive Transformer also compresses older FIFO memories into a coarse queue.The compression rate and coarse-queue size determine how older segments are retained.Source: §4.1 · p. 15
  • ERNIE-Doc [64] reuses previous segments’ same-layer hidden states and retrospective feeding to expand effective context and pass higher-level representations to lower layers.Its first pass skims segments; a second pass enables bidirectional information flow.Source: §4.1 · p. 15
  • ∞-former [148] represents token embeddings as a continuous signal for unbounded context with fixed memory storage, but requires continuous attention to recover embeddings.The signal is formed from radial basis functions with coefficients fitted by multivariate ridge regression.Source: §4.1 · pp. 15–16
  • Alternate caches include RMT’s global memory tokens, Memorizing Transformer’s [238] top-layer key-value cache with kNN retrieval, and Memformer’s [237] attention-based memory.RMT is reported to scale effective context size to 1M tokens [26].Source: §4.1 · p. 16

5 EXTRAPOLATIVE PES

This section surveys attention-bias and RoPE-based approaches for extending positional generalization and inference length beyond the training context. It covers extended-RoPE designs, their mechanisms and trade-offs, and training strategies for improving positional robustness.

5.1 Enhancing Understanding

  • Trigonometric positional encodings hinder extrapolation because high-frequency oscillations are difficult to generalize and training contexts may not span low-frequency cycles.As a result, poorly learned low-frequency components make direct extrapolation prone to failure.Source: §5.1 · p. 18

5.2 Attention Bias

  • A local causal mask that keeps RoPE and restricts attention to keys within L_max is a simple, low-cost super-baseline that performs sufficiently well against more elaborate designs.Su’s experiments support this inference-time approach.Source: §5.2 · pp. 18–19

5.3 Extended RoPE

  • Positional Interpolation (PI) [35] uses linear scaling on each position and is experimentally more stable, requiring fewer finetuning steps than direct extrapolation; nonlinear scaling extends context for RoPE-pretrained LLMs like Llama without further finetuning.Source: §5.3 · p. 19
  • ReRoPE performs very well without finetuning on perplexity and QA tasks, even outperforming NTK-based schemes.Rectified truncation leaves positions unscaled within a local window and scales positions outside it; with κ set to infinity, ReRoPE can accommodate infinite contexts.Source: §5.3 · p. 20
  • ReRoPE and Leaky ReRoPE need two attention-matrix computations per stage and boolean merging, raising inference cost, limiting effective length, and lacking Flash Attention compatibility [54].A Triton-based reimplementation of the Flash Attention forward kernel incorporates ReRoPE and somewhat alleviates its computational burden.Source: §5.3 · p. 20
  • Giraffe [169] uses Basis Truncation with cutoff thresholds a and b, retaining high-frequency basis components.Source: §5.3 · p. 20
  • To address rear-position undertraining, SHAPE [111] random-shifts absolute positions; Random Padding [212] moves padding tokens to the front; Randomized PE [191] samples positions from a broader range.Source: §5.3 · p. 21
  • PoSE [268] finetunes models to adapt all relative positions in the target context window by adding a distinct skipping bias to training-sample position indices.This simulates longer inputs.Source: §5.3 · p. 21
  • Extrapolative positional embeddings aim to help LLMs infer long contexts in real-world scenarios when an L_max setting is available during training.The survey characterizes this as a promising, rapidly developing research area.Source: §5.3 · p. 21

6 CONTEXT PROCESSING

Context-processing approaches work around model context-window limits by selecting, aggregating, or compressing long inputs. These strategies can preserve or combine information, but some add computation or rely on assumptions about relationships among context windows.

6 CONTEXT PROCESSING

  • Black-box or gray-box context processing respects each call’s L_max limit without improving inherent long-context ability, and may increase computation or reduce answer precision.These methods make multiple model calls and rely on in-context learning.Source: §6 · p. 21

6.1 Context Selection

  • Context selection partitions long texts and ranks segments by criteria or scores, as in Map Rerank and CogLTX’s coarse-to-fine MemRecall.CogLTX adds top-scoring segments to a candidate queue over successive reasoning steps.Source: §6.1 · p. 21

6.2 Context Aggregation

  • Fusion-in-Decoder [93] encodes segments separately and lets the decoder attend to their contextualized representations to produce the output.SLED overlaps neighboring segments to create context paddings.Source: §6.2 · pp. 21–22
  • Map Reduce synthesizes parallel segment answers, Refine iteratively updates an answer, and PCW [83] [185] aggregates information across parallel context windows.PCW assigns each window a maximum length C := L − T and lets task-related tokens attend to all context tokens.Source: §6.2 · p. 22
  • NBCE [202] and PCW assume negligible relationships among context windows, and performance may suffer with tightly interconnected or excessive windows.These methods treat windows uniformly and as unordered.Source: §6.2 · p. 22

6.3 Context Compression

  • Context compression aims to fit long raw contexts under the L_max constraint by producing condensed, abstract representations through learned embedded alternatives (soft compression) or filtering redundancies using pretrained-model scores (hard compression).Source: §6.3 · p. 22

7 MISCELLANEOUS

The section surveys additional approaches to long-context modeling and systems efficiency, spanning task-specific objectives, expert routing, distributed parallelism, and weight or KV-cache compression.

Specific Objectives

  • Task-specific pretraining objectives target long-range dependencies and discourse structure in longer texts, with XLNet using permutation and ERNIE-Doc extending the approach to long documents.The section also notes DANCE’s Segment-Reordering Objective, divide-and-conquer summarization, PEGASUS’s Gap Sentence Generation, and PRIMERA’s Entity Pyramid method for multi-document summarization.Source: §7 · p. 23

Mixture of Experts

  • Mixture-of-Experts replaces dense FFN layers with specialized experts selected by a gate, with top-k routing or Switch Transformer’s k=1 routing.Selected experts’ contributions are combined by weighted summation; the section reports improved versatility and reduced computational demands for large-scale contexts.Source: §7 · p. 23

Parallelism

  • The section summarizes data, tensor, pipeline, and sequence parallelism as commonly used strategies for leveraging aggregated GPU memory to scale model sizes and extend sequence length.Source: §7 · pp. 23–24

Weight Compression

  • Weight compression methods include pruning, factorization, quantization, partitioning, and distillation, while lower parameter precision reduces memory demands and accelerates training and inference.MQA and GQA reduce the number of KV heads and distribute them across multiple queries to mitigate KV-cache size during inference.Source: §7 · p. 24

8 EVALUATION NECESSITY & OPTIMIZATION TOOLKIT

The section assembles evaluation resources for long-context LLMs—datasets, metrics, and baselines—and catalogs toolkits for improving efficiency and effectiveness across development stages.

Datasets

  • The survey collects long-context evaluation datasets, including LRA, SCROLLS, LEval [7], LongBench [15], and InfiniteBench.Appendix A records dataset language, task types, length statistics, quality, and splits.Source: §8 · p. 24

Metrics

  • The survey summarizes nine categories of general metrics across ten NLP task types, with task-specific metrics listed in Appendix B, Table 2.Covered task types include language modeling, question answering, summarization, math solving, code generation, and open-ended writing.Source: §8 · p. 24

Baselines

  • The survey gathers pretrained and fine-tuned LLM baselines for downstream long-context evaluation, including Claude2 and GPT4 as closed-source examples and LLongMA [170] and LongChat as open-source examples.Appendix C, Table 3 includes basic model information, maximum context length, parameter size, memory occupancy, and relevant links.Source: §8 · p. 24

Toolkit

  • The toolkit inventory includes libraries such as vLLM [119], compilers such as Triton, and frameworks such as DeepSpeed and Megatron-LM [200].The survey presents these tools as supporting LLM efficiency and effectiveness across development stages.Source: §8 · pp. 24–25

9 DISCUSSION

The discussion identifies persistent challenges in memory, length extrapolation, training objectives, evaluation metrics, and efficient attention, and outlines corresponding research directions.

Attention Trade-off

  • Long-context attention must balance full-scale dependencies and attention-score precision against computational demands while preserving global dependencies and relevancy.The discussion points to IO-aware methods such as Flash Attention [54] as improving runtime and memory efficiency without sacrificing attention precision.Source: §9 · p. 25

Memory Efficacy and Efficiency

  • Long-term memory mechanisms can incur overhead from intricate heuristics and potentially degrade performance over time, motivating more efficient memory organization and read/write throughput.Paged Attention [119] is cited as a relevant recent advance.Source: §9 · p. 25

Length Extrapolation Mining

  • Length-extrapolation advances, including strategies applied to RoPE [32] [170], often rely on simplified observations of positional embeddings and heuristic adjustments.The discussion calls for stronger theoretical foundations for modeling sequentiality and further investigation of learnable embeddings with many hyperparameters, citing CLEX [32] as an example.Source: §9 · p. 25

Specific yet Universal Objective

  • Many long-text pretraining objectives are task-specific or compatible only with MLM rather than the more common CLM objective.The discussion calls for broadly applicable causal language-modeling objectives that capture long-range dependencies from early training, aligned with an effective positional-embedding scheme.Source: §9 · p. 25

Reliable Metric Demand

  • Common evaluation metrics can differ substantially from human judgment [117], while using GPT4 as a substitute judge faces high costs that challenge wider adoption.The discussion identifies more dependable metrics as a need, especially for generative tasks without precise ground truth.Source: §9 · p. 25

10 CONCLUSION

The conclusion presents a lifecycle-spanning survey of Transformer architectural advances for long-context LLMs, alongside evaluation and optimization resources and future research challenges.

10 CONCLUSION

  • The survey organizes Transformer architectural methods for extensive context windows across development stages in a taxonomy targeting different Transformer module designs.The taxonomy is presented in Figure 1(b).Source: §10 · p. 26
  • The survey covers evaluation necessities for long-text tasks and optimization toolkits intended to improve LLM efficiency and efficacy.It also identifies key challenges and corresponding future directions.Source: §10 · p. 26
  • The authors maintain a repository intended to keep readers updated with research in the field.The conclusion expresses the hope that the survey serves as a resource for researchers developing long-context LLMs.Source: §10 · p. 26

A DATASETS

Table 1 catalogs long-text datasets across NLP tasks and reports information about their contents, lengths, quality, splits, counts, and formats. The length convention counts context and question words while excluding fixed prompt-template text, and common dirty data can produce extremely short samples.

A DATASETS

  • Table 1 catalogs datasets for varied NLP tasks with long-text inputs and summarizes basic dataset information.The table’s metadata includes language, task types, sample lengths, quality, splits, counts, and file formats.Source: Table 1 · p. 36, A DATASETS · p. 36
  • Common dirty data can leave some datasets with minimum sample lengths approaching zero.The note attributes these extremely short samples to common dirty data.Source: A DATASETS · p. 36
  • Sample lengths are reported as average, minimum, and maximum kilo-words, counting context and question text but excluding fixed prompt-template text.In English and code, words are approximately space-separated; individual Chinese characters count as words.Source: A DATASETS · p. 36
  • Dataset quality is classified as human-labeled or model-assisted, while split descriptions and sample counts summarize dataset partitioning and scale.Model-assisted labels may be generated by off-the-shelf LLMs.Source: A DATASETS · p. 37
  • The table also records dataset file formats, including JSONL, JSON, CSV, TXT, TSV, and Parquet.Source: A DATASETS · p. 37

B METRICS

Table 2 maps common evaluation metrics to NLP task types, while the text introduces general metric categories and task-specific measures. The survey describes measures for language modeling, objective-answer tasks, text similarity, translation, entity coverage, code or math, and open-ended writing.

B METRICS

  • Perplexity is formalized as exp(loss), where loss is the test-set cross-entropy, and BPC/BPW measure average bits required to encode characters or words.BPC/BPW are calculated as average loss divided by the number of characters or words, respectively.Source: B METRICS · p. 37
  • Accuracy and F1 assess objective-answer tasks, while exact match evaluates exact sequence matches and is useful for tasks such as code completion.F1 balances precision and recall; ROUGE-1/-2/-L measures n-gram overlap and is used for tasks such as summarization.Source: B METRICS · p. 37
  • BLEU, METEOR, and TER are machine-translation metrics covering n-gram overlap, linguistic factors, and edit distance, respectively.Source: B METRICS · p. 37
  • Entity Mention evaluates important-entity coverage and correctness, Pass@k checks whether an answer ranks among the top k, and human or model judges assess subjective text quality.The cited examples include summarization, code generation or some math tasks, and story generation, respectively.Source: B METRICS · p. 38

C BASELINES

Table 3 summarizes basic information about long-context models used as baselines. Its descriptions cover context limits, model statistics, language, access links, openness, base structure, and primary usage.

C BASELINES

  • Table 3 presents basic information about long-context models widely used as baselines.The descriptions include maximum context length, language, resource statistics, and access links.Source: Table 3 · p. 38, C BASELINES · p. 39
  • The table distinguishes open-source models from closed-source models accessible only through official remote APIs.Source: C BASELINES · p. 38
  • Baseline descriptions identify each model’s base structure and primary usage, including instruction-following, code/math, generation, and chat.Source: C BASELINES · pp. 38–39
  • Model statistics include parameter count, memory footprint, and disk storage, with models loaded in float16 on Nvidia GPUs without quantization.Source: C BASELINES · p. 39

D TOOLKITS

Table 4 organizes toolkits for improving LLM efficiency and effectiveness across lifecycle stages. The survey describes toolkit types and stage-specific utilities spanning pretraining, finetuning, inference, and application.

D TOOLKITS

  • The survey distinguishes libraries, frameworks, systems, and compilers as toolkit types.Libraries implement specific tasks; frameworks combine interconnected modules, systems provide deployment environments, and compilers optimize GPU execution.Source: D TOOLKITS · p. 39
  • Toolkit utilities are summarized with keywords indicating core techniques relevant to their corresponding lifecycle stages.Source: D TOOLKITS · p. 39
  • Table 4 summarizes toolkits for enhancing LLM efficiency and effectiveness across different stages.The survey groups lifecycle activity into pretraining, finetuning, inference, and application.Source: D TOOLKITS · p. 39, Table 4 · p. 40

Related papers

Loading 2311.12351v2…