PaperlayerView exact revision on arXiv

Source-linked AI summary

Efficient Long-Text Understanding with Short-Text Models

Maor Ivgi, Uri Shaham, Jonathan Berant

arXiv:2208.00748v3cs.CLcs.AIcs.LG

TL;DR

Long sequences are costly for pretrained transformers because their input-length complexity is quadratic, while many efficient alternatives require specialized implementations and expensive pretraining. SLED reuses pretrained encoder-decoder models by encoding overlapping chunks and fusing their information in the decoder, and it performs competitively on SCROLLS against larger models with dedicated long-range pretraining.

  • Problem

    Pretrained transformers’ quadratic input-length complexity makes them prohibitive for long texts, while many efficient variants require custom implementations and expensive pretraining.

    Source: §1 · p. 1

  • Method

    SLED encodes overlapping chunks of a long input with a pretrained encoder and uses the pretrained decoder to fuse their representations.

    Source: §1 · pp. 1–2

  • Results

    SLED achieves competitive SCROLLS performance against models with dedicated long-range pretraining, including larger models.

    Source: §1 · p. 2, §8 · p. 11

  • Takeaways & Limitations

    Future pretrained encoder-decoder models can be plugged into SLED to pursue further performance gains on SCROLLS and other long-range tasks.

    Source: §8 · p. 11

    SLED assumes constant output length; tasks requiring long generated outputs may not fit this setting because decoder self-attention is quadratic in output length.

    Source: §7 · p. 10

Abstract

from arXiv · show

Transformer-based pretrained language models (LMs) are ubiquitous across natural language understanding, but cannot be applied to long sequences such as stories, scientific articles and long documents, due to their quadratic complexity. While a myriad of efficient transformer variants have been proposed, they are typically based on custom implementations that require expensive pretraining from scratch. In this work, we propose SLED: SLiding-Encoder and Decoder, a simple approach for processing long sequences that re-uses and leverages battle-tested short-text pretrained LMs. Specifically, we partition the input into overlapping chunks, encode each with a short-text LM encoder and use the pretrained decoder to fuse information across chunks (fusion-in-decoder). We illustrate through controlled experiments that SLED offers a viable strategy for long text understanding and evaluate our approach on SCROLLS, a benchmark with seven datasets across a wide range of language understanding tasks. We find that SLED is competitive with specialized models that are up to 50x larger and require a dedicated and expensive pretraining step.

1 Introduction

Long-input processing is costly for standard pretrained transformers, while many efficient alternatives require specialized implementations and expensive pretraining. SLED reuses pretrained encoder-decoder models and achieves competitive SCROLLS results, while also serving to analyze which tasks need remote context.

1 Introduction

  • Standard transformers’ quadratic input-length complexity makes long texts prohibitive, while many efficient variants require custom implementations and expensive pretraining.Source: §1 · p. 1
  • BARTlarge-SLED is competitive with LongT5base and surpasses UL2 despite UL2’s 50x more parameters; SLED improves pretrained models by 6 points on average.Source: Figure 1 · p. 1, §1 · p. 2
  • SLED applies off-the-shelf pretrained encoder-decoder models to long texts by encoding overlapping chunks and using the decoder to fuse their information.Source: §1 · pp. 1–2
  • SLED analysis finds that relatively local contextualization suffices for high performance on SCROLLS question-answering and natural-language-inference tasks.Source: §1 · p. 2
  • The authors provide an open-source SLED implementation integrated into the Transformers library.Source: §1 · p. 2

2 Background

Self-attention gives each token access to every other input token, creating quadratic time and space costs that limit long-sequence processing. SLED belongs to local-attention approaches but reuses short-range pretrained encoder-decoder models rather than requiring specialized pretraining or dedicated CUDA implementations.

2 Background

  • Self-attention’s all-to-all token interactions create quadratic time and space dependency on input length, limiting transformer processing of long sequences.Source: §2 · p. 2
  • Sparse-attention methods reduce long-sequence costs by limiting each token’s attention to local or semantically similar tokens, sometimes with global tokens.Source: §2 · pp. 2–3
  • SLED is a local-attention variant that reuses short-range pretrained encoder-decoder models without specialized pretraining or dedicated CUDA implementations.Source: §2 · p. 3
  • Unlike local attention whose receptive field grows across layers, SLED gives tokens the same-sized context regardless of layer depth, enabling better parallelization.Source: §2 · p. 3

3 Method

SLED assumes that tokens can be contextualized from local surroundings and that the decoder can handle reasoning across chunks. It splits a document into overlapping chunks, encodes each independently, retains effective-chunk representations, and lets the pretrained decoder generate an output from them.

3 Method

  • SLED assumes that chunks can be encoded independently and that the decoder can fuse information across them.Source: §3 · p. 3
  • The input is a document of length n and may include a short prefix, such as a question, instruction, or hypothesis.Source: §3 · p. 3
  • The document is split into overlapping chunks; each chunk’s middle (1 − ρ) × c tokens receive left and right context padding of ρ × c / 2 tokens.Source: §3 · p. 3
  • Each chunk is encoded independently, and the effective-chunk representations are concatenated to form contextualized input representations.Source: §3 · pp. 3–4
  • The pretrained decoder generates the output using standard cross-attention over the encoded input and any encoded prefix.Source: §3 · p. 4
  • With constant chunk size, encoding has linear memory and compute dependency on input length despite quadratic self-attention within each chunk; decoding remains linear when output length is constant.Source: §3 · p. 4

4 Efficacy of Fusion in Decoder

The controlled experiments test whether fusion-in-decoder can retrieve relevant information from long inputs and fuse information across independently encoded chunks. They also examine how SLED’s encoder and prefix design affect these operations.

4.1 Needle in a haystack; 4.2 Piecing a puzzle

  • SLED reaches F1 87.6 versus oracle BARTbase F1 88.1 on SQuAD, and HotpotQA F1 76.5 versus oracle F1 78.6, while outperforming second-paragraph BART.On SQuAD, SLED matches oracle performance in ordered and shuffled settings; on HotpotQA, it effectively fuses information from two chunks.Source: Figure 3 · p. 5, §4.1 · p. 5, Figure 4 · p. 5

4.1 Needle in a haystack

  • On shuffled SQuAD inputs, LED performs below SLED, while BART given only the first 1K tokens and LED without global tokens perform poorly.The setup adds nine distractor paragraphs and either keeps the answer paragraph first or shuffles paragraph order.Source: §4.1 · pp. 4–5, Figure 3 · p. 5

4.2 Piecing a puzzle

  • On HotpotQA, LED slightly outperforms SLED, but LED without global tokens drops sharply, showing that its receptive field alone does not suffice for information fusion.The experiment places the two gold paragraphs in separate chunks and compares SLED with a model given the answer-containing second paragraph only.Source: §4.2 · p. 5, Figure 4 · p. 5

4.3 Ablations of design choices

  • Removing local encoder context by setting chunk size to one produces poor performance, indicating that decoder fusion does not replace local contextualization.Source: §4.3 · p. 6
  • Treating the prefix as a separate chunk rather than prepending it to document chunks causes a significant performance drop across settings.Source: §4.3 · p. 6
  • SLED’s performance is practically unchanged between ordered and shuffled inputs, while LED without global tokens drops when paragraphs are shuffled; decoder access to prefix representations makes little difference.The findings suggest that the encoder can incorporate important prefix information into encoded document tokens.Source: §4.3 · p. 6

5 Experiments

The experiments evaluate SLED across seven SCROLLS datasets and analyze its results, context needs, and padding choices. SLED improves average benchmark scores, while the analyses show how chunk size, input length, and overlap relate to performance.

5 Experiments

  • Across seven SCROLLS datasets, evaluation covers summarization, question answering, and natural language inference, using each task’s official metrics.The settings compare BART and T5 backbones with SLED against backbones limited to the first 1,024 tokens, and also include LEDbase.Source: §5 · p. 6, §5.1 · p. 6

5.2 Results

  • BARTbase-SLED improves the average SCROLLS score over BARTbase by 4.8–7 points; BARTlarge-SLED is comparable to LongT5base and outperforms UL2, despite UL2 having 50x more parameters.The results cover SCROLLS development and test sets; BARTlarge-SLED remains moderately below larger LongT5 models.Source: §5.2 · p. 7, Table 1 · p. 8
  • SLED improves performance over the corresponding backbone on all tasks except QuALITY; Qasper and NarrativeQA show the largest gains, averaging 60%.Summarization datasets show gains of up to 35% across metrics and all three backbone models.Source: §5.2 · p. 7

5.3 Datasets analysis

  • With chunk size c varied at 16K input length, the best-performing size is at most 256 across datasets; larger chunks can hurt, while summarization gains more as c increases to that threshold.The authors relate the local-context pattern in QA and NLI to possible noise and optimization effects from larger chunks.Source: §5.3 · p. 7, Figure 5 · p. 9
  • As maximum input length increases, performance improves roughly monotonically on every dataset except QuALITY, with gains reported up to 16K tokens.The analysis varies the first n document tokens while fixing c = 256.Source: §5.3 · pp. 7–8

5.4 Effect of context padding

  • With padding ρ = 0.05, training can be up to 2x faster than at ρ = 0.5, but no padding value consistently beats 0.5; non-overlapping chunks lose at least 10% of gain versus the best setting.The ρ = 0.5 setting is top-two on all six evaluated datasets, and non-overlap losses exceed 50% in some cases.Source: §5.4 · p. 8

6 Related Work

Prior work explores efficient attention, sparse local windows, and fusion-in-decoder for long inputs. SLED builds on sliding-window and FiD ideas while applying them to long documents with off-the-shelf encoder-decoder models.

6 Related Work

  • Efficient-transformer research includes clustered, fixed-count, random-feature, and low-rank attention, but many variants were not established as pretrained-LM backbones and some did not beat sliding-window local attention downstream.Sparse approaches combine local attention with global tokens, with BigBird also sampling random tokens.Source: §6 · p. 9
  • FiD encodes independent passages separately and concatenates their encodings before decoding; this work tests that approach on a single coherent long document rather than independent passages.The authors describe their work as the first controlled analysis of FiD feasibility and limitations in this setting.Source: §6 · p. 10

Pretrained models with sliding windows

  • Earlier sliding-window work includes BERT applications to summarization and QA; SEGENC applied overlapping chunks and FiD to QMSum, while SLED is presented as plug-in across encoder-decoder models without added parameters or task-specific training.The authors distinguish their work from SEGENC by its broader, systematic analysis beyond summarization.Source: Pretrained models with sliding windows · p. 10

7 Limitations

SLED has two limitations: its linear-complexity setup assumes constant-length outputs, and its local-context assumption may fail when understanding depends on distant information. These constraints may limit applicability to some long-range tasks.

Long output

  • Constant output length is required for SLED’s linear complexity, constraining tasks that need long generated outputs.The decoder adds quadratic self-attention over the output and O(nk) input-output cross-attention; academic reports and script writing are examples of possible future long-output tasks. The same limitation affects LongT5, LED, and long-input pretraining with span-corruption or denoising objectives.Source: §7 · p. 10

Co-reference resolution and fact retention

  • When distant entity references, facts, or multi-hop context are needed, local encoding cannot access the earlier information, shifting more responsibility to the decoder and reducing contextual-encoding effectiveness.For example, a chapter’s reference to a room or person may depend on information from chapters earlier.Source: §7 · p. 10

8 Conclusions

SLED slides a pretrained short-range encoder over long documents and generates outputs by attending to the encoded tokens. It demonstrates core long-text operations and competitive SCROLLS performance, while remaining compatible with short-range pretrained LMs.

8 Conclusions

  • SLED extends a pretrained short-range encoder across a long document, then generates output by attending to its encoded tokens.The conclusion describes this as a simple approach for modeling long texts.Source: §8 · pp. 10–11
  • SLED demonstrates competitive SCROLLS performance against larger models and models requiring dedicated, expensive pretraining.The reported core operations include finding relevant information and fusing it at decoding time.Source: §8 · p. 11
  • SLED can be used with any short-range pretrained LM, allowing future encoder-decoder models to be plugged in for long-range tasks.The authors open-source SLED and express hope that it will encourage extending NLU models to longer inputs and real-world use cases.Source: §8 · p. 11

A SLED implementation details

SLED handles edge tokens by defining effective chunk tokens for the first and last chunks, which do not have the same bidirectional context as interior chunks.

A SLED implementation details

  • The first and last ρ×c/2 input tokens lack bidirectional context and require special handling.The implementation discussion notes that this detail is omitted from the main method description for brevity.Source: A SLED implementation details · p. 11
  • The first chunk’s effective tokens are the first (2−ρ)×c/2 tokens, preserving as much commonality between chunks as possible..Source: A SLED implementation details · p. 11
  • The final chunk starts at token t_(n−c+1), contains exactly c tokens, and treats as effective tokens those not assigned to an earlier chunk..Source: A SLED implementation details · p. 11

B Chunking vs. local-attention

SLED uses overlapping chunks, while LED applies local attention per layer; the approaches differ in contextualization, implementation, positional information, and memory use. Their relative results vary across datasets and model configurations.

Comparison setup

Contextualization and implementation

  • LED’s per-layer local attention yields a receptive field that grows with encoder depth, whereas SLED independently encodes overlapping chunks.SLED’s chunking is simpler to wrap around an existing encoder-decoder model; LED’s specialized CUDA kernel is more efficient but adds engineering overhead when extending it to new architectures.Source: B Chunking vs. local-attention · p. 11

Contextualization

  • LEDbase, using all prefix tokens for global contextualization, outperforms LEDSCROLLSbase, which uses one global token.The comparison is presented as evidence relevant to the benefit of broader contextualization.Source: B Chunking vs. local-attention · p. 11

Positional information

  • SLED’s independent chunking reuses the backbone’s positional encoding and may generalize to arbitrary input lengths; LED duplicates BART’s absolute embeddings 16 times for 16K inputs.The discussion says this may limit LED’s generalization to longer inputs and potentially require substantial long-input samples to tune the new parameters.Source: B Chunking vs. local-attention · p. 11

Results

  • In NarrativeQA and GovReport, LED is comparable to BARTbase-SLED and slightly outperforms it on some metrics; it does slightly worse on ContractNLI and significantly worse on other small-training datasets.The cited comparison notes approximately 71K NarrativeQA, 19K GovReport, and 10K ContractNLI training examples.Source: B Chunking vs. local-attention · p. 12

Complexity

  • Both encoders have O(l × c × n) memory complexity, but SLED may require up to 2x more memory than LED when ρ = 0.5.SLED’s overlap and full self-attention within each chunk account for the possible extra memory.Source: B Chunking vs. local-attention · p. 12

C Experimental details

Experiments followed the SCROLLS repository’s datasets, splits, and suggested epoch counts, with model selection over learning rates and effective batch sizes. The authors also state that reproduction materials will be made public upon publication.

C Experimental details

  • Experiments used SCROLLS-recommended datasets, splits, and epoch counts.The setup was based on the SCROLLS official repository.Source: C Experimental details · p. 12
  • Model selection finetuned nine models per model–dataset pair across three learning rates and three effective batch sizes.The learning rates were 2e−5, 5e−5, and 1e−4; batch sizes were 8, 16, and 32.Source: C Experimental details · p. 12
  • The authors said code, data, environment requirements, hyperparameters, and scripts would be made public upon publication.Source: C Experimental details · p. 12
Loading 2208.00748v3…