PaperlayerView exact revision on arXiv

Source-linked AI summary

Poolingformer: Long Document Modeling with Pooling Attention

Hang Zhang, Yeyun Gong, Yelong Shen, Weisheng Li, Jiancheng Lv, Nan Duan, Weizhu Chen

arXiv:2105.04371v2cs.CL

TL;DR

Transformer self-attention has quadratic cost, and input-length limits constrain its use on long documents. Poolingformer combines sliding-window attention with a larger-window pooling level to expand receptive fields at linear complexity. It reports state-of-the-art results on long-document QA and superior performance on long-document summarization.

  • Problem

    Transformer self-attention has quadratic computational and memory complexity, leading many pretrained models to cap input length at 512 tokens, which can worsen long-sequence task performance.

    Source: §1 · p. 1

  • Method

    Poolingformer uses neighboring-token sliding-window attention followed by larger-window attention over pooled key and value vectors.

    Source: §1 · p. 1

  • Results

    Poolingformer achieves new state-of-the-art performance on long-document QA and superior performance on long-document summarization.

    Source: §1 · p. 2, §5 · p. 8

  • Takeaways & Limitations

    Poolingformer has linear computational and memory complexity and does not require explicitly splitting long documents into paragraphs.

    Source: §1 · p. 2

    Replacing all layers with Poolingformer layers can degrade performance, and the best results often use Poolingformer in one fourth of the layers.

    Source: §3.4 · p. 8

Abstract

from arXiv · show

In this paper, we introduce a two-level attention schema, Poolingformer, for long document modeling. Its first level uses a smaller sliding window pattern to aggregate information from neighbors. Its second level employs a larger window to increase receptive fields with pooling attention to reduce both computational cost and memory consumption. We first evaluate Poolingformer on two long sequence QA tasks: the monolingual NQ and the multilingual TyDi QA. Experimental results show that Poolingformer sits atop three official leaderboards measured by F1, outperforming previous state-of-the-art models by 1.9 points (79.8 vs. 77.9) on NQ long answer, 1.9 points (79.5 vs. 77.6) on TyDi QA passage answer, and 1.6 points (67.6 vs. 66.0) on TyDi QA minimal answer. We further evaluate Poolingformer on a long sequence summarization task. Experimental results on the arXiv benchmark continue to demonstrate its superior performance.

1. Introduction

Transformer self-attention has quadratic cost, so models often limit input length to 512 tokens, constraining long-document tasks. Poolingformer combines local sliding-window attention with a second, pooled attention level to expand receptive fields at linear complexity, and reports state-of-the-art results on long-document QA and summarization.

1. Introduction

  • Transformer self-attention has quadratic computational and memory complexity, and models commonly cap input length at 512 tokens because of resource constraints.These limits can worsen performance on long-sequence tasks.Source: §1 · p. 1
  • Prior long-sequence approaches include combining local and global attention patterns or splitting documents into paragraphs for hierarchical attention.The cited examples are Longformer and Hierarchical Transformer.Source: §1 · p. 1
  • The second attention level uses a larger window and pools key and value vectors to expand receptive fields while reducing computational cost and memory consumption.The first level retains sliding-window attention over neighboring tokens.Source: §1 · p. 1
  • Poolingformer has linear computational and memory complexity, avoids explicitly splitting documents into paragraphs, and preserves first-level sliding-window attention to mitigate information loss from pooling.This design gives each token a larger receptive field than single-level local attention.Source: §1 · p. 2
  • Poolingformer reports new state-of-the-art results on the NQ and TyDi QA leaderboards and on the arXiv summarization benchmark.The paper evaluates monolingual NQ, multilingual TyDi QA, and long-document summarization.Source: §1 · p. 2

2. Model

Poolingformer replaces full self-attention with two levels: local sliding-window attention followed by larger-window pooling attention. The section describes how the second level, task-specific global tokens, and the attention pattern’s complexity fit into this design.

2.1. Transformer Self-Attention

  • Full Transformer self-attention aggregates value vectors using query-key attention, but its memory and computational complexity is O(n^2).This quadratic scaling limits processing extremely long text sequences.Source: §2.1 · p. 2

2.2. Poolingformer Self-Attention

  • Poolingformer combines first-level sliding-window attention with second-level pooling attention over a larger window.Figure 2 depicts the sliding-window block on the left and pooling-attention block on the right.Source: §2.2 · p. 2, Figure 2 · p. 3

2.2.1. FIRST-LEVEL: SLIDING WINDOW ATTENTION

  • Each first-level token attends only to neighbors within its window, whose limited receptive field could hurt long-document understanding.The neighbor set is denoted N(i, w1).Source: §2.2.1 · p. 2

2.2.2. SECOND-LEVEL: POOLING ATTENTION

  • The second-level module builds on first-level outputs Y and produces new query, key, and value matrices.Its larger window w2 can extend to the full sequence, while pooling compresses the key and value matrices to sizes ξ times smaller.Source: §2.2.2 · pp. 2–3
  • A residual connection makes the final two-level output the sum of first- and second-level outputs; pooling choices include mean, max, convolution, LDConv, and mean-LDConv.The paper describes LDConv and mean-LDConv as trainable pooling mechanisms.Source: §2.2.2 · p. 3

2.2.3. TASK SPECIFIC GLOBAL ATTENTION

  • For question answering, query tokens join a global set; each non-global token attends to that set and its local window, while global tokens attend to the full sequence.Source: §2.2.3 · p. 3

2.2.4. COMPLEXITY ANALYSIS

  • With w1 and w2/ξ configured as constants, both attention levels have O(n) computational complexity, giving Poolingformer overall O(n) complexity.The first level is O(w1n), and the second level is O(nw2/ξ).Source: §2.2.4 · p. 4

3. Experiments

Experiments evaluate Poolingformer on monolingual and multilingual question answering, long-document summarization, and ablations of its attention design. It reports leading QA and summarization results, while the ablations examine window sizes, pooling and fusion choices, and layer count.

3.1. Datasets

  • Experiments cover NQ and multilingual TyDi QA plus arXiv scientific-paper summarization, whose documents average about 5K tokens and reach 16.5K at the 90th percentile.NQ evaluates short and long answers; TyDi QA evaluates passage selection and minimal-answer spans.Source: §3.1 · p. 4

3.3.1. GOOGLE NQ RESULTS; 3.3.2. TYDI QA RESULTS

  • NQ long-answer F1 was 79.8 vs. 78.0; TyDi QA passage-answer F1 was 79.5 vs. 77.7 and minimal-answer F1 was 67.7 vs. 66.1, setting new state-of-the-art results.The comparisons are reported for the blind NQ test set and TyDi QA leaderboard.Source: Table 2 · p. 5, §3.3.1 · pp. 5–6, Table 3 · p. 6

3.3.3. SUMMARIZATION RESULTS

  • Poolingformer at 16K input length greatly outperformed prior methods on arXiv; at 4K, it led on ROUGE-1 and ROUGE-L.With the same receptive field, its attention complexity is half that of LED; the authors describe gains in both accuracy and complexity.Source: Table 4 · p. 6, §3.3.3 · pp. 6–7

3.4. Ablation Study

  • NQ dev ablations found a 256-token first-level window best among single-window settings; whole-sequence pooling performed poorly, motivating coarse compression of distant tokens.The authors conjecture that remote-token redundancy makes attention less useful, and describe discarding very distant tokens directly.Source: Table 5 · p. 7, §3.4 · p. 7
  • LDConv and MAX were slightly better pooling variants; stacked levels and separate mappings beat Mix and shared weights, while the best results often used about one-quarter Poolingformer layers.Replacing all layers degraded performance, which the authors associate with incomplete compatibility with pretrained models and possible catastrophic forgetting.Source: §3.4 · pp. 7–8, Table 6 · p. 7, Table 7 · p. 8

4. Related Work

Related work addresses long-sequence attention through approximations and sparse patterns that reduce the context each token processes. These approaches vary in computational cost, context organization, and compatibility with pretrained models.

4. Related Work

5. Conclusion

Poolingformer combines two attention levels, using local sliding windows and pooled keys and values at a larger receptive field. It achieves state-of-the-art results on long-document QA and superior performance on summarization, with theoretical analysis and extension to other data types left for future work.

5. Conclusion

  • Poolingformer uses a smaller sliding window at the first level and a larger window with pooled keys and values at the second to reduce computational cost.Source: §5 · p. 8
  • The authors report state-of-the-art performance on long-document QA and superior performance on long-document summarization.Source: §5 · p. 8
  • Future work will examine theoretical properties of multi-level attention and extend Poolingformer to long-sequence data such as images and music.Source: §5 · p. 8

Related papers

Loading 2105.04371v2…