PaperlayerView exact revision on arXiv

Source-linked AI summary

CDLM: Cross-Document Language Modeling

Avi Caciularu, Arman Cohan, Iz Beltagy, Matthew E. Peters, Arie Cattan, Ido Dagan

arXiv:2101.00406v2cs.CL

TL;DR

Multi-document tasks require models to capture relationships across related texts, not only structure within individual documents. CDLM addresses this gap by pretraining on related-document sets with dynamic global attention over masked tokens. It reports new state-of-the-art results across several cross-document benchmarks, with smaller models than competing systems.

  • Problem

    Multi-document tasks require modeling cross-text relationships, such as aligning matching information across documents, beyond internal text structure.

  • Method

    CDLM combines pretraining on related document sets with dynamic global attention over masked tokens to encode cross-document relationships.

  • Results

    CDLM achieves new state-of-the-art results across several cross-document benchmarks and outperforms all methods on the reported coreference results.

  • Takeaways & Limitations

    The two pretraining ideas jointly provide a general language model that can be applied to multiple cross-document tasks without task-specific architectures.

  • Takeaways & Limitations

    Comparisons with SMASH and SMITH on multihop question answering are not fully comparable because those methods used a somewhat different dataset version.

Abstract

from arXiv · show

We introduce a new pretraining approach geared for multi-document language modeling, incorporating two key ideas into the masked language modeling self-supervised objective. First, instead of considering documents in isolation, we pretrain over sets of multiple related documents, encouraging the model to learn cross-document relationships. Second, we improve over recent long-range transformers by introducing dynamic global attention that has access to the entire input to predict masked tokens. We release CDLM (Cross-Document Language Model), a new general language model for multi-document setting that can be easily applied to downstream tasks. Our extensive analysis shows that both ideas are essential for the success of CDLM, and work in synergy to set new state-of-the-art results for several multi-text tasks. Code and models are available at https://github.com/aviclu/CDLM.

1 Introduction

CDLM addresses multi-document tasks by pretraining on related document sets and using dynamic global attention to learn cross-document relationships. The resulting model improves over prior approaches across several multi-document tasks.

  • 1 Introduction: Multi-document tasks require models to represent cross-text relationships, such as aligning matching events across documents, in addition to internal text structure.Examples include cross-document coreference, document-pair relation classification, and multihop question answering.
  • 1 Introduction: Dynamic global attention lets masked tokens access the entire input during pretraining, while attention is used selectively during finetuning.The approach builds on efficient long-range transformers to process larger amounts of cross-document context.
  • 1 Introduction: CDLM combines pretraining over related documents with a global attention pattern to learn and implicitly encode informative cross-document relationships.Related documents provide abundant overlapping information that can guide masked-token prediction.
  • 1 Introduction: CDLM improves consistently over previous approaches on cross-document coreference resolution, multihop question answering, and document matching tasks.The paper also reports controlled ablations of related-document pretraining and dynamic global attention.

2 Method

CDLM pretrains a long-range transformer on clusters of related documents, using cross-document masking and dynamic global attention over masked tokens. This design encourages alignment across documents while retaining efficient local attention for other tokens.

  • 2 Method: The method uses two ideas: related-document pretraining and dynamic global attention over masked tokens to reference the entire cross-text context.These ideas are presented as the core components of the new cross-document language-modeling approach.
  • Pretraining Over Related Documents: CDLM pretrains on related-document clusters so overlapping information can teach the model cross-text mapping and alignment.The clusters describe the same topic and are available in multi-document benchmarks such as Multi-News and ECB+.
  • Pretraining With Global Attention: CDLM is based on Longformer-style sparse attention, combining local windows with global attention while scaling to long inputs.The setup is intended to support contextualization across multiple documents and can generalize to similar efficient transformers.
  • Pretraining With Global Attention: During pretraining, a masked token globally attends to the concatenated document sequence while unmasked tokens retain local attention.For the alleges example, the masking scheme encourages using the other document, especially the matching suing event, to predict the masked word.
  • Experimental Details: The pretraining corpus uses MultiNews document clusters, retaining clusters with at least three documents for cross-document pretraining.The source dataset contains 44,972 training clusters with two to ten documents each.

3 Evaluations and Results

CDLM is evaluated through intrinsic perplexity, multiple downstream multi-document tasks, and attention analysis, with ablations isolating related-document pretraining and global attention. It generally improves over prior and ablated approaches, while performance comparisons have dataset, compute, and model-scope qualifications.

  • 3 Evaluations and Results: The evaluation covers intrinsic cross-document perplexity, coreference, document matching, multihop question answering, and attention analysis using ablations and baselines.The ablations vary document relatedness, global-attention placement, and whether additional pretraining is performed.
  • 3.2 Cross-Document Coreference Resolution: CDLM outperforms prior methods on event and entity cross-document coreference, with statistically significant results and especially large gains on entity coreference.The model surpasses state-of-the-art entity results despite competitors using external information and larger pretrained models.
  • 3.3 Document matching: CDLM outperforms prior document-matching methods on citation recommendation, including task-specific CDA, but does not improve plagiarism detection.On PAN, BERT-HAN+CDA reaches an F1 score of 79.6, substantially below the authors’ models; the authors suggest PAN’s small size may contribute.
  • 3.4 Multihop Question answering: CDLM outperforms all ablated and comparably sized models on HotpotQA-distractor, except BIGBIRD, especially for supporting-evidence detection.BIGBIRD used substantially more pretraining data and compute, so the comparison is not resource-matched.
  • 3.5 Attention Analysis: Up to a 10% MPR gap between Local and Global models on positive examples indicates that global attention helps encode cross-document alignment.The attention analysis assigns global attention to source-span tokens and measures the target span’s percentile rank among tokens in its document.

4 Related Work

Prior work processes long or paired inputs and explores cross-document attention, but CDLM is presented as a general pretrained model for varied multi-document tasks without a two-document restriction.

  • Long-context models concatenate multiple documents into one sequence, but their pretraining objectives consider only single documents.
  • Cross-document attention methods such as CDA and DCS focus on document-pair encoding or matching rather than general multi-document pretraining.
  • CDLM is a general pretrained language model for diverse multi-document tasks without restrictions on the number of input documents, within Longformer's input length.
  • Related cross-encoder approaches jointly encode paired information sources, including translated sentences in TLM and images with text in crossmodality encoders.

5 Conclusion

CDLM introduces a cross-document pretraining strategy that uses related document clusters and an extended global-attention mechanism to improve cross-document representations.

  • Together, the pretraining strategy and attention technique provide better encoding for cross-document downstream tasks.
  • CDLM pretrains on clusters of related documents through cross-document masking to learn cross-document relationships.
  • The method extends Longformer’s global attention mechanism so it is applied during pretraining.

A Dataset Statistics and Details

This section describes the pretraining corpus and the benchmarks used for evaluation.

  • The experiments use a specified pretraining corpus and a set of evaluation benchmarks.
  • The section provides details about the corpus used during pretraining.
  • The section also provides details about the benchmarks used in the experiments.

A.1 Multi-News Corpus

The Multi-News corpus uses preprocessed documents organized into related clusters, with reported corpus size and input-length statistics.

  • Multi-News Corpus: The corpus is the preprocessed, untruncated version of Multi-News, totaling 322MB of uncompressed text.
  • Multi-News Corpus: Each preprocessed document contains up to 500 tokens, while average and 90th-percentile input lengths are 2.5k and 3.8K tokens.
  • Multi-News Corpus: Related-document cluster sizes are listed in Table 7 and follow the dataset construction proposed by Fabbri et al. (2019).

A.2 ECB+ Dataset

The ECB+ dataset uses predefined topic-based training, validation, and test splits, with statistics reported for topics, documents, mentions, and coreference clusters.

  • ECB+ follows the split used by previous works, assigning topics 1, 3, 4, 6-11, 13-17, 19-20, 22, and 24-33 to training.
  • Table 8 reports ECB+ statistics for topics, documents, mentions, and coreference clusters, including separate event/entity counts for mentions and clusters.

A.3 Paper Citation Recommendation & Plagiarism Detection Datasets

The document-to-document benchmarks use statistics compiled from prior work and preprocessing procedures that restrict document pairing, downsample some datasets, and randomly sample negative pairs.

  • The benchmark statistics cover training, validation, and test splits, plus document-pair and unique-document counts.
  • AAN retains document pairs with abstracts and uses only those abstracts, whereas OC keeps one citation per paper and is significantly downsampled.
  • S2ORC forms pairs from citing sections and cited-paper abstracts and is significantly downsampled during preprocessing.
  • Negative pairs are sampled randomly for all datasets, followed by filtering characters to digits, letters, punctuation, or whitespace.

B CDLM Pretraining Hyperparameters

CDLM pretraining uses long input sequences, accumulated batches, mixed-precision training, and a scheduled learning rate, while downstream representations combine cross-document mention vectors with attention-based inputs.

  • B CDLM Pretraining Hyperparameters: Pretraining uses sequences of length 4,096, an effective batch size of 64, a maximum learning rate of 3e-5, and 500-step linear warmup.The warmup is followed by power-3 polynomial decay.
  • B CDLM Pretraining Hyperparameters: Figure 4 distinguishes global-attention tokens from local-attention tokens in the CD-coreference pairwise-mention representation.
  • B CDLM Pretraining Hyperparameters: The reported downstream representation uses cross-document contextualized vectors for mentions i and j and the [CLS] token, combined with an element-wise product.
  • B CDLM Pretraining Hyperparameters: Mixed-precision 16-bit training reduces memory consumption and speeds training; pretraining took 8 days on eight 48GB RTX8000 GPUs.

C Finetuning on Downstream Tasks

The downstream-task implementation section covers the hyperparameter choices and algorithms used for the experiments.

  • C Finetuning on Downstream Tasks: The implementation details concern downstream tasks rather than pretraining alone.
  • C Finetuning on Downstream Tasks: The section documents the hyperparameter choices used in the downstream experiments.
  • C Finetuning on Downstream Tasks: It also describes the algorithms used for the downstream-task experiments.

C.1 Cross-Document Coreference Resolution

CDLM’s cross-document coreference scorer encodes concatenated documents with marked mentions and global attention, then classifies mention pairs using a learned pairwise scorer.

  • Mention pairs are scored after concatenating relevant documents and encoding them with CDLM, while within-document examples use only the containing document.
  • Special mention markers and global attention focus the model on candidate mentions, [CLS], and the marked boundaries.
  • The pairwise representation concatenates the [CLS] vector with cross-document mention representations, where each mention vector sums its candidate-token representations.
  • At test time, agglomerative clustering merges the most similar cluster pairs to form coreference clusters.
  • Training uses all same-cluster mention pairs as positives and randomly sampled negatives before a one-hidden-layer MLP pairwise scorer.
Loading 2101.00406v2…