Source-linked AI summary
Hierarchical Transformers for Long Document Classification
Raghavendra Pappagari, Piotr Żelasko, Jesús Villalba, Yishay Carmiel, Najim Dehak
TL;DR
BERT’s limited input length makes long-document classification difficult, especially for lengthy call transcripts. The paper addresses this by segmenting inputs, encoding segments with BERT, and aggregating them with an LSTM or Transformer. Both extensions improve simple segment-prediction baselines, with state-of-the-art Fisher results and significant CSAT gains over MS-CNN.
Problem
BERT is limited to relatively short input sequences, creating a gap for classification of long documents such as call transcripts.
Method
The paper splits long inputs into segments, obtains BERT representations, and performs document classification with either an LSTM or another Transformer.
Results
ToBERT achieves state-of-the-art results on Fisher, significantly improves CSAT prediction over MS-CNN, and improves simple segment-wise aggregation baselines.
Takeaways & Limitations
RoBERT and ToBERT support BERT-based classification of long texts across customer satisfaction prediction and topic identification tasks.
Takeaways & Limitations
BERT still has long-sequence limitations from quadratic self-attention complexity and learned positional embeddings that may not generalize beyond trained positions.
Abstract
from arXiv · showhide
BERT, which stands for Bidirectional Encoder Representations from Transformers, is a recently introduced language representation model based upon the transfer learning paradigm. We extend its fine-tuning procedure to address one of its major limitations - applicability to inputs longer than a few hundred words, such as transcripts of human call conversations. Our method is conceptually simple. We segment the input into smaller chunks and feed each of them into the base model. Then, we propagate each output through a single recurrent layer, or another transformer, followed by a softmax activation. We obtain the final classification decision after the last segment has been consumed. We show that both BERT extensions are quick to fine-tune and converge after as little as 1 epoch of training on a small, domain-specific data set. We successfully apply them in three different tasks involving customer call satisfaction prediction and topic classification, and obtain a significant improvement over the baseline models in two of them.
1. INTRODUCTION
The paper addresses BERT’s limited input context for long textual sequences by introducing hierarchical extensions that segment documents and aggregate segment representations. RoBERT and ToBERT enable long-text classification, achieving state-of-the-art Fisher topic classification and significant CSAT improvement over MS-CNN.
- Motivation: BERT-based classifiers are limited to relatively short input sequences, restricting their use for long-document classification.Relevant transcripts can exceed 5000 words, and customer satisfaction may depend on temporal changes throughout a call.
- Motivation: Long call transcripts motivate models that capture dependencies across segments rather than relying only on bag-of-words representations.A customer’s satisfaction can change as an issue is resolved during the conversation.
- Proposed approach: RoBERT and ToBERT split long inputs into shorter segments, encode them with BERT, and classify the resulting sequence using an LSTM or another Transformer.These hierarchical models combine segment-wise and document-wise representations.
- Proposed approach: The proposed extensions enable BERT-based classification of long texts through segmentation and an additional layer over segment representations.This is the paper’s central architectural contribution.
- Results: RoBERT and ToBERT achieve state-of-the-art results on Fisher topic classification and significantly improve CSAT prediction over MS-CNN.These are the paper’s two headline task-level outcomes.
2. RELATED WORK
Prior work used dimensionality reduction, linear classifiers, attention networks, CNNs, and recurrent or Transformer extensions for document and customer-satisfaction classification. Existing BERT document-classification work considered documents shorter than BERT’s maximum length, leaving longer inputs insufficiently addressed.
- Document classification: Earlier document-classification approaches reduced bag-of-words representations or used hierarchical attention networks before applying simple classifiers.The cited hierarchical attention experiments used datasets averaging around 150 words.
- Customer satisfaction: Customer satisfaction prediction has been studied with logistic regression, SVM, and CNN models using different representations.These methods form part of the prior work addressed by the paper.
- Transformer-based methods: Prior BERT document-classification work used documents averaging less than BERT’s maximum length of 512.This differs from the long sequences targeted by the present paper.
- Transformer-based methods: TransformerXL extends Transformers for long-input language modeling, but relies on autoregressive modeling rather than the tasks considered here.The distinction concerns the modeling objective and task setting.
3. METHOD
The method hierarchically processes long documents by encoding overlapping fixed-size segments with BERT and aggregating them with either an LSTM or a small Transformer. RoBERT reduces self-attention complexity, while ToBERT trades asymptotic efficiency for Transformer-based segment modeling without observed practical issues on the evaluated datasets.
- 3.1. BERT: BERT uses Transformer components and produces pooled segment representations H or posterior probabilities P for classification.The paper uses BERT-Base, containing 110M parameters, while noting applicability to BERT-Large.
- 3.1. BERT: BERT’s long-sequence limitations include self-attention complexity O(n^2) and learned positional embeddings that may not generalize beyond trained positions.These limitations motivate the hierarchical design.
- 3.2. Recurrence over BERT: The method splits each input into overlapping fixed-size segments, extracts BERT representations, and stacks them into a sequence for downstream aggregation.Representations can be based on either H or P.
- 3.2. Recurrence over BERT: RoBERT aggregates segment representations with a 100-dimensional LSTM, followed by fully connected ReLU and softmax layers for document prediction.The LSTM output serves as a document embedding.
- 3.2. Recurrence over BERT: RoBERT reduces computational complexity to O(n/k*k^2) = O(nk), with the LSTM contributing negligible linear complexity O(k).Here, k denotes segment size.
- 3.3. Transformer over BERT: ToBERT replaces the LSTM with a two-layer Transformer and optionally learns positional embeddings over segment representations.Its top-level Transformer captures relationships among segments while preserving the hierarchical structure.
- 3.3. Transformer over BERT: ToBERT has asymptotically inferior complexity because its top-level Transformer is quadratic in the number of segments, but no performance or memory issues were observed on the evaluated datasets.The number of segments is much smaller than the input sequence length.
4. EXPERIMENTS
The experiments evaluate hierarchical BERT extensions across customer satisfaction and topic-identification datasets containing spoken and written documents with varied lengths.
- Datasets: The models are evaluated on CSAT, 20 newsgroups, and Fisher datasets across customer satisfaction prediction and topic identification tasks.CSAT uses spoken ASR transcripts, 20 newsgroups uses written text, and Fisher uses manually transcribed spoken conversations.
- CSAT: CSAT contains 4,331 calls rated on a 1–9 satisfaction scale and converted to balanced binary labels at the 4.5 threshold.The split includes 2,866 training, 362 validation, and 1,103 test calls.
- Fisher: Fisher contains 1,374 training and 1,372 testing telephone conversations covering 40 topics.The documents are approximately 10-minute conversations between two people discussing a given topic.
- Document lengths: Fisher documents are generally longer than 1,000 words, while more than half of CSAT documents exceed 500 words and only 10% of 20 newsgroups documents do.A few CSAT and 20 newsgroups documents exceed 5,000 words.
- Processing and features: Documents are split into overlapping 200-token segments with a 50-token shift to extract BERT features.The experiments compare representations from pre-trained and fine-tuned BERT models.
5. RESULTS
Results show that fine-tuned hierarchical BERT models improve over simpler segment-prediction aggregation and achieve especially strong performance on Fisher, with gains over CNN baselines on CSAT and Fisher.
- Model comparison: ToBERT outperforms RoBERT on Fisher and 20 newsgroups by 13.63% and 0.81%, respectively, while performing slightly worse than RoBERT on CSAT without statistical significance.Fine-tuned BERT features substantially improve results compared with pre-trained BERT features.
- Document-length analysis: ToBERT outperforms average voting in every document-length interval on the Fisher dataset.The comparison uses fine-tuned BERT segment-level predictions.
- Aggregation baselines: Simple averaging and majority voting are competitive for CSAT and 20 newsgroups but not for Fisher.The authors relate the larger hierarchical-model improvements to Fisher’s greater fraction of long documents.
- Position information: Position embeddings do not significantly affect Fisher or 20 newsgroups performance but slightly improve CSAT.These results use fine-tuned BERT segment representations.
- CSAT results: The CSAT prediction improvement over the baseline is an absolute 0.64% F1-score.The passage attributes possible difficulty to CSAT’s changing satisfaction over the course of a call.
- Comparison with prior work: ToBERT outperforms CNN-based experiments by a significant margin on CSAT and Fisher, while its 20 newsgroups result is 0.6% below the state of the art.MS-CNN is used as the CSAT baseline because of its strong results on Fisher and 20 newsgroups.
6. CONCLUSIONS
The paper evaluates RoBERT and ToBERT for long-document classification across customer satisfaction prediction and topic identification. ToBERT outperforms RoBERT, while both methods improve simple segment-wise prediction baselines and achieve their strongest results on Fisher.
- RoBERT and ToBERT were evaluated on customer satisfaction prediction and topic identification using the CSAT, 20newsgroups, and Fisher datasets.
- ToBERT outperforms RoBERT on both pre-trained and fine-tuned BERT features across all tasks.
- Both methods improve simple baselines that average or select the most frequent segment-wise predictions for long documents.
- Position embeddings did not significantly affect performance, although they slightly improved CSAT accuracy.
- The strongest results were obtained on Fisher, with good CSAT improvements over the CNN baseline.
- Both methods provide competitive performance on long sequences with quick fine-tuning.