Source-linked AI summary
Long-context LLMs Struggle with Long In-context Learning
Tianle Li, Ge Zhang, Quy Duc Do, Xiang Yue, Wenhu Chen
TL;DR
Long-context evaluations have often failed to establish whether models can comprehend and reason over entire, context-rich inputs. The paper introduces LongICLBench, an extreme-label in-context classification benchmark, and finds substantial degradation on harder tasks, with strong sensitivity to example position. The benchmark is limited to this single evaluation application, leaving other full-context tasks for future work.
Problem
Perplexity, retrieval, question answering, and summarization evaluations provide insufficient evidence about LLMs’ ability to comprehend and reason over entire long input sequences.
Method
The paper builds LongICLBench from six extreme-label classification tasks with varied label spaces and context lengths, then evaluates recent long-context LLMs.
Results
Models generally perform better on simpler tasks but degrade as difficulty increases; on Discovery, all LLMs except Gemini-1.5-Pro achieved close-to-zero performance, while Gemini-1.5-Pro reached 14% accuracy.
Takeaways & Limitations
LongICLBench offers a more realistic test of long-context understanding and reveals that current LLMs still struggle with long in-context learning.
Takeaways & Limitations
LongICLBench currently covers only extreme-label classification with long in-context learning, leaving other full-context long-context tasks unaddressed.
Abstract
from arXiv · showhide
Large Language Models (LLMs) have made significant strides in handling long sequences. Some models like Gemini could even to be capable of dealing with millions of tokens. However, their performance evaluation has largely been confined to metrics like perplexity and synthetic tasks, which may not fully capture their true abilities in more challenging, real-world scenarios. We introduce a benchmark (LongICLBench) for long in-context learning in extreme-label classification using six datasets with 28 to 174 classes and input lengths from 2K to 50K tokens. Our benchmark requires LLMs to comprehend the entire input to recognize the massive label spaces to make correct predictions. We evaluate on 15 long-context LLMs and find that they perform well on less challenging classification tasks with smaller label space and shorter demonstrations. However, they struggle with more challenging task like Discovery with 174 labels, suggesting a gap in their ability to process long, context-rich sequences. Further analysis reveals a bias towards labels presented later in the sequence and a need for improved reasoning over multiple pieces of information. Our study reveals that long context understanding and reasoning is still a challenging task for the existing LLMs. We believe LongICLBench could serve as a more realistic evaluation for the future long-context LLMs.
1 Introduction
Existing long-context evaluations often test perplexity, retrieval, or question answering, but may not require models to comprehend and reason over an entire input. LongICLBench addresses this gap with extreme-label in-context classification and shows that performance degrades as task complexity increases.
- Motivation: Perplexity and passkey retrieval provide limited evidence about realistic long-sequence understanding, while question answering and summarization can exploit shortcuts or positional bias.These evaluations may not require reading and reasoning over the entire input sequence.
- Proposed benchmark: Extreme-label in-context classification requires scanning demonstrations to recognize the complete label space before predicting the correct class.Large label spaces can make demonstrations long; Discovery has 174 classes and already exceeds 10K tokens with one example per class.
- Proposed benchmark: LongICLBench evaluates six fine-grained classification tasks across varying context lengths and label-space difficulty using long-context LLMs.The benchmark is designed to assess how extended inputs affect in-context learning performance.
- Findings: Open-source models’ performance generally declines as tasks become more complex, while longer demonstrations help only within a certain input-length range.As inputs grow longer, performance can worsen or fluctuate.
- Findings: On Discovery, all evaluated LLMs achieved close-to-zero performance except Gemini-1.5-Pro, which reached 14% accuracy.The result contrasts with a fine-tuned BERT model achieving 87% on the same task.
- Findings: The study also finds that instance position in prompts can dramatically influence some models’ performance.This analysis examines how the distribution of label positions affects long in-context learning.
2 Related Work
Related work develops methods for extending context windows and benchmarks for evaluating long-context models, but many evaluations emphasize retrieval or summarization. Extreme-label classification provides a complementary setting involving very large label spaces and precise distinctions among classes.
- Long in-context learning: Long in-context learning research reports that more demonstrations can improve performance, although longer prompts can also diminish performance.Prior effectiveness may be constrained by the maximum sequence length seen during training.
- LongICLBench: LongICLBench balances token length and evaluation difficulty by constructing one to five complete rounds of examples covering selected labels.Instances are sampled evenly across classes to reduce label-distribution bias.
- Long-context techniques: Prior long-context research extends usable sequence lengths through position methods, continued pre-training, sliding windows, and segmentation.These approaches address limitations associated with long inputs and Transformer computation.
- Long-context evaluation: Existing benchmarks span sequence lengths from 1K to 200K tokens and cover retrieval, question answering, summarization, reasoning, and other long-context tasks.Examples include Long-Range Arena, LongBench, L-Eval, LooGLE, and ∞Bench.
- Extreme-label classification: Extreme-label classification assigns data to one of a very large number of labels and requires precise differentiation across expansive label spaces.Applications include emotion classification, named entity recognition, and biological function prediction.
3 Long In-context Evaluation
LongICLBench evaluates long in-context learning through six extreme-label classification datasets and varied demonstration lengths, comparing long-context models under increasing task complexity. Performance is strong on simpler tasks but degrades sharply on difficult, large-label problems such as Discovery.
- Benchmark Construction: LongICLBench contains six datasets spanning 28 to 174 labels and supports evaluation sequences organized into one to five complete label-coverage rounds.Each round contains examples covering all selected labels, enabling controlled variation in demonstration length.
- Benchmark Construction: The benchmark covers diverse classification settings, including emotion, intent, relation, entity, dialogue-relation, and discourse-marker classification.The datasets include GoEmotions, BANKING77, TacRED, Few-NERD, DialogRE, and Discovery.
- Model and Experimental Setup: The evaluation compares open-source long-context models with API-based systems, using balanced demonstrations with evenly distributed labels across models.The evaluated systems include models around 7B parameters as well as Gemini-1.5-Pro, Claude3-Opus, and GPT-4-turbo.
- Experimental Results: On BANKING77, most models benefit from longer demonstrations between 2K and 4K tokens, after which additional examples provide limited gains.BANKING77 contains 77 intents and reaches approximately 14K tokens across five rounds.
- Experimental Results: Qwen and Mistral show nearly linear performance changes with demonstration length, suggesting a predictable relationship between task complexity and extreme-label ICL performance.This observation is reported from the left sub-graph of Figure 3.
4 Exploratory Experiment
The exploratory experiments examine how example positioning affects long in-context classification. Models are sensitive to grouped demonstrations, with some relying primarily on labels near the prompt’s end.
- 4.1 Scattered Distribution: GPT4-turbo consistently exceeds 80% accuracy for most label types in the scattered TacRED configuration.
- 4.2 Grouped Distribution: Grouping examples by class generally reduces performance across models, with Mistral and InternLM2 showing particularly large drops.
- 4.2 Grouped Distribution: InternLM2 and Mistral show strong distribution sensitivity, handling mainly instances whose labels occur near the prompt’s end.
- 4.2 Grouped Distribution: ChatGLM3-6B-32K is comparatively resilient, with only a 3.3% accuracy drop after instance positions are changed.
- 4.2 Grouped Distribution: GPT4-turbo and Gemini1.5-Pro experience accuracy declines of 20.3% and 22.3%, respectively, under grouped distributions.
5 Conclusion
The study evaluates long in-context learning through extreme-label classification and finds substantial degradation on more difficult tasks. Example distribution within prompts also affects model performance.
- LongICLBench covers long in-context learning tasks with varying difficulty, especially in extreme-label classification.
- LLMs show dramatic performance degradation on more difficult tasks.
- Exploratory experiments highlight that the distribution of examples within prompts affects model performance.
A.1 Additional Datasets
The additional datasets include GoEmotions and Few-NERD, while the accompanying material points to a result table for GoEmotion.
- GoEmotions contains 58k English Reddit comments labeled across 27 emotion categories or Neutral, with selected examples averaging 28 tokens.
- Few-NERD provides 8 coarse-grained and 66 fine-grained entity types in a human-annotated named entity recognition dataset.
- The GoEmotion result is presented in Table 7.
A.2 Prompting Template
The prompting template for each dataset is presented in Table 9.
- Table 9 presents the prompting template used for each dataset.
A.3 Additional Distribution Analysis
The analysis reorganizes same-class demonstrations so they appear adjacent, then compares performance before and after grouping. Distribution plots for additional models are provided in Figures 5 and 6.
- Same-class instances are organized adjacently within demonstration prompts to compare grouped and random distributions.The reorganization enables a pre-grouping versus post-grouping performance comparison in Table 10.
- Figures 5 and 6 present distribution plots for other models.
A.4 Data Accessibility
The benchmark materials are released under an MIT license, with dataset curation governed by the original datasets’ rules. The appendix also includes a Few-NERD results table and visualizes class-wise accuracy under alternative demonstration distributions.
- LongICLBench is released under an MIT license, free of charge, with no personally identifiable or offensive content in the dataset.The datasets are curated according to rules guaranteed by their original datasets.
- Table 8 reports results for the Few-NERD dataset.
A.5 Broader Society Impacts
The paper discusses potential benefits and risks of improved long-context evaluation while acknowledging that LongICLBench currently covers only extreme-label classification. Its broader-impact discussion therefore points toward future tasks requiring full understanding of long contexts.
- Long-context evaluation insights could support development of techniques for legal analysis, long-form journalism, and comprehensive academic summarization.
- Table 9 documents each dataset’s prompt format, designed to use the context and format of its respective data effectively.
- LongICLBench currently covers only one evaluation application: extreme-label classification with long in-context learning.Other long-context tasks requiring full understanding of the entire context remain to be developed.
- Table 10 compares random and grouped arrangements of same-typed demonstrations for TacRED 3 Round.