BAMBOO: A Comprehensive Benchmark for Evaluating Long Text Modeling Capacities of Large Language Models

Benchmark design

• BAMBOO provides ten datasets spanning five tasks to evaluate language generation, knowledge utilization, reasoning, and tool manipulation across long texts and domains.The benchmark covers question answering, hallucination detection, text sorting, language modeling, and code completion, with salient information distributed throughout documents.Source: §1 · p. 1

Overall findings

• ChatGPT-16k consistently achieves the strongest overall performance across nearly all datasets, while other models sometimes fall below random baselines.The main exception is a minor performance gap in hallucination detection.Source: §1 · p. 2

Key research questions

• Extending context windows benefits medium-length texts but can harm performance on short texts.This finding characterizes context extension as a double-edged sword rather than a uniformly beneficial change.Source: §1 · p. 2
• Using evidence and compressing long texts generally improves LLM performance, but smaller models remain limited even with more concise inputs.The result indicates that input reduction alone does not eliminate smaller models’ performance limitations.Source: §1 · p. 2
• Model performance is sensitive to instruction position, and placing evidence at both ends can help because those positions receive larger attention scores.The benchmark therefore probes positional effects rather than treating long inputs as position-invariant.Source: §1 · p. 2

Discussion

• The paper discusses instruction forgetting, format errors, reasoning limitations, and uncommon tasks as unresolved problems in long-text modeling.It uses these issues to motivate future directions for improving long-context capabilities.Source: §1 · p. 2

Long-Context Architectures

• Long-context architectures reduce scaling challenges through recurrent, local, sparse, or approximate attention, as well as state-space and retention-based alternatives.These approaches target the quadratic memory and computational cost of standard self-attention.Source: §2 · p. 2

Adapting LLMs for Long Contexts

• LLMs’ context windows can be extended through modified positional embeddings, external memory, fusion-in-decoder, retrieval augmentation, and recurrence.These methods either alter how positions are represented or limit the amount of long-context information processed at once.Source: §2 · p. 2

Evaluation for Long Text Modeling

• Long-text evaluation datasets commonly cover language modeling, summarization, and question answering, with models predicting tokens, condensing documents, or answering document-based questions.These categories emphasize sequential prediction, compression, and information retrieval or reasoning over long inputs.Source: §2 · p. 2

BAMBOO

• BAMBOO differs by targeting comprehensive evaluation, reduced contamination, more accurate automatic metrics, and multiple length levels.Its overview covers ten datasets across diverse domains and tasks.Source: Table 2 · p. 3

Evaluation for Long Text Modeling

• Scrolls, ZeroScrolls, L-Eval, and LongBench broaden long-text evaluation across tasks and domains, but existing benchmarks face contamination and inaccurate automatic metrics.These limitations concern both overlap between evaluation and training data and the reliability of some generation metrics.Source: §2 · pp. 2–3

3. BAMBOO Benchmark

• BAMBOO evaluates long-text modeling across ten datasets and five tasks covering language generation, knowledge utilization, reasoning, and tool manipulation.Its tasks assess both whole-text comprehension and fine-grained reasoning across diverse domains.Source: §3 · p. 3, §3.1 · p. 3

3.1. Design Principles

• BAMBOO addresses benchmark reliability through contamination mitigation, precise automatic metrics, and separate 4k and 16k input-length levels.The design retains mostly 2023 data, transforms some generation tasks into multi-choice formats, and measures performance changes with longer inputs.Source: §3.1 · pp. 3–4

3.2. Data Collection

• The corpus combines NLP papers, government reports, TV transcripts, and committee meetings, with samples divided into BAMBOO-4k and BAMBOO-16k subsets.Sources are publicly accessible and released in 2023; texts are parsed as plain text, filtered by length, and truncated when overly long.Source: §3.2 · p. 4

3.3. Task Construction

• The five tasks comprise question answering, hallucination detection, text sorting, language modeling, and code completion, each targeting distinct long-range abilities.Question answering uses multi-document reasoning; hallucination detection tests entailment or contradiction; sorting reconstructs order; language modeling predicts speakers; code completion uses API documents.Source: §3.3 · p. 4

3.4. Metrics

• BAMBOO evaluates tasks with accuracy, concordance index, pass@1, and precision, recall, and F1, treating incorrectly formatted answers as incorrect.The metrics cover question answering and language modeling, sorting, code completion, and hallucination detection respectively.Source: §3.4 · p. 5

3.5. Benchmark Usage

• BAMBOO enables direct comparison of long-context LLMs, diagnosis of weaknesses, and analysis of how prompt and context length affect performance.The benchmark is distributed with repository instructions and evaluation code.Source: §3.5 · p. 5

4. Experiment

• The experiments assess zero-shot long-text modeling and analyze key questions and problems involving long-context LLMs.Source: §4 · p. 5

4.1. Baselines

• Five instruction-tuned baselines include ChatGPT, Claude2, Vicuna, Longchat, and ChatGLM2, with context lengths above 16k tokens.The models include both API-based closed models and open-source models.Source: §4.1 · p. 5
• Random baselines return random sequences, speakers, labels, or choices, while PrivateEval and AltQA receive a zero baseline because their answer spaces are infinite.Source: §4.1 · p. 6

4.2. Overall Results

• ChatGPT-16k surpasses other models on nearly all datasets and length levels, while several alternatives perform below random baselines.Source: §4.2 · p. 6
• Models struggle with subtle hallucination judgments, ShowsSort, and PrivateEval, indicating weaknesses in hallucination detection, tool manipulation, and code synthesis.The authors infer that limited diversity in training data types contributes to these deficiencies.Source: §4.2 · p. 6
• Performance generally decreases as input length extends, with a more pronounced decline on tasks requiring diverse output formats because of incorrect generation formats.Source: §4.2 · p. 6

RQ1: Do LLMs with Long Context Windows Pay for Extension Tax?

• Vicuna-16k shows an extension tax on MMLU, whereas ChatGPT remains nearly unchanged and Vicuna-16k improves on BAMBOO-4k.The authors attribute the tax to longer training data and position interpolation affecting short-context tasks.Source: RQ1: Do LLMs with Long Context Windows… · p. 6, Figure 1 · p. 6

RQ2: To What Extent Do LLMs Struggle Due to The Long Input?

• Using only concise evidence generally improves performance slightly over complete long inputs, but Vicuna declines on PaperQA and repeated errors implicate weak reasoning and coding ability.Source: RQ2: To What Extent Do LLMs Struggle… · pp. 6–7, Table 4 · p. 6

RQ3: How Do Instruction Positions Affect Long Text Modeling?

• Instruction placement is dataset- and model-dependent: beginning-only instructions often degrade with length, content-relevant tasks favor later placement, and both-end placement is usually suboptimal.The authors associate beginning-only degradation with instruction forgetting and long-range RoPE attention decay.Source: RQ3: How Do Instruction Positions… · p. 7, Table 5 · p. 7

RQ4: Can LLMs Model Longer Text with Context Compression Methods?

• Retrieval-based compression can match or exceed long-context LLM performance, whereas truncation and summarization often lose relevant information and perform poorly.Source: RQ4: Can LLMs Model Longer Text with… · pp. 7–8, Table 6 · p. 7

RQ5: Why “Lost in The Middle” Happens?

• Evidence placed at either document end produces similar or improved performance, with larger gains for longer inputs despite disrupting discourse structure.Source: RQ5: Why “Lost in The Middle” Happens? · p. 8, Table 7 · p. 8
• Attention scores form a U-shaped curve, allocating more attention to tokens at the beginning and end and helping explain the lost-in-the-middle pattern.Source: RQ5: Why “Lost in The Middle” Happens? · p. 8, Figure 2 · p. 8

• Instruction forgetting can make generated responses fail task demands, especially when instructions appear at the beginning of long inputs; robust long-instruction datasets are needed.Source: §4.4 · p. 8
• Format errors can make correct-meaning predictions unrecognizable to automatic evaluators because models add explanations, repetitions, or alternate formats.The authors suggest reducing such errors during fine-tuning or RLHF and using post-processing to rewrite responses.Source: §4.4 · p. 8
• Providing evidence or condensed text does not reliably prevent false responses, indicating that insufficient ability persists across lengths and tasks.The discussion therefore emphasizes improving overall capabilities rather than focusing only on long-text context handling.Source: §4.4 · p. 8
• LLMs perform poorly on uncommon tasks such as text sorting and code completion, sometimes falling below random baselines.The discussion attributes this to limited diversity in fine-tuning data and recommends broader task and domain coverage.Source: §4.4 · p. 8

• BAMBOO combines five tasks and two length levels to evaluate LLM long-text capabilities across varied dimensions and domains.Source: §5 · p. 9
• Evaluations of several long-context models support overall analyses of model and task performance, alongside investigations of key long-text modeling questions.Source: §5 · p. 9
• The paper discusses long-context model problems and suggests directions for improving long-text comprehension abilities.Source: §5 · p. 9

• The references include long-context evaluation resources such as L-Eval, ZeroSCROLLS, SCROLLS, and LongBench.Source: §8 · pp. 13–14
• The section cites datasets and benchmarks for information-seeking and multihop question answering, including a research-paper question-answer dataset and HotpotQA.Source: §8 · pp. 13–14
• It also lists resources for long-document and conversational summarization, including SumScreen, QMSum, and discourse-aware long-document summarization.Source: §8 · pp. 13–14