PaperlayerView exact revision on arXiv

Source-linked AI summary

MGAL: A Multilingual Granularity-Aware Long-Context Benchmark

Chunhan Li, Chenglin Xu, Zongyang Zhang, Jiale Liu, Zhuoxi Rao, Xudong Jia, Junxiu He, Menglin Yang, Wenjuan Gong, Zhengzhe Liu, Chengwei Qin

arXiv:2608.20853v1cs.AI

TL;DR

Long-context benchmarks have largely emphasized document-level comprehension in higher-resource languages, leaving fine-grained multilingual and positional behavior insufficiently evaluated. MGAL addresses this gap with a UN-report benchmark spanning four granularities, six languages, and controlled evidence positions, then evaluates 12 LLMs. Models perform relatively well on fine-grained tasks but weaken on coarser ones, while analyses expose language-resource disparities, surface-cue reliance, factual drift, and option-order behavior.

  • Problem

    Existing long-context benchmarks primarily focus on document-level comprehension and higher-resource languages, limiting fine-grained multilingual and positional evaluation.

  • Method

    MGAL builds a benchmark from 8K–128K-token UN reports across six official languages, four granularities, seven tasks, and controlled evidence positions.

  • Results

    Models perform well on word-level tasks but struggle on coarser-grained ones, while closed-source systems retain an advantage in lower-resource languages and outputs show factual drift.

  • Takeaways & Limitations

    Multilingual, fine-grained, and position-aware evaluation is necessary for diagnosing long-context model behavior across languages and granularities.

  • Takeaways & Limitations

    Automatic metrics can mischaracterize quality, while LLM-as-a-judge evaluation adds runtime cost and has position and verbosity biases.

Abstract

from arXiv · show

Evaluation of long-context Large Language Models (LLMs) has advanced rapidly. However, most existing benchmarks are limited to the document level and focus mainly on high-resource languages, leaving many fine-grained challenges insufficiently evaluated. To address this gap, we present MGAL, the first multilingual, granularity- and position-aware long-context benchmark. MGAL is constructed from United Nations (UN) reports spanning 8K to 128K tokens across the six official UN languages. It covers four coherent levels of linguistic granularity (word, sentence, paragraph, and document) and further stratifies entries by their position within the document (begin, middle, and end), indexed at both the document and paragraph levels. This design enables systematic diagnosis of multilingual long-context comprehension across different granularities. Through extensive experiments and analyses, we find that: (1) LLMs perform well at word-level tasks but struggle with coarser-grained ones; and (2) Closed-source models retain a clear performance advantage in lower-resource languages. We further identify two new challenges: (1) Under local semantic crowding, where neighboring sentences share topics and entities, models tend to follow surface cues (e.g., connectives like ``however'' or repeated entities) rather than the discourse role of the sentence in surrounding context (e.g., background, outcome); and (2) A gap between fluency and consistency in generated outputs, where models produce text that reads smoothly but drifts from the source facts. In addition, we observe several patterns in line with prior studies, including reliance on nearby evidence and reuse of options under uncertainty.

1. Introduction

MGAL addresses limits in long-context evaluation by combining multilingual coverage with controlled linguistic granularity and evidence position. Evaluations reveal degradation on coarser tasks, shallow discourse cues, fluency–consistency gaps, and positional or option-order biases.

  • Motivation: Existing long-context benchmarks mainly emphasize document-level evaluation in higher-resource languages and provide limited control over evidence position.
  • Benchmark design: MGAL uses UN reports spanning 8K–128K tokens across six official languages, with four granularity levels and seven tasks per language.The levels are word, sentence, paragraph, and document; each language has 420 query–response pairs.
  • Benchmark design: Examples are stratified by evidence location at paragraph and document levels, while tasks include QA, cloze, paragraph filling, summarization, and translation.The dataset uses manually audited query–response pairs and precision-oriented reference metrics, supplemented by LLM-as-a-judge for some open-ended tasks.
  • Findings: Under local semantic crowding, models favor connectives and repeated entities over the sentence’s functional discourse role, while fluent generations can drift from source facts.
  • Findings: Sentence-cloze errors are highest for early blanks and models repeatedly favor earlier answer options, indicating recency and option-order biases.

2. Related work

Prior work extends long-context modeling through longer inputs, sparse or compressed attention, and broader task or language coverage. MGAL complements these efforts by organizing multilingual evaluation around granularity, position, and diagnostic analysis.

  • Long-context modeling: Long-context methods extend usable sequence lengths through positional interpolation, sparse attention, and compression techniques.
  • MGAL: MGAL’s overview links aligned multilingual UN documents to tasks including QA, cloze, paragraph filling, summarization, and translation.
  • Existing benchmarks: Existing benchmarks span document comprehension, multilingual suites, synthetic length-controlled tests, and comparisons with retrieval-augmented generation.
  • MGAL: MGAL instantiates evaluation across word, sentence, paragraph, and document granularities while diagnosing position sensitivity at paragraph and document levels.

3. MGAL

MGAL is a multilingual, granularity- and position-aware benchmark built from long UN reports to diagnose long-context comprehension across four linguistic levels and six languages. Its curated tasks combine controlled evidence positions with human-verified, cross-lingually aligned examples.

  • MGAL covers Arabic, Chinese, English, French, Russian, and Spanish across 8K–128K-token UN reports.
  • The benchmark spans word, sentence, paragraph, and document granularity while stratifying evidence positions as beginning, middle, and end.
  • MGAL is built from approximately 70,000 UN reports, with roughly 4,000 structurally suitable reports selected for construction.
  • Word-level tasks: Word-level evaluation uses Single-QA and Multi-QA to test precise extraction and integration of evidence across position-controlled paragraphs.
  • Sentence-level task: Sentence-level cloze masks salient sentences and supplies distractors, requiring recovery that preserves local coherence and global document consistency.
  • Coarser-grained tasks: Paragraph filling, summarization, and translation extend evaluation to free-form generation, document condensation, and cross-lingual meaning preservation.

4. Experiments

The experiments evaluate 12 long-context LLMs zero-shot on MGAL using granularity-specific metrics. Results show strong word-level performance but weaker coarse-grained performance, with closed-source advantages in lower-resource languages and sensitivity to task configuration.

  • Evaluation is zero-shot without fine-tuning, using Accuracy, ROUGE-L, and BLEU according to task granularity.
  • The study benchmarks 12 open-source and proprietary LLMs with context windows exceeding 128K tokens.
  • GPT-5 leads word- and sentence-level multiple-choice tasks, while Grok-4, GLM-4.5, and Gemini-2.5-Flash lead paragraph filling, summarization, and translation respectively.
  • Main results: Across models, performance remains strong on word-level tasks but weakens at coarser granularities.
  • Main results: In higher-resource languages, large open-source models perform on par with closed-source systems, whereas closed-source models retain an advantage in lower-resource languages.
  • Positional results: Accuracy for word- and paragraph-level tasks is lowest when evidence appears in the middle, while sentence-cloze accuracy is highest there.

5. More Analysis

Further analyses identify two new long-context failure modes—surface-cue reliance under semantic crowding and fluent but inconsistent generation—alongside recency and option-order biases observed in prior work.

  • Local semantic crowding: Under local semantic crowding, models overweight connectives, repeated entities, and lexical overlap instead of sentence discourse roles.
  • Local semantic crowding: Models may choose a plausible summary-style sentence over the correct development sentence when repeated entities and years create misleading lexical overlap.
  • Local semantic crowding: Failure analyses verified that surface-cue reliance causes errors in filling discourse-role patterns such as “Given α ⇒β” and “Although α, still β”.
  • Fluency–consistency gap: Paragraph filling and summarization outputs are often fluent and stylistically aligned but may introduce unsupported entities or drift from the source.
  • Prior patterns: Blank omissions are more frequent near document beginnings and decline toward the end, consistent with reliance on nearby evidence and recency-weighted attention.

6. Conclusion

MGAL provides a multilingual, fine-grained, position-aware testbed showing that current LLMs handle fine-grained QA better than coarser tasks. It also exposes discourse-role, factual-consistency, recency, and option-order weaknesses relevant to future evaluation and model development.

  • MGAL evaluates 12 LLMs across four granularities, six languages, and controlled evidence positions in 8K–128K-token UN reports.
  • Models perform relatively well on fine-grained QA but weakly on coarser tasks.
  • The benchmark identifies local semantic crowding and a fluency–consistency gap as new long-context challenges.
  • MGAL also confirms recency bias and option-order heuristics as previously observed weaknesses.

Impact Statement

The benchmark uses publicly available UN Digital Library reports under their licenses while addressing privacy, sensitive content, environmental impact, and rights-holder concerns.

  • MGAL uses publicly available UN Digital Library reports under public licenses without altering their substantive meaning.The UN is credited as the original rightsholder.
  • Sensitive personal data, offensive language, and social biases are screened during benchmark construction.Potential exposure to sensitive content from the original sources persists, but the benchmark focuses on long-context capability rather than social biases.
  • Released artifacts contain only the minimum text necessary for evaluation, while full documents remain at their original sources.The paper also commits to responding to takedown or correction requests and updating documentation if legal interpretations change.
  • The methodology evaluates existing models without additional pre-training and reports experimental settings to consider environmental impact.

A. Reproducibility Statement

MGAL supports reproducibility through documented data curation, released evaluation materials, and planned release of model configurations, inference scripts, and code, with LLM-assisted construction followed by validation.

  • Dataset: MGAL documents its UN Digital Library sources, curation and annotation pipeline, sampling strategies, and task splits for release.
  • Evaluation: The authors provide exact prompts, scoring scripts, and task definitions for all evaluations.
  • Models and code: For open-source models, configuration files and inference scripts will be released; API-model request formats and parameters are documented.All preprocessing, annotation, and evaluation code will also be released upon publication.
  • LLM assistance: LLMs assisted with language refinement and initial annotation and cleaning, while the resulting data underwent final validation.

C. Limitations

MGAL’s construction combines UN reports, multilingual positional alignment, LLM-assisted question generation, and human validation across granularities, but its evaluation metrics and language coverage impose important boundaries.

  • Evaluation limits: MGAL uses automatic metrics such as Accuracy, ROUGE-L, and BLEU alongside LLM-as-a-judge for open-ended tasks.These metrics are coarse proxies for human judgment, while LLM judging adds runtime cost and known position and verbosity biases.
  • Dataset scope: The six official UN languages are all conventionally high-resource, so the benchmark compares resource imbalance within a high-resource subset.
  • Dataset scope: MGAL draws on approximately 70,000 UN reports, selecting roughly 4,000 structurally suitable reports across 8K–128K-token lengths and six languages.
  • Dataset scope: Aligned UN documents preserve equivalent paragraph and sentence positions across language versions, supporting cross-lingual comparison.
  • Validation: Each instance is checked by two trained annotators for correctness, grounding, positional validity, and cross-lingual consistency before retention.Accepted/generated counts vary by task, including 420/582 for Single-QA, 420/604 for Multi-QA, 420/800 for Cloze, 420/500 for Paragraph Filling, and 420/800 for Summarization.
  • Question construction: LLMs generate granularity-specific questions and evidence, while evaluators verify grounding, semantic roles, positions, distractor validity, and cross-paragraph relationships.The pipeline includes Numerical, Reference, Classification, Comparison, Retrieval, and Synthesis question types.

E.1.3. INSTANTIATIONS OF DIFFERENT QA TASK SAMPLE

MGAL instantiates multiple long-context QA and generation tasks across document positions, using controlled evidence selection, distractor construction, human checks, and task-specific evaluation protocols.

  • Position control: MGAL selects target sentences and paragraphs across Begin, Middle, and End regions to balance document- and paragraph-level positional coverage.Documents of 8K–16K tokens receive fewer targets than longer documents.
  • Cloze: Cloze questions replace target sentences with blanks and offer fixed option sets containing gold sentences plus locally topical confusion sentences.The evaluation tests alignment with the blank’s local context.
  • Validation: Human evaluators verify selected sentences and paragraphs against the original UN PDFs, including positional correctness, semantic role, and balanced distribution.
  • Paragraph filling: Paragraph filling removes a paragraph and asks the model to reconstruct it from surrounding context.
  • Experimental setup: The experimental setup deploys Qwen3-30B-A3B-Instruct, Mistral-Small-3.2-24B-Instruct, and Gemma-3-27B locally while accessing larger models through APIs.External tools are disabled so models rely only on the provided text.
  • Evaluation: MGAL uses Accuracy for word- and sentence-level QA, ROUGE-L for paragraph filling, and ROUGE-L or BLEU for document-level summarization and translation.Because metric scales differ, analyses emphasize relative trends rather than raw scores across granularities.
  • Evaluation: LLM-as-a-judge scores paragraph filling on Topic Fidelity, Local Coherence, Entity Consistency, Instruction Following, and Format Compliance.Each dimension receives 0–20 points, producing an overall score from 0 to 100, with text-grounded reasons required.

F.5. Human Evaluation for Paragraph-filling

Human and model evaluations show a strong fluency–consistency gap in paragraph filling, while errors reflect position bias, local semantic crowding, and source drift.

  • r = 0.871 indicates strong positive correlation between LLM-as-a-judge and human evaluation indicators.
  • Both evaluation methods find high fluency but low consistency in generated paragraphs.
  • Blank omissions peak near the document beginning and decline toward the end, consistent with recency-biased attention.
  • Models often follow connectives or repeated entities instead of the sentence’s discourse role under local semantic crowding.
  • Models reuse earlier answer options, with A and B selected more often than later choices under uncertainty.
  • Generated paragraphs imitate surface register but underweight adjacent-paragraph cues, causing unsupported entities and drift from the source trajectory.

G.2. Generated Options Analysis

Generated-option analysis classifies cloze failures by discourse role, surface heuristics, and context deficits, finding that discourse-role errors dominate and positional failures are central.

  • The error taxonomy distinguishes discourse-role underuse, surface-cue overuse, and context or knowledge deficits.
  • M1 is prioritized when the gold sentence uniquely fills the required reasoning role, while M2 captures choices best explained by surface cues.
  • M1 dominates the highly skewed Cloze error distribution, especially in the S3 connective-or-overlap subset.
  • Under local semantic crowding, models choose lexically plausible sentences that fail the blank’s required discourse role.
  • Distractor error rates generally remain within 5–15%, whereas position errors and omissions are the dominant error types.
  • The two-level analysis indicates that Cloze failures involve positional and discourse-function mismatches more than distractor-generation artifacts.
  • Average accuracy drops from 0.73 with context to 0.31 without context, showing that performance mainly draws on input information.

H.1.2. PARAGRAPH FILLING USING LLM-AS-A-JUDGE AND HUMAN EVALUATION

The evaluation compares model performance across multilingual paragraph-filling metrics and translation directions, with leadership varying by metric, language pair, and model.

  • Gemini-2.5-flash achieves the highest overall LLM-as-a-judge score and leads in Entity Consistency and Instruction Following.
  • DeepSeek V3.1 achieves the highest overall human-evaluation score and leads in Instruction Following.
  • Gemini-2.5-flash leads English-to-Chinese, Spanish, French, and Russian translation, while Grok-4 leads English-to-Arabic translation.
  • Translation leadership varies across source languages: GPT-5, Gemini-2.5-flash, Grok-4, and DeepSeek V3.1 each lead selected language directions.
  • The later translation tables likewise report model leadership differing by language pair, including Spanish-, Chinese-, French-, and Arabic-source tasks.
Loading 2608.20853v1…