Source-linked AI summary
MUDDLE: Measuring Understanding of Documents under Distractor and Length Effects
Jason Luo, Saibilila Abudukelimu, Judy Song, Andrew Feng, Shivank Garg, Vasu Sharma, Kevin Zhu
TL;DR
Document QA systems must handle retrieved collections where relevant sources are mixed with plausible distractors, yet prior work often conflates context length with topical similarity. MUDDLE separates these factors using controlled source, hard-negative, and random-distractor conditions with matched provenance and length. In the markdown sweep, hard negatives hurt gpt-5-mini more than matched random documents, while the small effect is directionally consistent and significant when pooled across context sizes.
Problem
Existing evaluations do not isolate whether document-QA failures arise from longer contexts or topically similar distractors, a distinction relevant to retrieved-document curation.
Method
MUDDLE evaluates 270 human-annotated questions in five source-and-distractor conditions, matching hard negatives and random distractors in provenance and length across three rendering modalities.
Results
Hard negatives lower gpt-5-mini accuracy more than length-matched random documents at k=2 and k=4, with paired gaps of 0.030 and 0.041 and pooled p=0.016.
Takeaways & Limitations
The findings suggest that distractor type may matter beyond distractor count, so benchmarks should control topical adjacency rather than length alone.
Takeaways & Limitations
The full distractor sweep is markdown-only, residual token differences remain between arms, and the small effects cannot be fully separated from that gap.
Abstract
from arXiv · showhide
Document question-answering systems increasingly answer questions over collections of retrieved documents rather than one clean source, so robustness to distracting context matters as much as reading ability. When such systems fail, it is often unclear whether the context was too long or the distractors were too close to the topic, because prior work tends to conflate these two effects. We present MUDDLE, a controlled benchmark that separates them. MUDDLE uses 270 human-annotated questions, each tied to a single source document, and instantiates every question in five conditions: the source alone, the source with two or four topically similar hard negatives, and the source with two or four random distractors. The random distractors are matched to the hard negatives in length and provenance, so an accuracy gap between the two arms reflects topical similarity rather than length. All five conditions are rendered in markdown, page images, and raw PDF, but the distractor sweep reported here is run in markdown, since a source plus its distractors exceeds current image and PDF input limits. We score answers with an LLM judge across three model families. In the complete markdown sweep, hard negatives lower accuracy more than length-matched random documents at both context sizes for gpt-5-mini, while random documents stay near the no-distractor baseline. The effect is small but directionally consistent, and for gpt-5-mini hard negatives significantly underperform length-matched random distractors when pooled across context sizes. We release the data and evaluation code for a reproducible study of context degradation.
1 Introduction
MUDDLE addresses whether document-question answering degrades because contexts grow longer or because retrieved distractors resemble the source. It introduces a controlled benchmark and finds a small, directionally consistent hard-negative effect in the complete markdown sweep.
- Motivation: Retrieved document collections often bury the answer among plausible look-alikes, making distraction robustness central to document-grounded systems.Longer contexts can degrade accuracy, while retrieval noise can pull answers off course.
- Research gap: Existing benchmarks do not separate context-length failures from failures caused by related-but-wrong documents.Single-document evaluations leave practitioners unable to decide between curating retrieved documents and truncating context.
- Benchmark: MUDDLE varies distractor type and count while holding the source document fixed and matching context length between hard-negative and random arms.The benchmark includes source-only, two-distractor, and four-distractor conditions across markdown, page images, and raw PDF.
- Benchmark: The hard-negative curation pipeline retrieves topical look-alikes, filters possible answer leakage, and verifies candidates by hand.Hard negatives are matched in length to random distractors, and questions are checked to be answerable from the source.
- Finding: Hard negatives lower gpt-5-mini accuracy more than length-matched random documents in the complete five-condition markdown sweep.The effect is significant when pooled across context sizes and remains small across the three model families.
2 Related Work
Related work shows that long contexts and noisy retrieval each challenge document understanding, but existing studies do not isolate topical adjacency in multimodal long-document settings.
- Long contexts: Long-context studies find uneven evidence use, position-dependent accuracy, and degradation as inputs grow.Models often perform best when evidence is near the beginning or end and worst in the middle.
- Retrieval noise: Retrieval-noise studies show that irrelevant or misleading context can lower answer quality, while some off-topic documents may help.The impact of retrieved noise depends on its relationship to the query.
- Hard negatives: MUDDLE imports hard negatives from dense retrieval as an evaluation condition rather than using them to train a retriever.The test asks whether models can use one relevant document surrounded by plausible look-alikes.
- Long-document benchmarks: Long-document benchmarks such as MMLongBench-Doc, DUDE, and LongDocURL broaden document understanding evaluation across length and modalities.MUDDLE responds to their shared gap by testing related distractors around a source document.
3 Benchmark Construction
MUDDLE constructs controlled multi-document contexts from human-annotated long-PDF questions, curated hard negatives, and matched random distractors, then renders them across three modalities.
- Pipeline: Figure 1 traces MUDDLE from source questions through distractor curation, multimodal rendering, and LLM-judge scoring.The construction begins with human-annotated questions over long PDFs.
- Dataset: MUDDLE contains 270 questions over 85 source documents and keeps questions answerable from text and tables rather than graphs.Each question and reference answer comes from original human annotations.
- Conditions: Each question has five contexts: source-only, HN-2, HN-4, Random-2, and Random-4.Crossing 270 questions with five conditions yields 1,350 cells per modality.
- Hard negatives: Hard negatives are retrieved, filtered for topical similarity and answer leakage, and verified by human review.The pipeline also applies automatic overlap and other screening gates before release.
- Random distractors: Random distractors come from other questions’ hard-negative pool, preserving provenance and approximately matching the hard-negative length distribution.At k=2, median page counts are 84 versus 86 pages, so same-k accuracy gaps mainly reflect topical similarity.
- Rendering: The source is placed at a seeded, reproducible position, and documents are presented as markdown, page images, or native PDF.The full distractor sweep is constrained by input-size limits for image and PDF conditions.
4 Experimental Setup
The experiments evaluate three model families with fixed prompts and a semantic LLM judge, using paired same-question comparisons for distractor effects.
- Models: The model panel comprises gpt-5-mini, gemini-3.5-flash, and grok-4.3, queried through a uniform client and message format.Reasoning effort is fixed where supported and omitted for grok-4.3.
- Prompting: Each query contains a fixed instruction, the question, and documents in their placed order.The instruction changes only the named document format across markdown, image, and PDF conditions.
- Metrics: The primary metric is boolean semantic correctness from an LLM judge receiving the question, reference answer, and model prediction.Exact match and token-level F1 are secondary metrics.
- Analysis: Distractor effects are computed as paired per-question differences restricted to questions answered in both comparison arms.Pairing is used because the effects are only a few points in size.
5 Results and Analysis
MUDDLE’s paired markdown results indicate that topical similarity harms document question answering more than matched context length, with effects varying across modalities and model families.
- Effect of Topical Similarity: 0.030 at k=2 and 0.041 at k=4: hard negatives lower gpt-5-mini judge accuracy than length-matched random distractors.The pooled topical gap is statistically significant (paired p=0.016), though the effect remains small.
- Effect of Length: 0.019 versus 0.030: increasing random-distractor load reduces accuracy less than increasing hard-negative load for gpt-5-mini.Random conditions remain close to control, while added context has a larger effect when documents are topically adjacent.
- Effect of Modality: 0.763, 0.804, and 0.804: gpt-5-mini control accuracy is reported for markdown, PDF, and image inputs, respectively.Image and PDF are evaluated only in the source-only control condition.
- Effect of Model Family: gpt-5-mini shows the clearest hard-negative ordering, while gemini-3.5-flash is flatter and grok-4.3 degrades only under the heaviest load.For gpt-5-mini, the control-to-HN-4 drop is significant (paired p=0.008); for grok-4.3, control-to-HN-4 is significant (paired p=0.026).
- Effect of Model Family: The main open question is whether the topical effect generalizes beyond the reported model and evaluation coverage.The paper identifies generalization of the topical effect as an open question.
6 Limitations
The study’s coverage and controls are limited by input-size constraints, approximate length matching, single-run evaluation, human curation, and an incomplete model panel.
- Coverage: The full five-condition distractor sweep is limited to markdown because source-plus-distractor inputs exceed current image and PDF size limits.Image and PDF results therefore cover only source-only controls.
- Experimental Control: Residual token differences between hard-negative and random pools prevent fully separating topical adjacency from context-length effects.The paper suggests a mixed-effects model with token count as a covariate for resolving this issue.
- Evaluation: Single evaluation runs per cell make the reported hard-negative penalty directional rather than precise.This limits the precision of the accuracy estimates.
- Scale: Human checking supports trustworthy curation at 270 questions but limits benchmark scale.The paper identifies judge-ensemble automation as a route toward a larger version.
- Scope: The model panel excludes claude-sonnet-4.6, and real documents vary in style, complexity, and information density.These choices constrain control and model-family coverage relative to synthetic suites and broader evaluations.
7 Conclusion
MUDDLE separates topical adjacency from added context length by holding the source fixed and matching distractor lengths. Its markdown results indicate that hard negatives degrade gpt-5-mini more than length-matched random documents, although the effect is small and directional.
- MUDDLE holds the source document fixed while matching distractor length across hard-negative and random conditions.Its curation pipeline produces topical, answer-free hard negatives that enable the comparison.
- Hard negatives cost gpt-5-mini more accuracy than length-matched random documents at both tested context sizes.Random documents remain near the no-distractor baseline.
- The degradation effect is small, model-dependent, and most pronounced at the larger context size.
- The directional findings suggest benchmarks should control topical adjacency rather than distractor length alone.The authors state that confirmation requires a stronger design.
Ethics Statement
The benchmark uses an existing public document-QA dataset and web-searched distractor documents, with two source documents triggering a provider content filter during curation.
- MUDDLE is built from an existing publicly available document-QA dataset and web-searched distractor documents.Questions and answers trace to existing human annotations.
- Two source documents triggered a provider content filter and were handled during curation.
- The released data and evaluation code target robustness to distracting context.
A.1 Benchmark construction details
The benchmark construction combines reviewed hard-negative curation, fixed source documents, reproducible distractor sampling and placement, length controls, and modality-specific serving constraints.
- Of 274 manually reviewed items, 232 passed and 42 were flagged for repair across answerability, leakage, duplication, topicality, and related checks.The answer-leak gate uses word-boundary matching while skipping short tokens and bare small numbers.
- Each of 270 questions receives five conditions with the same source and changing surrounding distractors across three modalities.
- Hard negatives and random distractors are bounded to 10–40 pages to avoid confounding distractor type with length.Hard negatives have a median length of 23 pages; 86 out-of-band documents were replaced.
- Random-2 uses the first two documents from the same seeded four-document draw as Random-4, creating a nested sampling design.
- Source insertion positions are fixed reproducibly with a cell-keyed seed shared across models and modalities.
- Multi-document image and PDF requests exceed provider limits, so the distractor sweep is restricted to markdown.Among cells, 4% of controls, 16% of k=2, and 43% of k=4 exceed 15 MB; image resolution is also mixed in controls.
A.2 Dataset and run statistics
The appendix documents condition sizes, prompts, and evaluation outputs, with markdown covering all five conditions while image and PDF runs cover controls only.
- Table 3 reports median markdown characters, median pages or page-images, and page ranges for sources and distractors.Source documents span 9–198 pages, while distractors are bounded to 10–40 pages.
- The markdown instruction requires answers using only the converted documents and “Not answerable” when the answer is absent.
- Image and PDF prompts replace markdown documents with page images and PDF documents, respectively.
- Table 4 reports judge accuracy, F1, and exact match for every model and modality run.Markdown includes all five conditions; image and PDF include controls only.
- Judge scoring marks factual equivalence as correct, matches numbers and units by value, allows minor paraphrase, and gives partial credit for missing list items.Refusals are incorrect.