PaperlayerView exact revision on arXiv

Source-linked AI summary

Beyond Monolingual Deep Research: Evaluating Agents and Retrievers with Cross-Lingual BrowseComp-Plus

Yuheng Lu, Qingcheng Zeng, Heli Qi, Puxuan Yu, Fuheng Zhao, Rui Yang, Hitomi Yanaka, Naoto Yokoya, Weihao Xuan

arXiv:2606.15345v2cs.CLcs.IR

TL;DR

Existing browsing benchmarks largely keep queries and supporting evidence in one language, leaving cross-language deep research unmeasured. XBCP preserves English questions and answers while varying evidence languages across two settings. Translated corpora reduce evidence recall, increase search effort, and lower citation reliability, while translated-oracle accuracy remains below original.

  • Problem

    Existing browsing benchmarks largely assume queries and supporting evidence share a language, leaving the cross-lingual deep-research setting unmeasured.

    Source: §1 · p. 2

  • Method

    XBCP preserves English questions and answers while varying evidence languages in single-language-per-query and equally distributed 12-language settings.

    Source: §3.1 · p. 3

  • Results

    Translated corpora reduce evidence recall, increase search effort, and lower citation reliability even with dense multilingual retrieval; translated-oracle accuracy remains below original.

    Source: §5 · p. 8

  • Takeaways & Limitations

    Cross-lingual deep research involves distinct bottlenecks: finding language-mismatched evidence and using it faithfully once found.

    Source: §5 · p. 8

    Main experiments used one run per agent–retriever–corpus configuration, leaving variance estimates and multi-run significance tests for future work.

    Source: Limitations · p. 9

Abstract

from arXiv · show

Deep research agents are increasingly evaluated on their ability to search for evidence, reason over retrieved sources, and produce grounded answers. Existing browsing benchmarks, however, largely assume that the user's query and the supporting evidence are written in the same language, leaving open whether agentic search systems can operate when relevant evidence appears in another language. We introduce XBCP (Cross-lingual BrowseComp-Plus), a controlled benchmark that preserves the English question-and-answer space of BrowseComp-Plus but varies the languages of the supporting documents. XBCP instantiates two complementary settings: in the cross-lingual setting, each query is paired with evidence in a single assigned language. In the multilingual setting, the full evidence corpus is distributed equally and randomly across 12 languages spanning high-resource and low-resource regimes. We evaluate four deep research agents using sparse and dense multilingual retrievers, measuring answer accuracy, evidence recall, search behavior, calibration, citation fidelity, and oracle retrieval. Results reveal substantial degradation when evidence is translated. Even strong, dense retrievers lose evidence recall, and agents become less calibrated and cite evidence less reliably. Notably, accuracy remains lower even when all gold evidence is supplied directly. These findings suggest that cross-lingual deep research exposes both retrieval failures and an independent, agent-side difficulty in integrating language-mismatched evidence.

1 Introduction

XBCP extends controlled browsing evaluation to English questions and answers supported by evidence in other languages. Its experiments reveal degradation in retrieval and agent-side evidence integration.

1 Introduction

  • BrowseComp-Plus (Chen et al., 2025) uses a fixed, human-verified corpus to separate controlled browsing evaluation from time-varying live search.Deep research agents plan searches, inspect sources, judge evidence sufficiency, and compose grounded answers; LLM agents more broadly acquire, filter, and synthesize external evidence.Source: §1 · p. 1
  • XBCP experiments reveal large drops in accuracy and evidence recall across retrievers, reduced citation reliability, and persistent degradation even under oracle retrieval.The findings indicate that cross-lingual deep research challenges both retrieval and agents’ integration of language-mismatched evidence.Source: §1 · p. 2
  • XBCP preserves BrowseComp-Plus’s English question-and-answer space and fixed corpus while varying evidence languages in cross-lingual and multilingual settings.Cross-lingual queries use evidence in one assigned language; multilingual evidence is randomly and equally distributed across 12 high- and low-resource languages.Source: §1 · p. 2
  • Unlike monolingual browsing benchmarks such as BrowseComp-ZH (Zhou et al., 2025), XBCP tests English questions answered from evidence in another language.It is presented as the first benchmark to formalize cross-lingual deep research and evaluate retrieval and evidence integration in an agentic search process.Source: §1 · p. 2

2 Related Works

Deep-research systems and multilingual retrieval have advanced, but cross-lingual deep research remains underexplored: existing work is largely monolingual, domain-specific, or evaluated outside agents’ iterative research loop.

2 Related Works

  • Existing deep-research systems and benchmarks are largely monolingual or domain-specific, while CLIR evaluations omit agents’ iterative search, evidence selection, and synthesis loop.The literature includes long-horizon research agents and benchmarks for Chinese browsing, financial search, and noisy or conflicting results, but not this cross-lingual agent setting.Source: §2 · pp. 2–3
  • Deep-research agents extend tool-augmented LLMs to long-horizon information seeking, while newer systems study scaling and verification and benchmarks broaden task difficulty.Examples include OpenAI Deep Research, Tongyi DeepResearch (Team et al., 2026), MiroThinker (MiroMind Team et al., 2026), and Marco DeepResearch (Zhu et al., 2026); benchmarks include BrowseComp-ZH (Zhou et al., 2025), FinSearchComp (Hu et al., 2025), and SealQA (Pham et al., 2026).Source: §2 · p. 2
  • Multilingual retrieval research has expanded beyond translation-mediated CLIR, but its benchmarks remain fixed-collection ranking tasks rather than agentic research loops.Systems include ME5 (Wang et al., 2024), MGTE (Zhang et al., 2024), ARCTIC-EMBED 2.0 (Yu et al., 2024), and QWEN3 EMBEDDING (Zhang et al., 2025); evaluations include MMTEB (Enevoldsen et al., 2025), MIRACL, and MLDR. Retrieval quality also depends on representation alignment, resource imbalance, domain transfer, and evaluation design (Goworek et al., 2025); CLIR experiments find multilingual bi-encoders and translation-based lexical retrieval dominate across datasets and language regimes (Zuo et al., 2025), while fact-checking studies report different rankings and adaptation gains (Ramponi et al., 2025).Source: §2 · pp. 2–3

3 Building XBCP

XBCP holds questions and answer evaluation in English while varying evidence languages through translation and two evaluation configurations. Expert verification found that translated documents were generally usable, with all language-level mean scores above 4.0/5.

3.1 Translation-Based Construction

  • XBCP keeps questions and answer evaluation in English while translating evidence into 11 non-English languages spanning high- and low-resource settings.The language set includes Chinese, English, French, German, Japanese, Korean, Portuguese, Spanish, Swahili, Wolof, Yoruba, and Zulu; the translation prompt covers titles, terminology, proper nouns, and metadata while preserving URLs, email addresses, formulas, and code.Source: §3.1 · p. 3
  • Cross-lingual evaluation assigns each query one evidence language (830 query assignments; 5,040 document assignments), while multilingual evaluation distributes 5,040 instances evenly across 12 languages (420 each).The cross-lingual setting uses English as an untranslated reference; the multilingual assignment is random and equal across languages.Source: §3.1 · p. 3

3.2 Verification and Quality Control

  • All language-level translation means exceeded 4.0/5 across 2,200 expert-rated instances, suggesting generally usable evidence despite possible residual artifacts.Annotators rated accuracy, fluency, and completeness on a 1–5 scale.Source: §3.2 · pp. 3–4

4 Experiments and Results

XBCP tests four agents with controlled sparse and dense retrieval across original and translated evidence, and finds that language mismatch degrades retrieval, answer quality, calibration, and citation fidelity. Oracle and supplementary analyses show that better retrieval, query formulation, and increased reasoning effort can help, but do not eliminate the cross-lingual difficulty.

4.1 Experimental Setup

  • The evaluation pairs four agents with controlled retrievers over three corpus conditions, measuring answer accuracy, evidence recall, search calls, calibration, citation behavior, and oracle performance.Oracle evaluation supplies all supporting evidence directly to isolate reasoning errors from retrieval errors; supplementary analyses examine languages, query expansion, and reasoning effort.Source: §4.1 · p. 4

4.2.1 End-to-End Agent Evaluation

  • With QWEN3-EMBEDDING-8B, accuracy drops roughly 16–23 pp across agents on translated evidence, while multilingual and cross-lingual results are close for most agent–retriever pairs.Agents generally search more after translation without recovering lost accuracy, and calibration error increases in both translated settings.Source: §4.2.1 · pp. 4–5
  • DEEPSEEK-V4-PRO with QWEN3-EMBEDDING-8B achieves 64.70% accuracy on the original, 48.80% on the multilingual, and 42.29% on the cross-lingual corpus, the strongest overall performance.Among agents evaluated with the full retriever suite, QWEN3-EMBEDDING-8B also gives the strongest original-corpus performance.Source: §4.2.1 · p. 4, Table 1 · p. 5

4.2.2 Retriever Evaluation

  • QWEN3-EMBEDDING-8B retrieves the most supporting evidence, while BM25 drops sharply on translated evidence and other dense multilingual retrievers recover only part of the loss.Citation coverage, precision, and recall also decline after translation, so language mismatch affects both retrieval and faithful source attribution.Source: §4.2.2 · p. 5, Table 3 · p. 6

4.2.3 Oracle Retrieval

  • Oracle retrieval improves accuracy by over 55 pp on the original corpus and roughly 65–75 pp after translation, yet translated-evidence oracle accuracy remains lower for every agent.Even with supporting evidence supplied, agents must align relevant facts with English questions and synthesize answers without losing evidential constraints.Source: §4.2.3 · pp. 5–6, Table 4 · p. 6

4.3.1 Per-Language Decomposition

  • For QWEN3.6-35B-A3B with QWEN3-EMBEDDING-8B, high-resource languages average 18.39% tool accuracy and 28.48% evidence recall, versus 10.87% and 18.00% for low-resource languages.Their oracle accuracies are closer at 89.67% and 87.32%, respectively; Swahili and Wolof retain roughly 86–90% oracle accuracy while tool accuracy falls to roughly 15%. Resource level alone does not explain variation: Japanese and Korean lag within the high-resource group, and Zulu shows a similar pattern among low-resource languages.Source: Table 5 · p. 6, §4.3.1 · p. 6

4.3.2 The Impact of Query Expansion

  • Reasoning-based query expansion increases original-corpus accuracy by 3.25 pp and evidence recall by 4.86 pp, with smaller gains after translation and slightly fewer search calls.Concatenating the agent’s reasoning trace to its query improves search without retriever training or index changes, but does not by itself bridge the language gap.Source: Table 6 · p. 7, §4.3.2 · p. 7

4.3.3 The Impact of Reasoning Effort

  • From low to high reasoning effort, original-corpus accuracy rises from 15.18% to 36.02%, while high effort requires over 26 searches per query versus roughly 2 at low effort.Accuracy and evidence recall improve in all three corpus settings, but high-effort translated runs reach only about low-effort original accuracy despite over 14 times as many searches.Source: Table 7 · p. 7, §4.3.3 · p. 7

5 Discussion

XBCP frames cross-lingual deep research as two linked challenges: finding language-mismatched evidence during agentic search and integrating it faithfully once found. Its findings point beyond retriever strength toward language-aware coordination across the search and answer pipeline.

  • By varying only evidence language, XBCP isolates how mismatch propagates through evidence seeking and connects multilingual retrieval evaluation with iterative agentic search.Source: §5 · p. 8
  • Translated corpora reduce evidence recall, increase search effort, and lower citation reliability even with dense multilingual retrievers, although dense retrievers outperform BM25 after translation.Source: §5 · p. 8
  • Even when all gold evidence is supplied, translated-oracle accuracy remains below original-language accuracy, revealing an evidence-integration bottleneck beyond finding documents.Agents must identify relevant facts in non-English sources, align them with the English question and answer space, and preserve evidential constraints during synthesis.Source: §5 · p. 8
  • Low-resource languages have substantially lower tool-based accuracy and evidence recall than high-resource languages, but comparable oracle accuracy, locating their largest penalty before evidence reaches the model.Source: §5 · p. 8
  • Cross-lingual deep research calls for coordinated language-aware retrieval, query planning, reading, and citation, including cross-language queries, entity variants, and selective translation.Source: §5 · p. 8

Limitations … C Translation Verification Rubrics

The appendix documents limitations on the evaluation design and details how XBCP documents are translated and checked. It also describes the language assignments used in the corpus settings.

Limitations

  • Each agent–retriever–corpus configuration was run once, leaving formal variance estimates and multi-run significance tests to future work despite large, consistent gaps.The authors cite the computational cost of running full search trajectories across retrievers and corpus conditions.Source: Limitations · p. 9
  • Agents used one recommended inference configuration without temperature or top-p tuning; condition-specific tuning, particularly for low-resource languages, may partly reduce observed gaps.A systematic study of inference configuration is beyond the work’s scope.Source: Limitations · p. 9

A XBCP Construction Details

  • The cross-lingual setting reports language-assignment statistics, while the multilingual setting reports per-language coverage and retains 420 English documents alongside 4,620 translated document instances.The 4,620 translated instances are produced from the remaining documents; the English source documents are retained unchanged.Source: Table 9 · p. 11, Table 8 · p. 11

B Translation Prompt

  • The translation prompt requests complete translation, including proper nouns, titles, terminology, and metadata field names, with culturally appropriate language and established translations for titles where available.The instructions illustrate translating metadata field names according to target-language conventions.Source: B Translation Prompt · p. 11
  • The prompt requires URLs, email addresses, math formulas, and code blocks to remain unchanged and requests output consisting only of the translated document..Source: B Translation Prompt · p. 11

C Translation Verification Rubrics

  • Following MMLU-ProX (Xuan et al., 2025), expert bilingual evaluators compare source and machine-translated documents and score accuracy, fluency, and completeness from 1 to 5, explaining any score below 5.The rubric’s fully complete criterion requires retaining the full meaning without omissions or additions and accurately conveying details, data, and annotations.Source: C Translation Verification Rubrics · pp. 11–12

D Translation Verification Results · E Judge Prompt

The appendix reports per-language translation verification procedures and specifies how an LLM judge evaluates answers when evidence and answers may use different languages.

D Translation Verification Results

E Judge Prompt

  • The judge is instructed to assess whether extracted answers match the English correct answer across languages, including translations or transliterations.Source: E Judge Prompt · p. 13
  • The judge extracts the response’s final exact answer, using “None” when no exact final answer can be extracted.Source: E Judge Prompt · p. 13
  • The judge explains correctness only by whether answers identify the same entity or value, including whether different-language answers are translations or transliterations.Source: E Judge Prompt · p. 13
  • The judge marks an answer correct for a match, translation, transliteration, or small numerical error margin, and incorrect for inconsistency, ambiguity, non-equivalence, or error.Source: E Judge Prompt · p. 13
  • The judge extracts the response’s confidence score as a percentage, defaulting to 100 when no confidence score is available.Source: E Judge Prompt · p. 13

F Decomposing the Agent Cross-lingual Bottleneck

Gold evidence alone does not restore monolingual performance, indicating an agent-side cross-lingual bottleneck. The section tests whether removing language switching helps by placing the prompt, query, and evidence entirely in the target language.

G Citation Precision Error Analysis

GPT-OSS-120B’s citation precision drops sharply on translated evidence, and no-gold retrieval accounts for most of its citation errors. Its higher citation coverage also accompanies more total errors than the reference agents.

  • 50.89% citation precision on the original corpus fell to 24.30% (multilingual) and 26.26% (cross-lingual) for GPT-OSS-120B, roughly a 50% reduction.Source: G Citation Precision Error Analysis · p. 14
  • The analysis separates queries with citations but no gold citation into mapping failures (gold retrieved, other documents cited) and no-gold-retrieved errors (no gold retrieved, English negatives cited).Source: G Citation Precision Error Analysis · p. 14, Table 12 · p. 14
  • 66.13% of GPT-OSS-120B’s multilingual citation errors were no-gold-retrieved errors, up from 57.08% on the original corpus; mapping failures’ share fell as fewer gold documents were retrieved.In no-gold cases, the retriever never surfaced the gold document, and the agent cited topically related English negatives instead.Source: G Citation Precision Error Analysis · p. 14
  • 226–272 total errors for GPT-OSS-120B exceeded the 108–172 for GPT-OSS-20B and QWEN3.6-35B-A3B, alongside higher citation coverage (60.6% vs. 50.4% and 41.5%).More frequent citations created more opportunities for incorrect citations.Source: G Citation Precision Error Analysis · p. 14

H Additional Per-Language Results

This appendix reports per-language results in the cross-lingual setting and defines embedding-model shorthand. The supplied captions identify tool-based accuracy tables for GPT-OSS-20B and GPT-OSS-120B, but provide no values.

H Additional Per-Language Results

  • Q3-4B and Q3-8B denote QWEN3-EMBEDDING-4B and QWEN3-EMBEDDING-8B, respectively.All appendix tables report cross-lingual per-language results.Source: H Additional Per-Language Results · p. 15
  • The appendix includes per-language tool-based accuracy tables for GPT-OSS-20B and GPT-OSS-120B, with values expressed as percentages.The supplied captions do not expose the per-language accuracy values.Source: Table 13 · p. 15, Table 14 · p. 15

I Tongyi-DeepResearch Results · J Inference Hyperparameters

Tongyi achieves the highest reported original-corpus accuracy among agents of comparable parameter counts, but translation reduces its accuracy. Its ReAct-style format prevents reliable per-query confidence reporting, and inference settings vary by model while remaining uniform across corpus conditions and evidence languages.

I Tongyi-DeepResearch Results

J Inference Hyperparameters

  • GPT-OSS-20B and GPT-OSS-120B use local vLLM serving with temperature 1.0 and top-p 1.0, following release-recommended generation configurations uniformly across conditions and languages.Source: J Inference Hyperparameters · p. 16
  • QWEN3.6-35B-A3B uses local vLLM at temperature 0.7 and top-p 0.8; DEEPSEEK-V4-PRO uses its official API defaults of temperature 1.0 and top-p 1.0, with other parameters left at model defaults.Source: J Inference Hyperparameters · p. 16

K License Statement · L GenAI Statement · M Ethics

XBCP is to be released under the MIT License with attribution, while its construction and evaluation use are disclosed alongside limitations from translation artifacts. The authors report no new personal-data collection and intend the benchmark for research evaluation, not deployment-grade safety claims.

K License Statement

  • The models used have Apache 2.0 or MIT licensing, while GPT-5.4 was used for translation and DeepSeek-V4-Pro and GPT-5.4 involve API terms.GPT-OSS-20B and GPT-OSS-120B (OpenAI et al., 2025), QWEN3-EMBEDDING-4B and -8B (Zhang et al., 2025), ARCTIC-EMBED-L-2.0 (Yu et al., 2024), and QWEN3.6-35B-A3B are Apache 2.0; MULTILINGUAL-E5-LARGE (Wang et al., 2024) and DeepSeek-V4-Pro weights (DeepSeek-AI, 2026) are MIT-licensed. DeepSeek-V4-Pro is accessed via its official API under platform terms; proprietary GPT-5.4 is used only for translations under OpenAI’s terms. All model use is for non-commercial academic research.Source: K License Statement · pp. 16–17
  • XBCP will be released under the MIT License with translated corpora, query–language assignments, evaluation scripts, and attribution to BrowseComp-Plus and contributing models.The underlying BrowseComp-Plus benchmark is MIT-licensed, and the authors retain its original copyright notice and license text in derived artifacts.Source: K License Statement · pp. 16–17

L GenAI Statement

  • GPT-5.4 translated BrowseComp-Plus evidence into eleven non-English languages, with sample translations expert-verified using a rubric.The prompt and verification rubric are provided in Appendices B and C, respectively.Source: L GenAI Statement · p. 17
  • The evaluated agents and retrievers are LLM-based or neural systems whose use is the subject of study and is described in Section 4.The agents include GPT-OSS-20B, GPT-OSS-120B, QWEN3.6-35B-A3B, and DEEPSEEK-V4-PRO, alongside four multilingual embedding models.Source: L GenAI Statement · p. 17
  • Claude and ChatGPT provided surface-level writing assistance; human authors verified scientific claims, analyses, and conclusions and take responsibility for the paper.AI assistants were not used to generate citations, statistical results, or reported factual content; the disclosure follows the ACL AI Writing Assistance policy.Source: L GenAI Statement · p. 17

M Ethics

  • Residual GPT-5.4 translation artifacts may skew low-resource-language results; XBCP collects no new personal data and is intended for research evaluation, not deployment-grade safety claims.The benchmark derives from BrowseComp-Plus, which uses publicly available web documents; expert verification covers samples, not necessarily every translation.Source: M Ethics · p. 17
  • Expert bilingual annotators were recruited through commercial language-service companies and compensated at standard professional translation-evaluation rates.They assessed samples from all eleven non-English languages using the stated rubric.Source: M Ethics · p. 17, L GenAI Statement · p. 17

Related papers

Loading 2606.15345v2…