PaperlayerView exact revision on arXiv

Source-linked AI summary

Beyond Monolingual Deep Research: Evaluating Agents and Retrievers with Cross-Lingual BrowseComp-Plus

Yuheng Lu, Qingcheng Zeng, Heli Qi, Puxuan Yu, Fuheng Zhao, Rui Yang, Hitomi Yanaka, Naoto Yokoya, Weihao Xuan

arXiv:2606.15345v2cs.CLcs.IR

TL;DR

Existing browsing benchmarks rarely test English questions answered from evidence in other languages. XBCP varies evidence languages while preserving the original task, finding that cross-linguality reduces retrieval and citation reliability and creates an additional evidence-integration challenge.

  • Problem

    Existing browsing benchmarks largely assume questions and supporting evidence share a language, leaving cross-lingual deep research insufficiently measurable.

  • Method

    XBCP preserves English questions and answers while varying evidence languages in controlled cross-lingual and multilingual configurations across 12 languages.

  • Results

    Translated corpora reduce evidence recall, increase search effort, and lower citation reliability, while translated oracle accuracy remains below the original.

  • Takeaways & Limitations

    Cross-lingual deep research requires evaluating both whether agents find language-mismatched evidence and whether they use it faithfully once found.

  • Takeaways & Limitations

    Each agent–retriever–corpus configuration was evaluated in a single run without multiple random seeds, leaving variance estimates and significance tests for future work.

Abstract

from arXiv · show

Deep research agents are increasingly evaluated on their ability to search for evidence, reason over retrieved sources, and produce grounded answers. Existing browsing benchmarks, however, largely assume that the user's query and the supporting evidence are written in the same language, leaving open whether agentic search systems can operate when relevant evidence appears in another language. We introduce XBCP (Cross-lingual BrowseComp-Plus), a controlled benchmark that preserves the English question-and-answer space of BrowseComp-Plus but varies the languages of the supporting documents. XBCP instantiates two complementary settings: in the cross-lingual setting, each query is paired with evidence in a single assigned language. In the multilingual setting, the full evidence corpus is distributed equally and randomly across 12 languages spanning high-resource and low-resource regimes. We evaluate four deep research agents using sparse and dense multilingual retrievers, measuring answer accuracy, evidence recall, search behavior, calibration, citation fidelity, and oracle retrieval. Results reveal substantial degradation when evidence is translated. Even strong, dense retrievers lose evidence recall, and agents become less calibrated and cite evidence less reliably. Notably, accuracy remains lower even when all gold evidence is supplied directly. These findings suggest that cross-lingual deep research exposes both retrieval failures and an independent, agent-side difficulty in integrating language-mismatched evidence.

1 Introduction

Deep research agents actively search, assess, and synthesize external evidence rather than relying solely on parametric knowledge. Existing largely monolingual benchmarks leave unanswered whether agents can retrieve and integrate evidence written in another language, motivating a cross-lingual BrowseComp-Plus extension.

  • LLM agents shift answering from parametric knowledge alone toward actively acquiring, filtering, and synthesizing external evidence.
  • Deep research agents plan searches, inspect retrieved sources, assess evidence sufficiency, and compose grounded answers for complex information needs.
  • Existing beyond-English browsing benchmarks remain primarily monolingual, with questions, evidence, and answers staying within the same language.
  • A cross-lingual BrowseComp-Plus benchmark would test multilingual evidence retrieval and agents’ ability to integrate language-mismatched evidence.

2 Related Works

Deep research systems extend tool-augmented LLMs into long-horizon search, evidence verification, and grounded synthesis, while multilingual retrieval advances through shared embedding spaces. Existing retrieval work offers strong models and ranking evaluations but does not examine cross-lingual retrieval within deep research agents’ iterative search and answer-synthesis loop.

  • Deep Research Systems: Deep research systems require agents to plan searches, interact with external sources, verify intermediate evidence, and synthesize grounded answers.OpenAI Deep Research exemplifies this paradigm, which has motivated a growing line of open research agents.
  • Multilingual and Cross-lingual Retrieval: Multilingual and cross-lingual retrieval has shifted from translation-mediated CLIR toward shared embedding spaces.ME5 uses billion-scale multilingual contrastive pre-training and supervised fine-tuning; MGTE and ARCTIC-EMBED 2.0 expand long-context and efficiency-aware embedding designs.
  • Multilingual and Cross-lingual Retrieval: These retrieval works provide strong retrievers and ranking-oriented evaluations but do not study cross-lingual retrieval inside deep research agents’ iterative search, evidence selection, and answer-synthesis loop.Prior work also finds that multilingual and cross-lingual retrieval can yield different model rankings and gains from supervised adaptation.

3 Building XBCP

XBCP preserves BrowseComp-Plus’s English questions and answers while translating only the evidence documents into 12 languages. It evaluates both query-level single-language evidence and randomly balanced multilingual evidence, with expert checks indicating generally high translation quality.

  • Dataset construction: XBCP keeps questions and final answers in English while translating evidence documents into target languages with a prompt that preserves URLs, formulas, and code blocks.The translation also requests complete coverage of titles, terminology, proper nouns, and metadata field names.
  • Evaluation configurations: The cross-lingual configuration assigns each query one evidence language, so all supporting documents appear in that language, with English as an untranslated reference.This configuration contains 830 query assignments and 5,040 evidence-document assignments.
  • Evaluation configurations: The multilingual configuration randomly and equally distributes 5,040 evidence-document instances across 12 languages, yielding 420 evidence documents per language.The passage describes this as a complementary evaluation configuration to the query-level language assignment.
  • Translation verification: Expert verification samples 200 translated documents from each of 11 non-English languages, producing 2,200 instances rated for accuracy, fluency, and completeness on a 1–5 scale.The study follows the MMLU-ProX translation-evaluation rubric and compares each translation with its original English document.
  • Translation verification: All language-level mean verification scores exceed 4.0, suggesting that the translated evidence generally preserves the quality needed for retrieval and answer synthesis.The verification focuses on whether translations preserve evidence needed for retrieval and answer synthesis.

4 Experiments and Results

Experiments show that translating supporting evidence substantially harms retrieval, answer accuracy, calibration, and citation fidelity, with language mismatch creating both retriever-side and agent-side bottlenecks. Reasoning-based expansion and greater effort improve performance, but neither fully restores monolingual results.

  • Evaluation setup: The evaluation pairs four agents with BM25 and four dense multilingual retrievers across original, multilingual, and cross-lingual evidence corpora.Metrics include answer accuracy, evidence recall, search calls, calibration error, and citation fidelity.
  • End-to-end results: 64.70% accuracy on the original corpus, 48.80% multilingual accuracy, and 42.29% cross-lingual accuracy are achieved by DEEPSEEK-V4-PRO with QWEN3-EMBEDDING-8B.With this retriever, translated evidence lowers accuracy by roughly 16–23 pp across agents, and multilingual and cross-lingual results are close across most pairs.
  • Retrieval and behavior: Translated evidence causes agents to search more without recovering lost accuracy, while calibration error increases in both translated settings.QWEN3-EMBEDDING-8B retrieves the most supporting evidence, whereas BM25 drops sharply because lexical matching poorly suits English queries over non-English documents.
  • Citation and oracle analysis: Citation coverage, precision, and recall all decline after evidence translation, showing that language mismatch also disrupts faithful attribution of retrieved sources.The oracle comparison further shows retrieval/search gaps exceeding 55 pp on the original corpus and roughly 65–75 pp after translation.
  • Language-resource effects: High-resource languages average 18.39% tool accuracy and 28.48% evidence recall, versus 10.87% and 18.00% for low-resource languages, while oracle accuracies remain 89.67% and 87.32%.These results attribute much of the low-resource penalty to retrieval failure, although language-specific variation remains within resource groups.
  • Reasoning and effort: Higher reasoning effort improves accuracy and evidence recall but raises search cost: original-corpus accuracy increases from 15.18% to 36.02%, with high effort requiring over 26 calls versus roughly 2.Reasoning-based query expansion also improves original-corpus accuracy by 3.25 pp and evidence recall by 4.86 pp while reducing search calls, but translated-evidence gains are smaller.

5 Discussion

The discussion identifies cross-linguality as a structural challenge spanning retrieval and evidence integration, rather than merely a first-stage retrieval perturbation. XBCP shows that agents must both find language-mismatched evidence and use it faithfully once found, motivating language-aware agentic search.

  • Two cross-lingual bottlenecks: Cross-linguality creates linked retrieval and evidence-integration bottlenecks throughout the deep-research pipeline.By varying only evidence language, XBCP isolates how language mismatch propagates through evidence seeking.
  • Retrieval and search: Dense multilingual retrievers outperform BM25 after translation, but translated corpora still reduce evidence recall, increase search effort, and lower citation reliability.These degradations persist even when the retriever is dense, showing that conventional retrieval success does not guarantee successful iterative evidence finding.
  • Evidence integration: Translated oracle accuracy remains below original oracle accuracy, indicating that language-mismatched evidence remains difficult to integrate after retrieval.Providing all gold evidence substantially raises accuracy, confirming that finding evidence is a major bottleneck, while the residual gap reflects integration difficulty.
  • Evidence integration: Faithful cross-lingual synthesis requires identifying relevant facts in non-English sources, aligning them with an English question-and-answer space, and preserving evidential constraints.The discussion frames these requirements as a distinct agent-side challenge beyond evidence retrieval.
  • Implications: The findings motivate language-aware agentic search rather than simply stronger multilingual retrieval.This direction emphasizes dynamically deciding when and what to retrieve and improving cross-lingual representation alignment.

Limitations … C Translation Verification Rubrics

The paper notes limitations from single-run evaluation and fixed inference settings, then documents XBCP’s corpus construction, translation procedure, and expert verification rubric. Translations are evaluated for accuracy, fluency, and completeness on 1-to-5 scales.

  • Limitations: Each agent–retriever–corpus configuration is evaluated once, without multiple random seeds, leaving variance estimates and significance tests for future work.The authors report that comparisons remain large and consistent across agents, but repeated-run statistical analysis is computationally expensive and omitted.
  • Limitations: Agents use one recommended inference configuration per model without temperature or top-p tuning, which may leave condition-specific gains, especially for low-resource languages, unexplored.The fixed settings control comparisons, while systematic inference-configuration analysis is outside the study’s scope.
  • A XBCP Construction Details: In the multilingual setting, 420 English source documents remain unchanged, while 4,620 remaining document instances are produced by translation.These statistics describe corpus coverage for the multilingual construction.
  • B Translation Prompt: The translation prompt requires complete translation into the target language, including proper nouns, titles, terminology, and metadata field names, while preserving URLs, emails, formulas, and code.It also requests cultural appropriateness, established translations for known titles, and output without explanations.
  • C Translation Verification Rubrics: Expert bilingual evaluators compare source and machine-translated documents, scoring accuracy, fluency, and completeness from 1 to 5 with justification for scores below 5.The rubric follows MMLU-ProX’s verification approach.
  • C Translation Verification Rubrics: The fluency rubric ranges from highly fluent, natural target-language expression to unreadable output with severe grammatical mistakes.Intermediate criteria include stiffness, inappropriate word choices, awkward transitions, and structural or grammatical errors.
  • C Translation Verification Rubrics: The completeness rubric ranges from retaining all meaning, details, data, annotations, length, and depth to significant omissions or additions that obscure the intended meaning.Intermediate scores allow minor omissions, missing important details, or content requiring readers to infer some information.

D Translation Verification Results · E Judge Prompt

Translation quality is verified per language on accuracy, fluency, and completeness over 200 samples, with results averaged on a 1–5 scale. The judge compares extracted answers with English references, allowing translations, transliterations, and small numerical deviations while separately recording reasoning and confidence.

  • D Translation Verification Results: Translation verification evaluates accuracy, fluency, and completeness for each language using 200 samples and reports average values.The evaluation covers three dimensions per language.
  • D Translation Verification Results: Table 10 reports per-language translation verification results on a 1–5 scale.
  • E Judge Prompt: The judge determines whether a response is correct by comparing it with the precise English correct answer.The question, response, and correct answer are supplied as separate fields.
  • E Judge Prompt: Because evidence may be in another language, the extracted answer may also be non-English while the reference answer remains English.The judge must assess whether the answers refer to the same entity or value.
  • E Judge Prompt: The judge extracts the final exact answer from the response and outputs “None” when no exact final answer is available.
  • E Judge Prompt: The reasoning field explains correctness only by checking whether the extracted and correct answers denote the same entity or value.Different-language answers are assessed as possible translations or transliterations, without solving the underlying problem.
  • E Judge Prompt: The judge outputs “yes” for matching answers, translations or transliterations, or numerical answers within a small error margin; otherwise it outputs “no.”Inconsistency, ambiguity, non-equivalency, or incorrectness leads to “no.”
  • E Judge Prompt: The judge extracts a confidence score between 0% and 100%, assigning 100 when the response provides no confidence score.

F Decomposing the Agent Cross-lingual Bottleneck

Oracle experiments reveal an agent-side cross-lingual bottleneck: gold evidence does not fully restore monolingual performance. Fully translating the prompt, query, and evidence worsens performance further, showing that evidence understanding and language configuration contribute separately.

  • Motivation: Gold evidence does not fully recover monolingual performance, revealing an agent-side cross-lingual bottleneck.The experiments motivate separating evidence reasoning from language switching between an English prompt and non-English content.
  • Experimental variant: ORACLE-TQ+TP presents the system prompt, query, and evidence documents entirely in the target language, removing language switching.This tests whether a monolingual non-English environment improves reasoning.
  • Results: 5.92 pp lower GPT-OSS-20B accuracy and 5.38 pp lower GPT-OSS-120B accuracy occur with Oracle-tq+tp versus standard Oracle.The agent reasons less effectively when the prompt is also in the target language, despite eliminating language switching.
  • Bottleneck decomposition: 12.77 pp is the GPT-OSS-20B evidence understanding loss from EN Oracle to Oracle.The passage identifies this as one component of the agent’s crosslingual weakness.

G Citation Precision Error Analysis

GPT-OSS-120B’s citation precision drops by roughly half under translated evidence, driven mainly by searches that retrieve no gold document. Its higher citation coverage also creates more opportunities for incorrect citations than smaller or alternative agents.

  • Citation Precision Error Analysis: GPT-OSS-120B’s citation precision falls from 50.89% on the original corpus to 24.30% multilingual and 26.26% cross-lingual, a reduction of roughly 50%.This is the steepest citation precision drop among all evaluated agents.
  • Citation Precision Error Analysis: Mapping failures account for the remaining 33.87–42.92% of GPT-OSS-120B’s errors and decline as a share after translation.A mapping failure occurs when the agent retrieves at least one gold document but cites other documents instead.
  • Citation Precision Error Analysis: No-gold-retrieved errors dominate GPT-OSS-120B’s citation failures, rising from 57.08% on the original corpus to 66.13% on the multilingual corpus.The retriever never surfaced the gold document, so the agent cited topically related English negative documents instead.
  • Citation Precision Error Analysis: GPT-OSS-120B makes 226–272 total errors versus 108–172 for GPT-OSS-20B and QWEN3.6-35B-A3B, partly because its citation coverage is higher.Citation coverage is 60.6% for GPT-OSS-120B, versus 50.4% and 41.5% for the comparison agents.

H Additional Per-Language Results

This appendix reports per-language results for the cross-lingual setting, including tool-based accuracy tables for GPT-OSS-20B and GPT-OSS-120B. It also defines shorthand for the QWEN3 multilingual embedding models.

  • All appendix tables report per-language results in the cross-lingual setting.
  • Q3-4B and Q3-8B denote QWEN3-EMBEDDING-4B and QWEN3-EMBEDDING-8B, respectively.
  • Table 13 reports per-language tool-based accuracy for GPT-OSS-20B, with all values presented as percentages.
  • Table 14 reports per-language tool-based accuracy for GPT-OSS-120B, with all values presented as percentages.

I Tongyi-DeepResearch Results · J Inference Hyperparameters

Tongyi-DeepResearch performs strongly with QWEN3-EMBEDDING-8B but degrades after evidence translation. Its ReAct-style interface also prevents reliable calibration measurement, while inference settings follow each model’s release recommendations.

  • I Tongyi-DeepResearch Results: Tongyi-DeepResearch-30B-A3B uses a Qwen3-based MoE architecture and an in-band ReAct-style tool-calling protocol with <tool_call> XML tags.
  • I Tongyi-DeepResearch Results: Tongyi’s ReAct-style output format does not reliably produce per-query confidence scores despite prompt-level instructions, so calibration error is excluded from the main results.
  • I Tongyi-DeepResearch Results: 39.64% accuracy on the original corpus with QWEN3-EMBEDDING-8B is Tongyi’s highest result among agents at comparable parameter counts.
  • I Tongyi-DeepResearch Results: 58.12% evidence recall with QWEN3-EMBEDDING-8B exceeds GPT-OSS-20B (42.91%) and QWEN3.6-35B-A3B (43.14%).
  • I Tongyi-DeepResearch Results: 13.50–14.58 pp accuracy drops after translation with QWEN3-EMBEDDING-8B, compared with 20.84–20.96 pp for GPT-OSS-20B.
  • J Inference Hyperparameters: All corpus conditions and evidence languages use each agent’s recommended generation configuration uniformly.GPT-OSS models use temperature 1.0 and top-p 1.0; QWEN3.6-35B-A3B uses temperature 0.7 and top-p 0.8; DEEPSEEK-V4-PRO uses default API settings.

K License Statement · L GenAI Statement · M Ethics

The paper specifies licensing for XBCP, its models, and release artifacts; it also documents generative-AI use in benchmark construction, experiments, and writing. Ethical disclosures note translation-related validity risks, inherited public-data provenance, and compensated expert annotation.

  • K License Statement: XBCP derives from BrowseComp-Plus and is used under the MIT License, retaining the original copyright notice and license text.
  • K License Statement: The evaluated models are distributed under Apache License 2.0 or MIT License terms, with some accessed locally via vLLM.The passage identifies Apache License 2.0 for GPT-OSS, QWEN3 embedding models, ARCTIC-EMBED-L-2.0, and QWEN3.6-35B-A3B, and MIT License for MULTILINGUAL-E5-LARGE and DEEPSEEK-V4-PRO weights.
  • K License Statement: The planned XBCP release will use the MIT License and include translated corpora, query–language assignments, evaluation scripts, and attribution.
  • L GenAI Statement: GPT-5.4 generated translations of BrowseComp-Plus evidence into eleven non-English target languages, with expert human verification on samples.The exact translation prompt is provided in Appendix B.
  • L GenAI Statement: The evaluated agents and retrievers are LLM-based or neural systems whose use is the subject of study, not an auxiliary writing tool.The systems include GPT-OSS-20B, GPT-OSS-120B, QWEN3.6-35B-A3B, DEEPSEEK-V4-PRO, and four multilingual embedding models.
  • L GenAI Statement: AI assistants supported grammar, rephrasing, and LaTeX formatting, while human authors authored and verified scientific claims, analyses, conclusions, citations, results, and factual content.The authors take full responsibility for the paper, including text initially drafted or edited with AI assistance.
  • M Ethics: GPT-5.4 translation artifacts may persist despite sample-based expert verification, potentially under- or over-estimating performance in low-resource languages.XBCP also inherits BrowseComp-Plus’s basis in publicly available web documents and collects no new personal data from individuals.
  • M Ethics: Expert bilingual annotators were recruited through commercial language-service companies and compensated at standard professional translation-evaluation rates.
Loading 2606.15345v2…