PaperlayerView exact revision on arXiv

Source-linked AI summary

K-BrowseComp: A Web Browsing Agent Benchmark Grounded in Korean Contexts

Nahyun Lee, Dongkeun Yoon, Guijin Son, Geewook Kim, Dayoon Ko, Jeonghun Park, Haneul Yoo, Jaewon Cho, Junghun Park, Changyoon Lee, Kyochul Jang, Jaeyeon Kim, Eunsu Kim, Woojin Cho, Seungone Kim

arXiv:2606.02404v1cs.CL

TL;DR

Korean agentic benchmarks are scarce, limiting standardized evaluation of web-browsing agents in Korean contexts. K-BrowseComp addresses this gap with verified and synthetic diagnostic splits, finding low performance even among strong frontier models.

  • Problem

    Korean agentic benchmarks are virtually nonexistent, leaving no standardized way to measure progress in linguistically and culturally distinct contexts.

  • Method

    K-BrowseComp combines 300 manually crafted, native-speaker-validated problems with 100 synthetic diagnostic problems generated using hard exemplars and targeted failure modes.

  • Results

    GPT-5.5 and DeepSeek-V4-Pro achieve only 45.67% and 30.00%, respectively, on K-BrowseComp-Verified, while synthetic problems yield 0.00–26.00% accuracy.

  • Takeaways & Limitations

    K-BrowseComp provides a challenging target for reliable Korean web-browsing agents, highlighting the need for stronger trajectory-level state maintenance.

  • Takeaways & Limitations

    The verified set is modest and unevenly distributed across domains, while evaluation uses a single browsing harness, backend, budget, and pass@1 setting.

Abstract

from arXiv · show

Frontier model evaluations are shifting from foundational capabilities (e.g., instruction following and reasoning) toward compositional, agentic ones, but Korean agentic benchmarks remain scarce. We introduce K-BrowseComp, a web-browsing agent benchmark grounded in Korean contexts, consisting of 400 problems. The 300-problem K-BrowseComp-Verified subset is manually constructed and validated by native Korean speakers. On this subset, frontier LLMs, including GPT-5.5, DeepSeek-V4-Pro, and GLM-5.1, reach only 30.00--45.67\%, a substantial drop from BrowseComp, while Korean LLMs released through Korea's Proprietary AI Foundation Model program obtain only 0.00--10.33\%. We further construct a 100-problem synthetic split using hard few-shot exemplars and failure-mode-targeted generation to exploit the asymmetry between solving and creating web browsing problems. On the adversarially filtered synthetic diagnostic split, the strongest model reaches only 26.00\%, and we report this split separately as a targeted stress test. We publicly release our data and code.

1 Introduction

K-BROWSECOMP addresses the scarcity of Korean agentic benchmarks with 400 Korean-context browsing problems, including manually validated and synthetic diagnostic splits. Frontier models perform substantially worse on its verified subset, while targeted synthetic construction yields a difficult stress test.

  • Motivation: Korean agentic benchmarks remain virtually nonexistent as evaluation shifts from foundational capabilities toward compositional agentic evaluation.
  • Benchmark rationale: K-BROWSECOMP grounds browsing-agent evaluation in Korean contexts, where local and cultural knowledge is especially consequential and multiple agentic capabilities are jointly exercised.
  • Benchmark composition: 400 problems comprise 300 manually crafted, native-Korean-speaker-validated K-BROWSECOMP-VERIFIED problems and a 100-problem SYNTHETIC diagnostic split.
  • Verified results: 45.67% and 30.00% are achieved by GPT-5.5 and DeepSeek-V4-Pro, respectively, on K-BROWSECOMP-VERIFIED, where strongest evaluated models reach only 30.00–45.67%.These results represent a substantial drop from the models’ reported performance on the original BrowseComp.
  • Synthetic construction: 0.00–26.00% accuracy is achieved on the 100 machine-generated SYNTHETIC problems after hard few-shot exemplars or failure-mode-targeted generation improve problem quality and difficulty.Naive generation produced problems that were either solvable by frontier models or ill-defined; the corresponding verified subset spans 0.00–45.67%.

2 Related Work

This section situates K-BrowseComp within the shift from fixed-source or short-turn retrieval benchmarks toward long-horizon web-browsing agents, while highlighting unmet needs in Korean agentic evaluation and synthetic benchmark quality.

  • Web browsing agents: Earlier open-domain QA work established evidence retrieval and reasoning, but typically used predefined evidence sources or fixed numbers of short turns.Recent benchmarks instead evaluate tool-using agents over longer horizons and multiple websites.
  • Web browsing agents: BrowseComp introduced questions that humans with browsers could not solve within 2 hours, and BrowseComp-ZH extended this paradigm to multiple Chinese websites.
  • Korean and regional language evaluation: Korean benchmarks mainly assess static language understanding, factual knowledge, reasoning, cultural intelligence, or locally grounded norms rather than web-searching agents.Existing resources generally do not require maintaining evidence state or synthesizing information across pages.
  • Korean and regional language evaluation: K-BROWSECOMP fills this gap by grounding agentic evaluation in Korean cultural, institutional, educational, geographic, and media knowledge.
  • Synthetic task generation: LLM-based synthetic task generation can scale evaluation data, but generated benchmarks may be underspecified, too easy, or contaminated by highly indexed sources.

3 K-BROWSECOMP

K-BrowseComp evaluates web-browsing agents on difficult, publicly verifiable questions grounded in Korean contexts, using multi-hop and parallel-branching reasoning formats. Its verified and synthetic subsets combine manual validation with failure-mode-targeted generation to stress distinct browsing weaknesses.

  • Benchmark design: Questions require hard-to-find Korean-context information, a single temporally stable answer, and either multi-hop reasoning or parallel constraint satisfaction.Parallel-branching questions intersect independent constraints, whereas multi-hop questions use an intermediate finding.
  • Verified subset: All finalized items are manually checked for recoverable public evidence, natural wording, temporal stability, answer uniqueness, and consistency among the problem, answer, trajectory, sources, and checklist.Items with inaccessible, insufficient, or inconsistent evidence are returned for revision.
  • Verified subset: 300 verified questions comprise 160 multi-hop items (53.3%) and 140 parallel-branching items (46.7%).Entertainment and Media is the largest category with 109 questions (36.33%), followed by Transportation, Places, and Regions with 48 (16.00%).
  • Synthetic subset: The benchmark’s trajectory-level failure taxonomy is built from manually inspected recurring model errors across search, evidence retrieval, metadata interpretation, information combination, and answer derivation.The taxonomy supplies failure modes for constructing the synthetic subset.
  • Synthetic subset: 100 of 268 generated synthetic candidates survive filtering, yielding 37.3%.Accepted questions are produced across four iterations, with 55, 32, 10, and 3 accepted on iterations one through four; semi-structured parsing and constraint accumulation are the most frequently exploited weaknesses.
  • Synthetic subset: The synthetic split preserves the reasoning-format balance, with 53.0% multi-hop and 47.0% parallel examples, but shifts category composition toward Science, IT, and Academia.Entertainment and Media declines from 36.3% to 9.0%, while Science, IT, and Academia rises from 6.7% to 33.0%; synthetic questions are also longer on average.

4 Experimental Setup

The evaluation compares proprietary, open-weight, and Korean open-weight models on K-BROWSECOMP-VERIFIED using a standardized browsing-agent protocol. It also reports pass@1, calibration error, and a separate synthetic diagnostic accuracy under a shared retrieval pipeline.

  • Baselines: The baselines span proprietary models, open-weight models, and Korean open-weight models.Proprietary baselines include GPT-5.5, GPT-5.4-mini, and Gemini-3.1-Flash-Lite; open-weight baselines include DeepSeek-V4-Pro, GLM-5.1, Qwen3.6-35B-A3B, and Gemma-4-31B-it; Korean models include K-EXAONE-236B-A23B, A.X-4.0, HyperCLOVAX-SEED-Think-32B, and Kanana-2-30B-A3B-Thinking-2601.
  • Evaluation protocol: 300 questions are evaluated once per model on the verified subset, with each agent receiving a budget of 10 search calls per question.The evaluation uses the search_evals framework, the deepresearch agent, and Perplexity Search as the backend.
  • Evaluation protocol: Pass@1 accuracy (%) and calibration error (%) are computed on the 300-question K-BROWSECOMP-VERIFIED subset.Single-run accuracy corresponds to pass@1 in this setting.
  • Evaluation protocol: SYNTHETIC accuracy is computed separately on the 100-question diagnostic synthetic split and is not pooled with the verified score.All runs use the same external retrieval pipeline with the Perplexity Search API.

5 Experimental Results

K-BROWSECOMP-VERIFIED remains challenging: GPT-5.5 leads at 45.67%, while other frontier and Korean open models score substantially lower. On the separate 100-question SYNTHETIC split, accuracy reaches at most 26.00%, below verified-subset performance for the strongest models.

  • K-BROWSECOMP-VERIFIED: 45.67% is GPT-5.5’s highest accuracy on K-BROWSECOMP-VERIFIED, while GPT-5.4-mini and GLM-5.1 reach 30.67% and DeepSeek-V4-Pro 30.00%.These results indicate that the benchmark remains challenging even for recent high-performing models.
  • K-BROWSECOMP-VERIFIED: 23.33% is Gemma-4-31B-IT’s score, ahead of Qwen3.6-35B-A3B at 12.00%, while Korean open models range from 2.33% to 10.33%.The Korean open-model scores are K-EXAONE-236B-A23B at 10.33%, A.X-4.0 at 5.33%, and HyperCLOVAX-SEED-Think-32B at 2.33%.
  • Calibration: Expected calibration error is computed over five equal-width confidence bins as the weighted average gap between mean confidence and empirical accuracy, with lower values indicating better alignment.The metric follows BrowseComp and BrowseComp-ZH.
  • SYNTHETIC: 26.00% is the strongest accuracy on the 100-question SYNTHETIC split, with evaluated models ranging from 0.00% to 26.00% and none exceeding 30.00%.The split uses the same browsing-agent harness and grading protocol as Section 4.
  • SYNTHETIC: 26.00%, 22.00%, and 19.00% are the SYNTHETIC scores for GPT-5.5, DeepSeek-V4-Pro, and GLM-5.1, respectively, all below their verified-subset accuracy.The results are treated as a diagnostic stress test and are not pooled with the verified score; GPT-5.4-mini is reported separately because it was used during adversarial filtering.

6 Analysis

The analysis finds that browsing failures frequently arise after relevant evidence has been retrieved, when models lose track of candidates, constraints, entity roles, or final-answer state. Search-call comparisons and model-level diagnoses indicate stage-specific bottlenecks rather than insufficient retrieval alone, including post-retrieval state maintenance, tool-use protocol, and answer finalization.

  • Trajectory-level failure dynamics: Post-retrieval errors compose across steps when models lose track of candidates, constraints, and evidence after finding partially relevant information.The recurring trajectory-level patterns include candidate capture, unmerged evidence branches, and misbound evidence chains.
  • Trajectory-level failure dynamics: Ladies’ Code satisfies the intersected constraints, but the model returns Winner after applying separate clues to different groups and violating the album-release constraint.This exemplifies unmerged evidence branches: queries remain separate instead of becoming filters over a shared candidate set.
  • Trajectory-level failure dynamics: F0, F3, F5, and F7 are the dominant failure modes once retrieval succeeds, motivating stronger maintenance of candidates, constraints, role bindings, and final-answer state.The analysis also identifies controlled finalization as necessary because some models produce incomplete trajectories, unstable conclusions, or malformed answers.
  • Search-call analysis: 7.08 vs. 9.30 search calls for GPT-5.5, 7.47 vs. 9.80 for DeepSeek-V4-Pro, 5.20 vs. 8.10 for Gemma-4-31B-it, and 8.11 vs. 9.69 for Qwen3.6-35B-A3B compare correct and incorrect trials.Incorrect trials generally use more searches and often approach the 10-call budget, so failures are not explained simply by insufficient search-call allocation.
  • Model-level diagnosis: Korean open-weight models exhibit distinct stage-specific bottlenecks: A.X-4.0 commits early, K-EXAONE-236B-A23B loses entity chains, HyperCLOVAX-SEED-Think-32B fails at finalization, and Kanana-2-30B-A3B-Thinking-2601 violates the tool protocol.The findings show no clear Korean-centric advantage on K-BROWSECOMP and a substantial gap to closed frontier APIs.
  • Synthetic split analysis: The SYNTHETIC split remains difficult because models reach the correct source family or intermediate entity but fail to preserve page-level or candidate-level state long enough to extract exact values or enforce comparisons.Accepted synthetic questions emphasize semi-structured parsing (F4) and constraint tracking (F7), which also reappear as solving failures.

7 Conclusion

K-BROWSECOMP introduces a Korean web-browsing agent benchmark with human-verified and synthetic splits. Its results and failure analysis show that Korean browsing progress requires stronger trajectory-level state maintenance, beyond broader language coverage.

  • Benchmark: K-BROWSECOMP comprises a 300-question human-verified subset and a 100-question synthetic split.The benchmark is designed for Korean web-browsing agents.
  • Results: Strong frontier models achieve low scores, while Korean open-weight models lag substantially behind global counterparts.
  • Diagnostic split: The synthetic split remains similarly difficult, showing that failure-mode-targeted generation with verification and filtering yields useful diagnostic items.
  • Failure analysis: Many failures arise after retrieving relevant Korean web evidence, when models lose candidates, constraints, source pointers, or final-answer state across the trajectory.
  • Implications: Korean browsing progress therefore requires stronger trajectory-level state maintenance, not only broader language coverage.

Limitations · A Dataset Construction · B Dataset Construction Details

K-BROWSECOMP-VERIFIED is a manually constructed benchmark of 300 Korean-context browsing problems, developed under explicit difficulty, evidence, uniqueness, and review criteria. Its modest scale, uneven domain coverage, and evaluation under a single browsing setup limit how broadly its results generalize, while the released benchmark artifacts and code support further research.

  • Limitations: 300 human-written items remain modest in scale and are unevenly distributed, with substantial concentration in entertainment/media and place-related queries.This may not represent the full breadth of Korean web-search use cases.
  • Limitations: A single browsing harness and search backend constrain conclusions about performance across alternative browsing and search configurations.The supplied passage identifies single-harness measurement as a limitation, although its discussion is truncated.
  • A Dataset Construction: Contributors were instructed to write difficult Korean-context fact-seeking questions requiring web browsing, evidence synthesis, short answers, and public web support.The instructions specified the item format, goal, and exclusion rules for K-BROWSECOMP-VERIFIED.
  • A Dataset Construction: Questions used multi-hop or parallel-branching formats, requiring at least four reasoning steps or four independent constraints, respectively.Multi-hop items recover later evidence from an intermediate entity or value; parallel-branching items identify the unique entity satisfying all constraints.
  • A Dataset Construction: Items required publicly accessible textual evidence, excluded login- or payment-gated sources and non-textual artifacts, and preferably used multiple independent sources.Contributors were discouraged from relying on a single web platform as the sole basis for an item.
  • A Dataset Construction: Each item required a unique final answer, reference dates for time-sensitive facts, structured metadata, and manual review for accessibility, consistency, clarity, ambiguity, stability, and sensitive content.Submissions included the problem, gold answer, expected trajectory, source URLs, checklist values, Korean-specific keywords, and rationale; failing items were revised or removed.
  • A Dataset Construction: The verified and synthetic questions, source URLs, expected trajectories, checklist values, and evaluation code will be released under the MIT License, excluding rights to linked third-party web pages.The license applies to benchmark items, metadata, and code created by the authors and contributors.
  • B Dataset Construction Details: K-BROWSECOMP-VERIFIED consists of 300 human-written Korean browsing problems submitted by Korean-speaking researchers and non-researchers in structured JSON format.The format included the problem statement, gold answer, expected reasoning chain, source URLs, intermediate checklist values, and Korean-specific keywords.

B.1 Category-wise Performance Analysis · C Failure Modes Details · C.1 Candidate Capture Failure Details

K-BROWSECOMP-VERIFIED performance varies sharply by category, with Entertainment & Media and Sports & Games generally strongest while several knowledge-intensive categories remain difficult. The appendix further details candidate capture, in which premature commitment redirects browsing toward locally plausible but globally unsupported answers.

  • B.1 Category-wise Performance Analysis: Performance varies substantially across K-BROWSECOMP-VERIFIED categories, including for the strongest frontier models.Table 4 reports accuracy across major categories ordered by question count.
  • B.1 Category-wise Performance Analysis: Entertainment & Media and Sports & Games are generally highest-performing, whereas Science, IT & Academia, Products, Brands & Beauty, and Education, Colleges & Exams remain difficult.The stronger categories often involve public-web information, while the difficult categories frequently require linking sparse evidence.
  • B.1 Category-wise Performance Analysis: 45.67%: GPT-5.5 achieves the strongest overall performance, while K-EXAONE-236B-A23B and A.X-4.0 remain below 11%.The gap persists even in categories containing strongly Korean-specific cultural or institutional information.
  • B.1 Category-wise Performance Analysis: History, Culture & Politics shows high variance partly because it contains only ten questions and some items become recoverable once the correct evidence chain is identified.Science, IT & Academia remains consistently difficult because many questions require cross-source reasoning.
  • C Failure Modes Details: The appendix expands on three trajectory-level failure patterns: candidate capture, unmerged evidence branches, and misbound evidence chains.Examples show that failures can occur after partial retrieval has already succeeded, when the model fails to preserve candidate and evidence relationships.
  • C.1 Candidate Capture Failure Details: Candidate capture occurs when a model commits to a plausible candidate before fully verifying upstream constraints, then searches within that candidate’s local evidence space.The resulting answer may appear evidence-supported even though the candidate is not licensed by the full question.
  • C.1 Candidate Capture Failure Details: In the Figure 11(b) example, the model searches Olive Young award products before fixing the upstream company and brand.The question requires preserving the dependency chain from company to brand, H&B store, and award product.
  • C.1 Candidate Capture Failure Details: The resulting CLIO/Goodal/Peripera branch yields locally plausible products but not the unique gold answer, 어성초흔적에센 스패드(Abib Heartleaf Spot Pad Calming Touch).The products are unsupported by the required company–brand constraint.

D SYNTHETIC Split Diagnostics

The 100-problem SYNTHETIC split is distinguishable from K-BROWSECOMP-VERIFIED using question text, with separation driven largely by question length and category rebalancing. Reasoning format contributes little to the observed separation.

  • Embedding-based diagnostics: ROC AUC 0.8873 ± 0.0281 shows that a domain classifier separates the two splits from question text alone.The classifier uses 384-dimensional sentence-transformer embeddings and cross-validation.
  • Metadata-based diagnostics: ROC AUC 0.7885 ± 0.0558 for question length and 0.7773 ± 0.0342 for category indicate substantial metadata-based separability.Combining length, category, and type reaches ROC AUC 0.8277 ± 0.0275, while the multi-hop or parallel label alone reaches ROC AUC 0.4250 ± 0.0197.
  • Sources of separation: The embedding separability is attributed largely to longer SYNTHETIC questions and category rebalancing.These factors explain much of the separation identified by the metadata classifiers.
  • Sources of separation: ROC AUC 0.4250 ± 0.0197 for the multi-hop or parallel label indicates that reasoning format contributes little to split separation.This is consistent with near-identical multi-hop and parallel proportions across the two splits.

E Trajectory-Level Failure Diagnostics for Korean Open-Weight Models · F SYNTHETIC Split Trajectory Diagnostics · G AI assistants in research/writing

The trajectory diagnostics show that Korean open-weight models and SYNTHETIC-split failures arise at distinct stages of persistent browsing-state construction, from tool-use protocol reliability to evidence organization, entity binding, metadata extraction, and final comparison. The SYNTHETIC split provides a complementary diagnostic stress test, while AI assistants were used mainly for grammar and fluency revision.

  • E Trajectory-Level Failure Diagnostics for Korean Open-Weight Models: Korean open-weight models fail at different browsing-loop bottlenecks rather than a single failure type, including state formation, dependency-chain preservation, trajectory completion, and tool-call validity.Successful browsing requires persistent candidate ledgers, constraint checks, entity-role bindings, and final-answer commitments.
  • E Trajectory-Level Failure Diagnostics for Korean Open-Weight Models: A.X-4.0 reaches relevant Korean historical evidence but fails to maintain a candidate ledger for jointly checking multiple constraints.The case demonstrates shallow snippet-level evidence control: retrieved snippets are not converted into a shared candidate table supporting elimination and verification.
  • E Trajectory-Level Failure Diagnostics for Korean Open-Weight Models: K-EXAONE-236B-A23B initially retrieves the correct web region but loses the event–emoticon–creator–channel dependency chain through premature creator search and unstable intermediate entities.Later plausible creator and channel findings cannot verify the cross-source link after misspelled and unrelated queries destabilize the target.
  • F SYNTHETIC Split Trajectory Diagnostics: The SYNTHETIC split targets reliably generated failure modes, with semi-structured parsing (F4) and constraint tracking (F7) appearing most frequently among accepted items.Its items make answers easy to verify once the correct state is fixed, but difficult when that state must persist across browsing.
  • F SYNTHETIC Split Trajectory Diagnostics: 1.56 MB was selected instead of the gold value 698.85 kB when the model localized the correct repository context but read metadata from a neighboring PDF record.This is a page-level state failure involving the exact repository item and file-size field.
  • F SYNTHETIC Split Trajectory Diagnostics: 05.14 was answered instead of the gold answer 05.08 because the model recovered An Woo-jin but failed to maintain a game-level ledger and verify the maximum opponent AVG across appearances.The required comparison spans rows containing each 2026 game’s date, appearance, and opponent AVG values.
  • F SYNTHETIC Split Trajectory Diagnostics: The SYNTHETIC split is a complementary diagnostic stress split rather than a replacement for K-BROWSECOMP-VERIFIED, exposing persistent-state failures in exact extraction and final selection.The repository and baseball examples show failures at source-item and candidate-record granularities despite relevant evidence retrieval.

H Computational experiments

The experiments evaluate disclosed open-weight models spanning dense and mixture-of-experts architectures, alongside closed-weight baselines whose parameter counts are undisclosed. They use OpenRouter API calls where supported, with approximately 320 USD in total API costs, and document representative browsing failures and structured submission formats.

  • Model Size: GPT-5.5, GPT-5.4-mini, and Gemini-3.1-Flash-Lite are closed-weight baselines whose parameter counts are not publicly disclosed by their providers.
  • Compute Budget: Approximately 320 USD was spent on API calls across the project, with experiments using OpenRouter except for unsupported Korean models.
  • Examples and Failure Analysis: K-BROWSECOMP-VERIFIED submissions use structured JSON containing the problem, gold answer, expected reasoning trajectory, intermediate checklist values, and Korean-specific keywords.The example concerns a Korean multi-hop literature question and includes an English translation.
  • Examples and Failure Analysis: Representative failures span ineffective search direction, cross-source linking, semi-structured parsing, search-result selection, entity normalization, constraint tracking, and intermediate reasoning.Examples include returning 1.56 MB instead of the gold 698.85 kB after reaching the correct repository neighborhood, and losing dependencies across multi-hop evidence chains.
Loading 2606.02404v1…