PaperlayerView exact revision on arXiv

Source-linked AI summary

WorldBench: Culturally Grounded Benchmark for Multilingual Agents

Leonardo Ranaldi, Sherrie Shen, Jushi Kai, Alexandra Birch

arXiv:2609.01056v1cs.AIcs.CL

TL;DR

Existing benchmarks rarely combine realistic cultural grounding, multilingual evaluation, and workspace preservation for multi-step agent tasks. WorldBench builds such a benchmark with sandboxed structured actions and final-state evaluation, finding that the strongest model reaches only 49.2% CTS and that agents exhibit preservation and non-English performance gaps.

  • Problem

    Existing benchmarks rarely test state preservation, multilingual performance, and realistic culturally grounded scenarios together for complex agent workflows.

  • Method

    WorldBench provides 1,600 persona-grounded tasks across seven languages and eight cultures, refined by expert annotators and evaluated with deterministic and LLM-based final-state functions.

  • Results

    49.2% CTS is achieved by the strongest model, while all models show preservation gaps and performance degrades outside English.

  • Takeaways & Limitations

    Environment preservation and multilingual robustness are central requirements for reliable agents and should be measured explicitly.

  • Takeaways & Limitations

    The current benchmark excludes visual desktop control and live web interaction to maximize reproducibility.

Abstract

from arXiv · show

Despite the growing use of LLM-powered agents to solve multi-step tasks in complex environments, existing benchmarks rarely test state preservation, performance across languages, and application to realistic, grounded scenarios. To address these concerns, we present WorldBench: a comprehensive, multilingual benchmark of genuine, persona-grounded everyday workflows, where agents can act in a sandbox via structured actions. WorldBench comprises 1,600 tasks across seven languages and eight cultures, filtered and refined through feedback from human annotators with language- and culture-specific expertise. For evaluation, we extend metrics from previous works and introduce Constrained Task Success (CTS), which combines natural language instructions and testbeds to score task completion, minimal modification, and other complementary metrics through deterministic and LLM-as-a-Judge evaluations. Our experiments show that frontier models reach only 49.2% CTS, with all models demonstrating large gaps between correctness and environment preservation. We thereby show that current agents remain brittle in multilingual, agentic scenarios, especially for long-horizon tasks and under state-preservation constraints

1 Introduction

WorldBench addresses gaps in contextual grounding, multilingual robustness, and workspace preservation by benchmarking culturally grounded workflows with structured actions and final-state evaluation. Its experiments show that agents remain brittle when correctness must coexist with preservation and multilingual, long-horizon execution.

  • Motivation: Existing agent benchmarks underexplore contextual grounding, state preservation, and multilingual evaluation in realistic workflows.Real-world tasks depend on personas, locations, languages, working habits, and preservation of unrelated workspace state.
  • Benchmark Design: WorldBench presents a culturally grounded benchmark in which persona-based tasks are executed through structured actions in sandboxed environments and evaluated from final states.Each task combines a natural-language instruction, testbed, and evaluation functions.
  • Benchmark Scope: WorldBench studies multilingual robustness across seven languages, with personas, tasks, documents, calendars, and emails localized to each cultural and linguistic context.The evaluation logic remains consistent across compositionally aligned settings.
  • Results: 49.2% CTS is achieved by the strongest evaluated model, while pass rates exceed CTS and performance shows a language gradient with English leading and Chinese trailing.The results also identify wrong final states, edits, iteration-limit hits, malformed actions, and execution errors as limitations.
  • Construction and Validation: Its construction pipeline expands human-written persona, scenario, and constraint seeds into localized instructions, testbeds, and executable evaluators, followed by expert human validation.Annotators refine or remove tasks based on realism, local appropriateness, solvability, and plausible testbed information.
  • Evaluation: Constrained Task Success combines task correctness with preservation of non-target files, distinguishing valid completion from completion that damages unrelated workspace state.The metric complements intermediate trajectory measures by evaluating the final environment.

2 Related Work

Related benchmarks cover interactive agents, document understanding, synthetic task generation, and multilingual evaluation, but leave culturally grounded multilingual file workflows with workspace preservation insufficiently addressed.

  • Language Agent Benchmarks: Agent benchmarks evaluate planning, tool use, and feedback-driven interaction in settings including web navigation, shopping, operating systems, multi-turn tools, and programming.WorldBench complements these settings with culturally grounded, file-based workflows.
  • Document Understanding and File-Based Workflows: Document understanding benchmarks test extraction from forms, receipts, invoices, tables, and rich documents but generally omit workspace maintenance and preservation during multi-step execution.Recent file-based agent benchmarks remain typically English-centred and lack culturally grounded multilingual construction.
  • Synthetic Benchmark Construction: Synthetic benchmark construction must produce coherent states, solvable instructions, and executable evaluators rather than only instruction-following variants.WorldBench treats each generated task as a complete artefact containing localized text, files, and evaluation functions.
  • Multilingual Agent Evaluation: Multilingual agent benchmarks show degradation beyond English, while translation-based designs can retain source-language scenarios, user assumptions, and artefacts.WorldBench instead constructs settings within target language and cultural contexts.

3 WORLDBENCH Benchmark

WorldBench is a culturally grounded benchmark of sandboxed, structured-action workflows whose persona-localized tasks, heterogeneous artefacts, distractors, and final-state evaluators support multilingual agent evaluation.

  • Benchmark Overview: WorldBench contains 1,600 persona-grounded tasks across seven languages and eight cultures, each pairing an instruction with a sandbox, structured action interface, and final-state evaluators.Human-written seeds, materialized environments, and language- and culture-specific audits support benchmark construction.
  • Task Formulation: Each task models interaction between an LLM agent and a sandboxed environment containing a persona, target-language instruction, state, tools, and final-state criteria.The agent observes outputs, continues through structured actions, and terminates or reaches the iteration cap.
  • Cultural Grounding: The benchmark preserves local variation by constructing personas and tasks within target settings while sharing evaluation logic across languages and locations.Persona attributes determine appropriate activities, artefacts, registers, dates, currencies, organizations, and assumptions.
  • Tool Environment and Workflow: Its heterogeneous tool environment uses restricted operations, requiring agents to select appropriate artefacts, tools, and parameters while enabling execution-error tracking.The workflow covers the main artefact types required by benchmark tasks.
  • Construction Pipeline: WorldBench construction comprises task synthesis, testbed synthesis, and human audit to create executable, heterogeneous, culturally grounded, and auditable tasks.Testbeds populate documents, spreadsheets, PDFs, calendars, and mailboxes, and include 20 distractor artefacts to test precise selection and preservation.
  • Evaluation Protocol: Final-state evaluation combines deterministic checks for directly verifiable properties with LLM judges for open-ended messages and notes.Pass rate requires all task-specific evaluators to succeed, while preservation checks that every non-target file remains unchanged; their conjunction yields CTS.

4 Experiments

WORLDBENCH evaluates agents with Constrained Task Success, requiring both complete task-specific evaluator success and preservation of non-target workspace state. The experiments use nine agents under a shared execution protocol and report complementary outcome and trajectory metrics.

  • Metrics: CTS requires every task-specific evaluator to pass while the preservation constraint holds on the final sandbox state.This distinguishes nominal task completion from completion that leaves unrelated workspace state intact.
  • Metrics: Any modification to a non-target file causes CTS failure, while target-file changes are assessed by task-specific evaluators.Non-target files must remain present and byte-identical between initial and final states.
  • Metrics: The evaluation reports pass rate, preservation rate, solved-task steps, malformed actions, execution failures, iteration-cap hits, and clean termination.It also examines CTS by language and tool family, pass rate by evaluator, and trajectory events linked to collateral edits.
  • Experimental setup: Nine LLM-based agents are evaluated with the same prompts, action schema, environment, and configurations.The evaluated models include Gemini, GPT, Qwen, Llama, and EuroLLM variants.
  • Experimental setup: Each task is run once per model, recording trajectories, final environment states, evaluator outcomes, and diagnostic events.Percentages cover CTS, pass rate, preservation, iteration-cap hits, and clean termination; malformed actions and solved-task trajectory length are averaged.

5 Results

Across WORLDBENCH, agents achieve limited CTS, with pass rates consistently exceeding CTS because correct task outcomes often violate workspace preservation. Performance also varies by language, application family, evaluator type, task complexity, and failure mode.

  • Preservation gap: Pass rate exceeds CTS for every model, with gaps ranging from 10.1 for Gemini-3.1-Pro to 16.8 for Llama-3.3-70B.The difference represents completed tasks that still modify files that should have remained unedited.
  • Language: English generally yields the highest CTS, while Chinese is among the most difficult settings for most models; Qwen-3-32B achieves the highest Chinese score.The authors interpret these as setting-level performance gaps and discuss limits to cross-locale comparability.
  • Application families: Calendar and document tasks achieve higher scores, whereas messaging and shell tasks show lower CTS and larger preservation gaps.Their difficulty reflects both incomplete execution and collateral modifications.
  • Evaluation functions: Deterministic file-existence and spreadsheet-cell functions achieve higher pass rates than email and note evaluators.Open-ended outputs remain more difficult to satisfy.
  • Task complexity: Performance generally declines as tasks require more substantive actions across the four reference-solution-length bins.The analysis treats reference-solution length as intrinsic task complexity.
  • Failure modes: Wrong output is the largest failure category for stronger models, while iteration-cap hits reach up to 43% of EuroLLM-9B failures.Collateral edits remain substantial across all agents, and weaker models execute more steps and malformed actions while terminating cleanly less often.

6 Discussion

The discussion identifies workspace safety, long-horizon execution, and multilingual performance as three persistent limitations of current agents. It points toward stronger state tracking, explicit preservation objectives, and localization-aware action grounding.

  • Limitations: Current agents lack a reliable notion of workspace safety, remain brittle over long horizons, and show unstable multilingual performance.The discussion notes that agents may identify target artifacts while adding noise to neighboring files, with errors compounding beyond the solution path.
  • Benchmark scope: The benchmark supports adding new personas, languages, and task patterns through seeds, including fresh tasks when contamination is suspected.The discussion presents this extensibility alongside the identified agent limitations.

7 Conclusion

WORLDBENCH is a multilingual, persona-grounded benchmark for executable agentic tasks that evaluates both final-state correctness and workspace preservation. Experiments find that the best model remains below 50% CTS, with preservation gaps and reduced performance outside English.

  • WORLDBENCH is a multilingual, persona-grounded benchmark for executable agentic tasks.
  • The benchmark materialises testbeds, evaluates final system states with deterministic and judge-based functions, and uses CTS to enforce correctness and preservation.
  • The best evaluated model does not surpass 50% CTS, while all models exhibit a notable preservation gap.
  • Performance degrades outside English, making multilingual robustness another central requirement for reliable agents.

Limitations

WORLDBENCH prioritises reproducible structured file-based workflows, but its current scope excludes visual desktop control and live web interaction. Its open-ended evaluators also rely on an LLM judge whose agreement is examined separately.

  • WORLDBENCH evaluates structured, file-based workflows through a fixed action interface for reproducible experiments.
  • The current version excludes visual desktop control and live web interaction, which are intended for future development versions.
  • Two open-ended evaluators use an LLM judge, introducing an evaluation component whose agreement with human annotations and alternative judges is examined in an appendix.

Ethics Statement

WORLDBENCH uses synthetic personas and data to avoid private information and personally identifying records from real users. The paper identifies sensitive workplace deployment as the main ethical risk and emphasises preservation, auditability, and failure reporting.

  • WORLDBENCH uses synthetic personas and synthetic data rather than private information or personally identifying records from real users.
  • The main ethical risks concern evaluating systems that may be deployed in sensitive workplace contexts.
  • The paper emphasises preservation, auditability, and failure reporting in addressing these evaluation risks.

I Judge Robustness

The judged evaluations show substantial robustness across alternative judges and human references. Judge choice changes pass rates only modestly and leaves the evaluated-agent ranking unchanged.

  • 94.1% of items receive agreement among the three judges, while judge choice shifts reported pass rates by at most 2.4 points on either function.
  • Agreement with the human reference is substantial, with Cohen’s kappa ranging from 0.76 to 0.81 and majority voting raising it to 0.84.
  • All three judges are slightly more permissive than human annotators because the default criteria reward well-formedness without penalising minor factual drift.
  • The ranking of evaluated agents is identical across all three judges and the human reference.

J Recurring Error Patterns

The benchmark’s failed trajectories cluster into recurring agent-error patterns, with preservation failures and locale mismatch standing out as especially important. These patterns explain why correctness-only scoring can overestimate successful execution.

  • Error taxonomy: Table 10 reports pattern shares across manually inspected failed trajectories, with shares computed within each language column and summing to 100%.The analysis inspects 400 failed trajectories stratified by model and language.
  • Error taxonomy: The six recurring patterns are distractor capture, redundant inspection, shell over-reach, localisation mismatch, premature termination, and schema violation.The patterns cover wrong-file edits, repeated non-action, overly broad shell operations, locale conventions, incomplete task termination, and malformed actions.
  • Recurring observations: 42.4% of failures come from distractor capture and shell over-reach, both of which modify non-target workspace state.These are preservation failures by construction and help explain the gap between pass rate and CTS.
  • Recurring observations: Locale mismatch rises from 6.8% in English settings to 21.4% in non-English settings and peaks at 26.9% in Chinese.It is the only pattern whose share varies substantially by language and is identified as the principal mechanism underlying the language gradient.

K Benchmark Composition

WorldBench combines persona-grounded workflows with balanced multilingual task construction, varied topics and evaluation functions, and sandboxed examples of both preserved and failed execution. Cross-locale comparisons remain bounded because settings are independently constructed rather than translated item by item.

  • Task complexity: Tasks span two to five substantive actions, with three-step tasks forming the largest group at 32.5%.Complexity excludes app switches and the final termination action; longer tasks require gathering information from several files before producing the output.
  • Topics: Finance and accounting is the largest topic family, followed by scheduling and correspondence, while all six families appear in every language setting.Other families include reporting, travel and expense, and records and administration.
  • Evaluation targets: Content containment is the most frequent evaluation target, complemented by spreadsheet cell checks, file-existence checks, and two judge-based functions.The judge-based functions account for slightly over a tenth of all attachments.
  • Language balance: Each language–locale setting contains 200 tasks with matched distributions of trajectory lengths, topics, application types, and evaluation functions.The benchmark is balanced at the aggregate level, while task sets are independently constructed without translations or item-level counterparts.
  • Cross-locale comparability: Cross-locale differences may combine instruction following, document conventions, cultural grounding, and residual task variation, so they should not be read as isolated language effects.The independently constructed settings preserve native language and locale-specific conventions but limit causal comparability.
  • Examples: The EN-UK and FR examples receive CTS, whereas the IT example passes its task-specific evaluator but fails preservation after editing a superseded file.The Italian distractor-capture example has CTS(t) = 0 despite pass(t) = true and accounts for 24.6% of manually inspected failures.
Loading 2609.01056v1…