Source-linked AI summary
OccuBench: Evaluating AI Agents on Real-World Professional Tasks via Language Environment Simulation
Xiaomeng Hu, Yinger Zhang, Fei Huang, Jianhong Tu, Yang Su, Lianghao Deng, Yuxuan Liu, Yantao Liu, Dayiheng Liu, Tsung-Yi Ho
TL;DR
Existing benchmarks cover only professional domains with public environments, leaving much occupational work unevaluable. OccuBench uses Language Environment Simulators to create solvable, diverse task environments and tests both occupational task completion and fault robustness. Its evaluations reveal distinct model capability profiles, greater difficulty with implicit faults, benefits from scaling, and the importance of simulator quality.
Problem
Most high-value professional domains lack public environments or APIs, while adding domains to existing benchmarks requires prohibitive engineering effort.
Method
OccuBench uses LLM-driven Language Environment Simulators to evaluate multi-step professional tasks across 100 scenarios, 10 industries, 65 domains, and controlled environmental faults.
Results
No model dominates all industries; implicit faults are hardest; scaling improves performance; and strong agents are not necessarily strong simulators, with capable simulators yielding 85.7% pairwise ranking agreement.
Takeaways & Limitations
OccuBench provides a systematic cross-industry evaluation of professional occupational tasks while showing that simulator quality is central to reliable LES-based assessment.
Takeaways & Limitations
LESs generate domain-consistent values rather than retrieving real data, so evaluations of precise numerical correctness should be complemented with real-environment testing.
Abstract
from arXiv · showhide
AI agents are expected to perform professional work across hundreds of occupational domains (from emergency department triage to nuclear reactor safety monitoring to customs import processing), yet existing benchmarks can only evaluate agents in the few domains where public environments exist. We introduce OccuBench, a benchmark covering 100 real-world professional task scenarios across 10 industry categories and 65 specialized domains, enabled by Language Environment Simulators (LESs) that simulate domain-specific environments through LLM-driven tool response generation. Our multi-agent synthesis pipeline automatically produces evaluation instances with guaranteed solvability, calibrated difficulty, and document-grounded diversity. OccuBench evaluates agents along two complementary dimensions: task completion across professional domains and environmental robustness under controlled fault injection (explicit errors, implicit data degradation, and mixed faults). We evaluate 15 frontier models across 8 model families and find that: (1) no single model dominates all industries, as each has a distinct occupational capability profile; (2) implicit faults (truncated data, missing fields) are harder than both explicit errors (timeouts, 500s) and mixed faults, because they lack overt error signals and require the agent to independently detect data degradation; (3) larger models, newer generations, and higher reasoning effort consistently improve performance. GPT-5.2 improves by 27.5 points from minimal to maximum reasoning effort; and (4) strong agents are not necessarily strong environment simulators. Simulator quality is critical for LES-based evaluation reliability. OccuBench provides the first systematic cross-industry evaluation of AI agents on professional occupational tasks.
1 Introduction
OccuBench addresses the lack of benchmarks for high-value professional domains by using Language Environment Simulators to evaluate multi-step occupational tasks and environmental robustness. Across 15 frontier models, results show distinct industry capability profiles, greater difficulty with implicit faults, consistent scaling benefits, and a separation between agent and simulator strength.
- Motivation: Existing benchmarks leave most high-value professional work untestable because domains often lack public APIs or environments, while adding domains requires prohibitive engineering effort.This limits coverage across healthcare, finance, legal, manufacturing, energy, governance, and logistics.
- Approach: Language Environment Simulators convert environment construction from an engineering problem into a configuration problem, extending coverage to domains an LLM can understand.An LES uses an LLM to simulate domain-specific tool responses from a configured environment.
- Benchmark: OccuBench covers 100 professional task scenarios across 10 industry categories and 65 specialized domains, evaluating task completion and robustness to explicit, implicit, and mixed faults.The benchmark is designed around real human job roles and multi-step tool use.
- Findings: No single model dominates all industries, with models leading different categories and exhibiting distinct occupational capability profiles.For example, Gemini 3.1 Pro leads Education and Science but struggles in Healthcare, while Claude Opus 4.6 leads Transportation but trails in Commerce.
- Findings: 53.4% average performance under implicit faults was lower than 62.6% under explicit faults and 67.5% without faults, because implicit degradation lacks overt error signals.Detecting truncated data or missing fields independently is difficult for most models.
- Findings: Higher model size, newer generations, and greater reasoning effort consistently improve performance, while strong agents are not necessarily strong environment simulators.GPT-5.2 improves by 27.5 points from none to xhigh effort; it ranks first as an agent but produces the worst simulation quality.
2 Related Work
Related benchmarks cover web, operating-system, code, API, and output-quality tasks, but remain limited in engineering scalability, contamination resistance, robustness evaluation, and professional-domain coverage. OccuBench instead targets interactive decision-making across specialized occupational domains through stateful tool-use environments.
- Agent Benchmarks: Existing web, operating-system, and mobile benchmarks require substantial engineering to construct and maintain, use static test sets, and lack systematic environmental robustness evaluation.Their combined domain coverage remains concentrated in web browsing, code editing, desktop operations, and a few API domains.
- Real-World Professional Task Evaluation: Professional-work benchmarks such as GDPVal, OneMillion-Bench, and TheAgentCompany emphasize output quality, monetary task value, or software-adjacent work rather than broad interactive occupational decision-making.OccuBench covers 65 specialized domains and requires multi-step tool use, state tracking, and error handling.
- Context Learning: CL-bench evaluates context-dependent reasoning, whereas OccuBench evaluates context-dependent action through multi-step workflows, environmental feedback, and adaptation to unexpected conditions.The distinction is between understanding task-specific context and executing actions within a stateful environment.
- World Models and Environment Simulation: Traditional world models are limited to low-dimensional state spaces, while earlier LLM simulation approaches generally lack tool-use interaction or stateful task execution.OccuBench positions LESs as a direct approach to language-based professional environment simulation.
3 Language Environment Simulator
A Language Environment Simulator is an LLM-driven, stateful environment that maps agent tool calls to observations using configuration and conversation history. Its configuration specifies behavioral rules, tools, initial conditions, and state semantics, while evaluation scores the resulting trajectory.
- Formalization: An LES maps an agent action and latent state to a new state and structured observation using an environment configuration.The configuration includes a system prompt, tool schema, initial state, and state description.
- Formalization: Unlike traditional world models, LESs use pretrained domain knowledge and prompts to encode operational logic rather than learning state transitions from data.Tool schemas define the action space, while initial state and state descriptions constrain causal consistency.
- Environment Configuration: Each environment is specified by a system prompt, tool schema, initial state, and semantic state description.Tool interfaces contain 2–10 callable tools, with a median of 5, reflecting operational interactions such as bookings or patient queues.
- Evaluation Loop: At evaluation time, the agent issues tool calls, the LES generates observations conditioned on configuration and conversation history, and a rubric-based verifier scores the trajectory.State is maintained implicitly through in-context history.
- Rationale: LLMs support professional-task simulation through formatting priors, domain knowledge, in-context state tracking, and handling of unexpected inputs.These capabilities help produce coherent multi-turn tool responses under configured simulation rules.
4 Multi-Agent Synthesis Pipeline
The multi-agent synthesis pipeline constructs professional evaluation instances from grounded reference documents, then verifies solvability, calibrates difficulty, and filters invalid or uninformative tasks.
- Instance Requirements: Each evaluation instance must be solvable, verifiable, discriminative, and diverse.These conditions require a valid verified solution, automated success criteria, calibrated difficulty, and structural variation.
- Instance Requirements: Sixteen non-overlapping sub-topics and professional reference documents ground structural diversity across terminology, workflows, state variables, edge cases, and constraints.The documents support generation that differs structurally rather than superficially.
- Synthesis and Quality Control: The pipeline generates environment configurations, tasks, tools, solution plans, and verification rubrics, then executes tasks with and without reference plans.A majority-vote verifier evaluates trajectories, while a repair module diagnoses and fixes failures before re-execution.
- Synthesis and Quality Control: Tasks with 100% autonomous success, 0% success, or invalid tool schemas are filtered out.This removes instances that are trivially easy, unsolvable, or technically invalid.
5 OCCUBENCH Benchmark
OCCUBENCH maps professional roles to multi-step scenarios and evaluates both clean task completion and robustness to controlled environmental faults.
- Benchmark Scope: OCCUBENCH contains 100 professional scenarios across 10 industry categories and 65 specialized domains, yielding 382 solvable task instances.Instances are quality-filtered to remove trivially solved, unsolvable, and invalid-schema tasks.
- Benchmark Scope: Every scenario maps to an actual professional role, balances domain coverage, targets tasks unavailable in existing benchmarks, and requires multi-turn state transitions.Examples include nuclear safety, drug screening, emergency coordination, and other professional settings.
- Fault Injection: Faults are injected during evaluation by appending fault rules to the LES system prompt, while all data is synthesized in clean environments.The benchmark separates clean generation from controlled robustness testing.
- Fault Injection: E1 produces visible failures such as timeouts and HTTP 500 responses, for which retrying is the correct behavior.Explicit faults provide clear signals that the tool call failed.
- Fault Injection: E2 silently degrades responses through truncation, incomplete lists, empty fields, or stale values, requiring agents to detect quality problems and re-query.These responses appear superficially correct because they lack explicit error signals.
- Fault Injection: E3 combines approximately half explicit and half implicit faults, with transient events controlled by fault count and duration.The default configuration uses two fault events and two affected tool calls per event.
- Metrics: Completion Rate is the fraction of 382 tasks passing automated rubric verification, while Robustness is min(CR_E1, CR_E2, CR_E3)/CR_E0.A robustness score of 1.0 indicates no degradation from environmental faults.
6 Experiments
Experiments evaluate 15 frontier models across industries, fault conditions, scaling factors, reasoning effort, and simulator choices. Results show distinct occupational profiles, substantial implicit-fault sensitivity, and simulator-dependent ranking reliability.
- Experimental Setup: The evaluation covers 15 frontier models spanning 8 model families, using thinking or reasoning mode where available.The default LES is Gemini-3-Flash-Preview.
- Cross-Industry Results: GPT-5.2 leads overall at 79.6%, while different models lead different industries and Qwen 3.5 Plus reaches 81% in Commerce versus GPT-5.2 at 67%.Gemini 3.1 Pro ranks second at 72.3%, and Claude Opus 4.6 ranks third at 71.5%.
- Cross-Industry Results: Open-source Qwen 3.5 Plus and DeepSeek V3.2 rank fourth and fifth at 69.9% and 69.6%, outperforming most Claude variants.The comparison challenges uniform closed-source superiority on professional tasks.
- Cross-Industry Results: Each model has a distinct cross-industry capability shape, revealing occupational specializations that single-domain benchmarks cannot expose.The radar chart visualizes these differing performance profiles across 10 industries.
- Environmental Robustness: Average completion falls from 67.5% in E0 to 53.4% in E2, a 14.1-point decline, even with only two fault events lasting two rounds each.Claude Opus 4.6 and Qwen 3.5 Plus also drop 17.6 and 18.3 points, respectively, under implicit faults.
- Environmental Robustness: E2 averages 53.4%, below E1 at 62.6% and E3 at 54.4%, because implicit degradation lacks overt signals and requires independent response-quality assessment.Four of nine models perform worse under E2 than E3.
- Environmental Robustness: Increasing fault count or duration further reduces completion, including Claude Opus 4.6 falling from 71.5% to 60.2% as count rises from 1 to 4.The corresponding duration comparison falls from 67.8% at duration 1 to 57.9% at duration 4.
- Scaling and Generations: Larger variants outperform smaller ones with gaps from 0.3% to 11.0%, including 11.0% for Gemini Pro versus Flash-Lite.Claude 4.5 is an exception, with Opus at 65.2% and Sonnet at 64.9%.
7 Analysis
OCCUBENCH reveals systematic differences in industry difficulty and model specialization, while case analyses illustrate failures in verification, procedural ordering, and fault handling.
- Industry Difficulty: Business & Enterprise averages 70.1% across models, whereas Transportation & Logistics averages 56.2%, making them the easiest and hardest industries respectively.Public Service & Governance averages 69.4%, while Education & Culture averages 57.6%.
- Industry Difficulty: Transportation and Education are the hardest industries in the average completion-rate comparison.Figure 12 groups industry averages into green, orange, and red performance bands.
- Implications: OCCUBENCH’s cross-industry profiles support selecting models for specific industries rather than relying only on aggregate rankings.This implication follows from the distinct patterns of strengths and weaknesses across models.
- Occupational Profiles: Gemini 3.1 Pro excels in Education at 84%, Science at 81%, and Technology at 78%.These knowledge-intensive domains reward factual accuracy and structured reasoning.
- Occupational Profiles: Claude Opus 4.6 leads in operational domains including Transportation at 77%, Business at 78%, and Industrial at 73%.These domains reward careful state tracking and multi-step planning.
- Occupational Profiles: Qwen 3.5 Plus leads in Commerce and Healthcare at 81% each and reaches 78% in Agriculture.The analysis associates these strengths with consumer-facing domains.
- Occupational Profiles: Kimi K2.5 is balanced across most industries but scores 56% in Commerce and 57% in Transportation.The reported weaknesses concern consumer interaction and logistics optimization tasks.
- Case Analyses: The case analyses cover realistic logistics, aquaculture, construction, explicit-fault transit, and silently truncated-data tasks.They illustrate skipped verification, procedural ordering errors, compliance failures, and implicit fault injection.
8 Discussion & Conclusion
OCCUBENCH broadens professional-task evaluation through Language Environment Simulators, while showing that simulator fidelity and dependence constrain interpretation. Its results emphasize varied occupational capabilities, environmental robustness, and the importance of checking degraded data and procedural order.
- Limitations: LES-based evaluation models domain logic rather than exact domain data, so precise numerical tasks should also use real-environment testing.An LES can represent operational behavior such as returning drug contraindications, but its specific values are generated rather than retrieved from a real database.
- Case Studies: Case studies show that reliable professional behavior requires proactive constraint checks, post-action verification, correct procedural order, and detection of truncated data.Examples include re-verifying chemistry after mixing, restoring valves after inspection work, retrying explicit errors, and re-fetching incomplete valuation data.
- Limitations: Simulator dependence means that solvability and agent rankings can change with the LES used during data synthesis.The simulator is part of the evaluation apparatus rather than a neutral observer.
- Conclusion: OCCUBENCH makes professional-task evaluation possible across 100 scenarios, 65 specialized domains, and 10 industry categories without real environment infrastructure.
- Findings: No model dominates every industry, while implicit faults are harder than explicit and mixed faults because they lack overt error signals.