Source-linked AI summary
EnterpriseClawBench: Benchmarking Agents from Real Workplace Sessions
Jincheng Zhong, Weizhi Wang, Che Jiang, Kai Tian, Zhenzhao Yuan, Junlin Yang, Dianqiao Lei, Kaiyan Zhang
TL;DR
Enterprise agent benchmarks still lack naturally occurring enterprise demand and multidimensional evaluation. EnterpriseClawBench converts proprietary workplace sessions into 852 reproducible tasks, and its best configuration reaches only 0.663, with performance substantially shaped by the harness used.
Problem
Existing benchmarks leave gaps between naturally occurring enterprise demand and benchmark tasks, and in multidimensional evaluation beyond base-model performance.
Method
EnterpriseClawBench converts proprietary workplace sessions into reproducible, artifact-centric tasks with fixtures, taxonomies, hard rules, and text/visual semantic rubrics.
Results
0.663 is the best score for Codex/GPT-5.5, while performance changes substantially with the harness used and enterprise artifact tasks remain unsaturated.
Takeaways & Limitations
Enterprise agent evaluation should report harness–model combinations, artifact delivery, semantic quality, cost, runtime, and skill-transfer behavior rather than a single score.
Takeaways & Limitations
Because EnterpriseClawBench is built from one enterprise deployment, its results may not represent all organizations.
Abstract
from arXiv · showhide
Enterprise agents increasingly operate inside workspaces: they read heterogeneous files, invoke tools, and deliver business artifacts. We introduce EnterpriseClawBench, an enterprise agent benchmark constructed from proprietary, real-world agent sessions. Starting from a large archive of workplace sessions, the EnterpriseClawBench produces 852 reproducible tasks, each paired with recovered fixtures, rewritten prompts, role classes, skill subclasses, hard rules, and semantic rubrics. Because the sessions contain internal enterprise content, we do not release the benchmark data; instead, our reusable contribution is the construction and evaluation protocol. On EnterpriseClawBench, the best configuration reaches only 0.663 (Codex with GPT-5.5). These results show that enterprise agent evaluation must report harness--model combinations, artifact delivery, visual quality, cost, runtime, and skill-transfer behavior, rather than collapsing performance into a single score. Code: https://github.com/FrontisAI/EnterpriseClawBench
1. Introduction
EnterpriseClawBench addresses gaps in enterprise-agent evaluation by converting historical workplace sessions into reproducible tasks while jointly evaluating harness–model performance, artifacts, quality, efficiency, and skill generalization. Its proprietary source data remain unreleased, so the paper contributes a construction and evaluation protocol.
- Motivation: Enterprise agents operate inside executable workspaces, where success requires more than producing a correct chat response.Sessions comprise chat turns, uploaded files, tool traces, generated artifacts, and persistent workspace state.
- Motivation: Existing benchmarks leave gaps between scalable task construction, enterprise realism, and naturally occurring enterprise data.Prior environments include workspace or enterprise-like settings, but many tasks remain human-authored, simulated, or based on public environments.
- Benchmark construction: EnterpriseClawBench converts historical internal agent sessions into reproducible benchmark tasks through automated recovery, rewriting, taxonomy, rubric, and sandbox checks.The pipeline also performs reproducibility and privacy checks and detects deliverables.
- Evaluation framework: The benchmark jointly reports harness–model performance, file delivery, text and visual semantic quality, cost, and runtime.This multidimensional framework treats these dimensions as evaluation outputs rather than collapsing them into a single success criterion.
- Evaluation framework: EnterpriseClawBench supports skill-generalization evaluation across held-out tasks from the same enterprise task class.Because the source includes proprietary sessions, attachments, and business context, the benchmark data are not released.
2. EnterpriseClawBench Data and Construction
EnterpriseClawBench converts proprietary workplace sessions into reproducible, single-turn benchmark tasks through mechanical reproducibility gates and structured annotation. Its 852-task set spans broad enterprise roles and skills, distilled from 5,291 recovered raw TaskInstances.
- Enterprise setting: The benchmark captures continuous enterprise-agent use at an AI startup, where employees upload files and expect deliverables in a persistent Linux workspace.Sessions occurred from March to May 2026 through private or group chats in an enterprise collaboration platform.
- Construction pipeline: The construction pipeline recovers task candidates, applies reproducibility gates, and rewrites passing candidates into single-turn prompts with role and skill annotations.Gates include length filtering, fixture lookup, redaction recovery, and network-dependency checks.
- Benchmark statistics: 5,291 raw TaskInstances yield 852 final benchmark tasks spanning seven role classes and 45 role-specific skill subclasses.Product/project and engineering/IT are the largest role blocks, while finance, marketing, sales, executive, and HR/admin form a specialized long tail.
3. Experiments
Experiments evaluate enterprise agents as harness–model combinations using hard-rule and semantic artifact scoring. Results show limited saturation, strong interaction and artifact effects, variable skill transfer, and weaker reliability for visual judging.
- Evaluation protocol: The benchmark evaluates independently sandboxed harness–model combinations, first checking objective artifact-delivery rules and then scoring five semantic quality dimensions.Hard rules cover file type, count, non-emptiness, openability, tracebacks, and placeholders; semantic judging routes text-extractable and HTML outputs differently.
- Main results: 0.663 is the best score for Codex/GPT-5.5 on the audited 120-task Lite subset, indicating that enterprise artifact tasks remain far from saturated.The 32-combination leaderboard uses Sonnet 4.6 as both text and visual judge.
- Harness–model interaction: 0.458 is Sonnet 4.6’s score under Hermes, versus 0.62–0.64 under Claude Code, DeepAgents, and OpenClaw, indicating strong harness–model coupling.Trace inspection attributes the Claude-family drop under Hermes to runtime mismatch involving environment probing, script execution, and multi-step repair.
- Role and artifact effects: GPT-5.5 is the most robust generalist, while marketing and finance/operations are harder role classes and deliverable format changes model rankings.GPT-5.5 leads HTML, code/JSON, and several delivery-heavy categories; Opus 4.6 leads spreadsheets, where visual judging also inflates scores relative to text-only deliverables.
- Skill transfer: +0.0681 is GPT-5.5’s average skill-transfer effect, compared with +0.0518 for Kimi K2.6 and -0.0941 for Haiku 4.5.Skill injection varies with creator quality, consumer behavior, creator–consumer fit, and consumer baseline, motivating consumer–creator matrices rather than one average.
- Judge reliability: 0.918 and 0.866 are correlations between alternative and main LLM judges, while human agreement is stronger for text (MAE 0.134; ρ=0.790) than visual artifacts (MAE 0.303; negative rank correlation).The visual route is substantially more conservative than humans, exposing multimodal evaluation as an immature component.
4. Conclusion
EnterpriseClawBench is an internal benchmark that converts proprietary workplace sessions into 852 reproducible, artifact-centric tasks with fixtures, taxonomy, and text/visual rubrics. Its unsaturated tasks reveal harness-sensitive performance, while cost, runtime, artifact delivery, semantic quality, and skill generalization complement final scores.
- Benchmark construction: 852 reproducible, artifact-centric tasks are automatically produced from proprietary workplace archives, with fixtures, taxonomy, and text/visual rubrics.The benchmark is built from real internal agent sessions.
- Evaluation behavior: Tasks remain unsaturated, and model performance changes substantially with the harness used.These properties make benchmark results sensitive to the agent configuration being evaluated.
- Evaluation scope: Cost, runtime, artifact delivery, and multi-dimensional semantic quality are necessary alongside final scores to explain system behavior.The task-class taxonomy also makes the benchmark naturally suited for assessing skill generalization.
5. Limitations
EnterpriseClawBench’s results may not generalize beyond its source deployment, while data restrictions limit reproducibility and imperfect LLM judges require calibration, especially for visual artifacts.
- Generalizability: The benchmark comes from one enterprise deployment, so its results may not represent all organizations.This limits the benchmark’s generalizability across organizational settings.
- Data access: The full dataset is unreleased because it contains internal sessions, attachments, tool traces, and business artifacts.These enterprise materials prevent public release of the benchmark data.
- Evaluation: LLM judges remain imperfect, especially for visual artifacts, requiring judge ablation and human-correlation experiments for calibration.The paper identifies these experiments as necessary calibration steps.
6. Ethical Considerations
Organizations reusing this approach should protect sensitive enterprise data through authorization, restricted access, auditing, and safeguards against disclosure.
- Data protection: Reuse requires authorization, restricted data access, sensitive-information review, preserved redaction and recovery audit records, and protection against leaking people or proprietary information.Benchmark construction must not expose employees, customers, or proprietary information.
A. Appendix: Judge Ablation · B. Appendix: Case Studies · B.1. End-to-End Construction Case
The appendices clarify judge comparability and illustrate how EnterpriseClawBench converts workplace traces into release-safe, reproducible benchmark instances. The construction case applies successive gates for length, fixtures, redactions, network access, self-containment, prompt rewriting, and taxonomy labeling.
- A. Appendix: Judge Ablation: Different judges use different absolute score scales, so their scores should not be mixed in one leaderboard.Strong judges are generally more conservative, while less-capable judges assign higher scores; capable judges nevertheless preserve the broad system ordering.
- B. Appendix: Case Studies: Case studies mask internal entities, URLs, identifiers, project names, session identifiers, and monetary values while preserving Chinese excerpts with literal English translations.The release-safe evidence boxes expose task packets, model outputs, skill files, and judge feedbacks behind benchmark claims.
- B.1. End-to-End Construction Case: The pipeline removes extremely short traces before expensive recovery stages when they lack enough linguistic information to recover a task objective.One example is rejected with effective_user_chars: 1 and min_user_chars: 10.
- B.1. End-to-End Construction Case: Missing required files cause fixture-gate rejection because retaining such instances would confound data availability with agent ability.The example declares one DOCX fixture but resolves zero files and has total_bytes: 0.
- B.1. End-to-End Construction Case: Redacted task-source values are accepted only when local context or a unique same-host candidate supports high-confidence recovery; otherwise, the instance is rejected.The illustrated URL is restored by context with confidence: high.
- B.1. End-to-End Construction Case: The network gate rejects tasks whose external links are not stably reachable during construction.The example records url_count: 1, reachable_count: 0, and unreachable_count: 1.
- B.1. End-to-End Construction Case: Tasks lacking a recoverable objective are excluded when they cannot be rewritten into clear, standalone prompts.The example is marked self_contained: false with reason_code: ambiguous_task, despite passing some mechanical checks.
- B.1. End-to-End Construction Case: Admitted instances are rewritten into single-turn specifications with explicit materials, templates, deliverables, benchmark classes, role taxonomies, and skill subclasses.The demonstrated project-report task receives confidence: 0.92 and the role_class product_project_delivery.
B.2. Representative Instances by Task Type
This section presents representative task instances through Table 4 in the Appendix, which summarizes role-class boundaries and provides representative prompt excerpts translated into English.
- Table 4 in the Appendix summarizes the role-class boundary for representative task instances.
- The table includes representative prompt excerpts for the task types.
- The prompt excerpts are translated into English.
B.3. One Evaluation Case: Prompt, Input, Output, Rubric, Score
This section presents a medium-scoring visual-judge case in which an agent calibrates and analyzes business data from two Excel files before delivering a new spreadsheet. Its 0.66 score illustrates how the judge distinguishes partial success from full task completion.
- Evaluation case: 0.66 is the case’s visual-judge score, making it useful for examining partial success versus full task completion.The section deliberately uses a medium-scoring case rather than a perfect example.
- Prompt and inputs: The task requires reading two Excel business-data files and calibrating revenue, cost, gross profit, gross margin, and business-unit cash-flow information.The files contain revenue, cost, gross-profit, and business-unit cash-inflow/outflow data across business lines during a specified period.
- Output: The requested deliverable is one spreadsheet file with a task-relevant filename.The agent may choose the filename, provided it is relevant to the task content.
- Output: After completion, the agent must briefly report the output path.This path-reporting requirement accompanies the spreadsheet deliverable.
Input evidence
The task required cross-file calibration and analysis of financial data containing abnormal cells, while the delivered workbook covered core calibration but incompletely addressed cash flow and anomaly handling. The visual judge scored the output 0.66.
- Input requirements: The two spreadsheet inputs required comparing files, detecting Err:504 cells, calibrating financial values, and analyzing cash flow.The inputs included revenue, outsourced cost, gross profit, and gross margin across masked business-line rows.
- Delivered artifact: The delivered artifact contained two sheets for revenue-cost calibration and business-unit statistics, with original and calibrated values plus deviation analysis.The workbook was saved at /workspace/outputs/business_data_calibration_report.xlsx.
- Evaluation: 0.66 was the visual judge’s overall score for the spreadsheet output.The judge assessed the artifact after rendering it into page images.
- Evaluation findings: The output did not fully cover required cash-flow analysis, and it did not explicitly handle abnormal values or provide anomaly-repair guidance.The judge found calibration generally consistent with the source files but incomplete against the task requirements.
B.4. Skill Transfer Cases
On the same held-out HTML task for three membership crown icons, skill injection produced both positive and negative transfer: scores rose by +0.088 in one case but fell by -0.201 in the other. The qualitative feedback likewise described clearer accessory progression and overall quality in the positive case, but limited core visual quality and crown-shape fidelity in the negative case.
- Task setup: The held-out task asks the agent to generate one HTML page displaying Basic, Standard, and Flagship membership crown icons.The crowns should differ in color, accessories, and detail while using a cute 3D style.
- Positive transfer: 0.844 after skill injection versus 0.756 without skill injection, a positive-transfer gain of +0.088.Judge feedback reported clear color differentiation and layout, but insufficiently deep cute 3D styling and accessory-detail progression without the skill.
- Positive transfer: The skill-injected positive-transfer output fully displayed the three icons with clear color differentiation and accessory progression, achieving overall good quality despite limited 3D depth.This assessment is based on the judge feedback for the generated membership_crown_icons.html artifact.
- Negative transfer: 0.555 after skill injection versus 0.756 without skill injection, a negative-transfer drop of -0.201.The negative-case judge found clear differentiation and complete display, but criticized deviation from the standard crown form and limited 3D depth.
- Negative transfer: The negative-transfer skill-injected output had relatively complete layout and level labels, but limited core visual quality because crown shapes diverged from the reference and lacked 3D detail.The feedback specifically identifies the crown icon shapes and 3D detail as the main weaknesses.