Source-linked AI summary
DuMateBench: Evaluating Autonomous Agents in Complex Real-World Workflows
Zechun Niu, Yukun Zhao, Jiaxin Zhang, Xu Shen, Jinhua Si, Han Tian, Can Xu, Yunfan Song, Jiaxin Mao, Yansong Gao, Yuchen Li, Jianmin Wu, Lingyong Yan, Shuaiqiang Wang, Dawei Yin
TL;DR
Existing benchmarks provide limited evidence about agents’ ability to coordinate real-world cross-capability workflows under imperfect conditions. DuMateBench reconstructs such tasks from privacy-screened production sessions, evaluates them in controlled perturbed environments, and finds substantial gaps in strict task completion across framework–model configurations.
Problem
Existing benchmarks typically separate capabilities and evaluate agents in cleaner, more stable environments than real-world workflows.
Method
DuMateBench reconstructs 200 human-verified tasks from privacy-screened production sessions and executes them in Docker containers with insufficient, unstable, and noisy conditions using hybrid checklist and LLM-judge evaluation.
Results
Substantial gaps in strict task completion appear across five autonomous-agent frameworks paired with four state-of-the-art LLMs.
Takeaways & Limitations
Performance under environmental perturbations is jointly shaped by the capabilities of the LLM and the surrounding agent framework.
Abstract
from arXiv · showhide
Autonomous agents are increasingly adopted to complete complex, multi-tool workflows in real-world settings. However, existing benchmarks typically separate tasks by application or capability and evaluate agents in environments that are cleaner and more stable than those encountered in practice. We introduce DuMateBench, a real-session benchmark reconstructed from anonymized and privacy-screened user sessions collected from a large-scale production agent platform. Each task preserves the relevant pre-solution interaction history, persistent configurations, and workspace state, and is then validated through human verification. The resulting benchmark comprises 200 tasks spanning 8 broad scenarios and 17 fine-grained capability categories, with most tasks requiring multiple capability coordination. We execute these tasks in isolated Docker containers injected with three forms of real-world environmental complexity: Insufficient, Unstable, and Noisy, and assess performance using a hybrid deterministic and LLM-as-Judge evaluation protocol. Experiments across five representative autonomous-agent frameworks paired with four state-of-the-art LLMs reveal substantial gaps in strict task completion. Complementary robustness, efficiency, and diagnostic analyses further show that performance under environmental perturbations is jointly shaped by the capabilities of the LLM and the surrounding agent framework. The code and data are publicly available at https://dumatebench.com/.
1 Introduction
Existing benchmarks provide limited evidence about agents’ ability to coordinate cross-capability workflows and operate under realistic environmental complexity. DuMateBench addresses these gaps with real-session tasks, controlled environmental perturbations, and complementary evaluation.
- Existing benchmarks rarely unify compositional workflows, environmental complexity, and heterogeneous capabilities in one evaluation setting.
- DuMateBench derives 200 tasks from anonymized, privacy-screened production sessions while preserving interaction context, persistent configurations, and workspace state.
- Most tasks coordinate multiple capabilities across eight broad scenarios and 17 fine-grained task types, including content generation, coding, document manipulation, and Web retrieval.
- The benchmark models insufficient, unstable, and noisy environments in isolated Docker containers to test reliability under imperfect execution conditions.
- Evaluation combines deterministic checklists for explicit requirements with artifact-specific LLM judges for semantic, organizational, perceptual, and presentation quality.
- Across 20 framework–model configurations, results reveal agent–model interactions, uneven robustness, quality–efficiency tradeoffs, and weaknesses in planning, recovery, and artifact verification.
2 Related Work
Related benchmarks increasingly evaluate multi-step workflows and reliability under perturbations, but these difficulties are usually isolated from session-derived tasks requiring coordinated tools and heterogeneous artifacts. DuMateBench integrates them in one setting.
- Workflow-oriented benchmarks extend evaluation from isolated tool calls to coordinated multi-step operations across office, enterprise, coding, and business applications.
- Reliability benchmarks examine insufficient information, distracting context, user and tool noise, API failures, data degradation, and incompatible execution environments.
- These benchmark families generally isolate environmental difficulties from session-derived workflows requiring coordinated tool use and heterogeneous artifact delivery.
- DuMateBench combines insufficient, unstable, and noisy conditions with real-session tasks to evaluate reliable request fulfillment and artifact delivery in imperfect environments.
3 DuMateBench Task Construction
DuMateBench reconstructs executable tasks from anonymized production sessions, verifies their fidelity and solvability, and characterizes their capability and knowledge-domain coverage. The resulting dataset emphasizes coordinated, multi-capability workflows.
- Task Derivation & Reconstruction: DuMateBench is derived from anonymized, privacy-screened sessions collected from a large-scale production agent platform and filtered through human review.
- Task Derivation & Reconstruction: Task derivation reconstructs interaction history, formulates a cutoff-based instruction, and rebuilds the workspace state available at that cutoff.
- Task Derivation & Reconstruction: Each task instance contains an instruction q_c, reconstructed workspace W_c, and metadata or execution constraints M_c, serialized into a standardized task package.
- Human Verification: Human verification retains tasks that faithfully preserve the request, provide complete and consistent workspaces, avoid solution leakage, and contain no unresolved risks.
- Benchmark Statistics: 200 tasks receive 14 fine-grained capability annotations across five coarse-grained scenarios, with 456 assignments averaging 2.28 capabilities per task.
- Benchmark Statistics: 159 tasks (79.50%) span at least two coarse-grained scenarios, with code development plus content generation the most common cross-scenario combination at 62 tasks.
- Benchmark Statistics: Business is the largest knowledge domain at 40.5%, followed by technology and engineering at 21.5% and humanities and social science at 17.5%.
4 DuMateBench Environment Design
DuMateBench treats environmental conditions as a controlled component of benchmark difficulty. Its isolated containers inject insufficient, unstable, and noisy conditions to evaluate reliable task completion.
- The environment dimensions are Insufficient conditions with missing dependencies or constrained resources, Unstable conditions with transient failures, and Noisy conditions with distractor files or data.
- Insufficient Environments: Insufficient environments require agents to inspect conditions, diagnose missing dependencies, install permitted packages, or use alternative implementations under resource limits.
- Unstable Environments: Unstable environments inject startup or periodic network and tool faults, requiring agents to distinguish transient failures and apply retries, backoff, fallbacks, or alternative plans.
- Noisy Environments: Noisy environments preserve and generate irrelevant, redundant, outdated, duplicate, or similarly named files and data without altering required inputs.
- Each task runs in an isolated Docker container, making environmental perturbations reproducible, explicit, controlled, and auditable rather than incidental execution details.
5 DuMateBench Evaluation
DuMateBench combines deterministic requirement checks with artifact-specific rubric judging to evaluate both explicit compliance and the quality of heterogeneous outputs. Its fixed, reviewed criteria support task-level aggregation across requirements and artifacts while preserving partial progress and penalizing missing evidence.
- Evaluation protocol: Deterministic checklist evaluation verifies objectively testable requirements, while artifact-specific judges assess semantic, organizational, and perceptual properties beyond fixed rules.The protocol covers textual documents, presentations, spreadsheets, PDFs, images, audio, and video.
- Deterministic checklist evaluation: LLM-generated checklists are reviewed by human annotators and cover artifact existence, location, format, content, structure, values, formulas, and protected-file integrity.Evaluator-only reference sources and human-checked gold answers support web-retrieval, numerical, and question-answering tasks without exposing evaluation materials to agents.
- Deterministic checklist evaluation: 200 tasks contain 1,257 atomic checks, averaging 6.29 per task, with existence, required-content, and format-validity checks comprising 93.01% of all checks.The checklist therefore emphasizes artifact presence, validity, and compliance with explicit requirements before qualitative evaluation.
- Rubric-based artifact evaluation: Rubrics contain three to sixteen weighted criteria with anchored scores from 0 to 4, and judges record supporting evidence or mark criteria cannot_assess when evidence is insufficient.Unassessed criteria contribute zero, and repeated judge runs aggregate criterion scores by their median.
- Rubric-based artifact evaluation: 454 rubric files cover 197 of 200 tasks and define 2,308 atomic criteria, averaging 11.54 criteria per task and 5.08 criteria per rubric file.The five dominant dimensions are requirement completeness, presentation readability, functional correctness, content relevance, and factual correctness and faithfulness.
- Score aggregation: The final task score weights deterministic requirement coverage at 30% and artifact quality at 70%, while P_t, J_t, and F_t are reported and macro-averaged across tasks.Missing expected artifacts receive zero, and tasks without an applicable artifact judge use the deterministic score.
6 Experiments
Experiments evaluate 20 agent–model configurations on DuMateBench across performance, environmental robustness, efficiency, and failure modes. Results show strong agent–model interactions, substantial noise sensitivity differences, quality–efficiency trade-offs, and recurring execution, implementation, grounding, and environment failures.
- Experimental Settings: 20 configurations pair five autonomous-agent frameworks with four base models while preserving each runtime’s native control loop and tool-use policy.Each task uses the same instruction and initial workspace in an isolated Docker container, with evaluation files inaccessible to the agent.
- Experimental Settings: The evaluation combines deterministic checklist partial-pass rates with LLM-as-Judge scores for artifact correctness, completeness, and quality.The final score is computed as 0.3 Partial + 0.7 Judge.
- RQ1: Agent Performance: DuMate’s Final-score range is 0.8046–0.8548 across models, while OpenClaw’s range is 0.5821–0.7887, showing markedly different model sensitivity across agent systems.The best-performing model differs by agent, and DuMate ranks first within each model block.
- RQ1: Agent Performance: DeepSeek-V4-Pro has the highest mean Final score (0.8106) and the smallest cross-agent range (4.63 percentage points) among the evaluated base models.Opus-4.8 produces the highest individual score despite not having the highest mean score.
- RQ2: Environmental Noise: From normal to high workspace noise, Hermes loses 20.08 percentage points, whereas DuMate decreases from 0.8548 to 0.8381, a loss of 1.67 percentage points.Noise introduces seeded filename and content distractors, including backups, duplicated text, conflicting values, and corrupted content.
- RQ3: Efficiency: DuMate with Opus-4.8 achieves the highest Final score (0.8548) but is slowest at 1,038.74 s per task and consumes 1.56M total tokens per task.Claude Code with GPT-5.5 is fastest at 274.99 s with a Final score of 0.7830, while OpenCode with GPT-5.5 uses the fewest total tokens but scores 0.6906.
- RQ4: Failure Modes: Incomplete execution or budget exhaustion accounts for 28% of DuMate’s sampled non-complete runs and 32% of Claude Code’s, while incorrect implementation or tool use accounts for 30% and 22%.Requirement or context grounding failure is more frequent for Claude Code, whereas environment or dependency failure is more frequent for DuMate.
- RQ4: Failure Modes: Concrete cases expose failures in execution planning, recovery from successive service failures, and tracing numeric claims to source documents.These cases include unfinished artifact export, incomplete sections after failed evidence retrieval, and conflicting values inserted into deliverables.
7 Conclusion
DuMateBench evaluates autonomous agents on complex workflows using reconstructed real-session context, standardized imperfect environments, and hybrid artifact evaluation. Experiments reveal substantial room for improvement in strict task completion and show that performance depends on both the base model and agent framework.
- DuMateBench comprises 200 de-identified, human-reviewed tasks derived from multi-turn sessions, preserving user-visible context and workspace state.
- The benchmark combines compositional workflows with Docker environments representing insufficient, unstable, and noisy conditions.
- Hybrid deterministic checklists and artifact-specific LLM-as-Judge rubrics assess explicit task completion and heterogeneous output quality.
- Experiments across five agent frameworks and four base models reveal substantial room for improvement in strict task completion.
- Performance under complex environmental conditions depends on both the base model and the surrounding agent framework.
8 Ethical Considerations
DuMateBench is constructed from anonymized and privacy-screened sessions, with sensitive information removed and unsafe tasks excluded. Its single-platform origin limits how broadly its usage patterns may generalize, while isolation measures address security and safety risks.
- Personally identifiable information, credentials, access tokens, private endpoints, and other sensitive data are removed before task inclusion.
- Tasks that cannot be safely de-identified are excluded, and remaining tasks undergo human review before inclusion.
- Because the benchmark reflects usage patterns from a single platform, it may not represent all users, occupations, or workflows.
- Isolated containers, restricted permissions, and task-level workspace boundaries mitigate security and safety risks.