Source-linked AI summary
FinCUABuild: Can Agents Build Reliable Benchmarks for Dynamic Financial Computer Use?
Jingpu Yang, Fengxian Ji, Jinri Guo, Tianhao Li, Qian Jiang, Fan Zhang, Min Peng, Qianqian Xie, Preslav Nakov, Zhuohan Xie
TL;DR
Financial CUA evaluation tasks remain costly to construct despite the diversity of financial workflows and runtime conditions. FinCUABuild introduces a standardized benchmark and multi-agent construction system; across four backbones, the system improves strict Qual by 13.8–29.4 points over matched tool-augmented prompting.
Problem
Financial CUA task construction remains expert-intensive, while evaluation needs diverse workflow coverage, comparable conditions, and assessment of complete task packages.
Source: Introduction · p. 1Method
FinCUABuildBench standardizes evaluation across 576 requests covering 24 financial workflows and three runtime variations, while FinCUABuildAgent jointly constructs tasks, environments, and validators.
Source: Introduction · p. 2Results
Across four backbones, FinCUABuildAgent improves strict Qual by 13.8–29.4 points over matched tool-augmented prompting; Qwen3.5 achieves the highest Qual at 41.9%.
Source: Main Result · p. 6, Conclusion · p. 7Takeaways & Limitations
Qualified generated tasks yield differentiated automation-point completion scores across existing agents, demonstrating downstream diagnostic value.
Source: Conclusion · p. 7Remaining challenges include qualification yield, broader transfer, production coverage, source licensing, and verifier maintenance.
Source: Conclusion · p. 8
Abstract
from arXiv · showhide
Financial scenarios are diverse and complex, spanning varying data conditions, tool configurations, and workflows. Yet existing CUA, Computer-Using Agent, evaluation tasks remain largely manually constructed, limiting scalable coverage of real-world financial scenarios. Then, can agents autonomously construct diverse CUA evaluation tasks for financial scenarios? Evaluating this capability poses three key challenges: scenario coverage of construction requests, fair comparison across construction methods, and reliable assessment of generated task quality. To solve these, we introduce FinCUABuildBench, a benchmark for evaluating financial CUA task construction, featuring: (i) 576 construction requests covering 24 financial workflows and three types of runtime variation; (ii) standardized input, budget, and output specifications; and (iii) a task qualification mechanism based on execution tests and quality checks. We further introduce FinCUABuildAgent, a multi-agent system for automatically constructing dynamic financial CUA evaluation tasks. It consists of three modules that jointly construct tasks, environments, and validators. On FinCUABuildBench, under the same model backbone, existing agent-based construction methods achieve strict qualification rates of only 1.3-8.3%, while FinCUABuildAgent reaches 31.3%. Downstream evaluations further show that the constructed tasks can effectively differentiate CUA task-execution capabilities. These results demonstrate that agents can autonomously construct financial CUA tasks with meaningful evaluation value, offering a practical path toward broader evaluation coverage in financial scenarios. Code: https://github.com/FengxianJi/FinCUABuild
Introduction
The paper addresses costly construction of dynamic financial CUA tasks by introducing a benchmark and a multi-agent builder, and reports that the resulting tasks distinguish downstream CUA capabilities.
Motivation
- Financial workflows combine cross-file retrieval, spreadsheet operations, computation, and reporting across varied data, tools, and processes; Finch required over 700 hours of expert annotation despite LLM-assisted task mining.These construction costs motivate asking whether agents can build tasks that cover diverse financial scenarios.Source: Introduction · p. 1
Challenges
- Financial task construction lacks a unified construction-capability benchmark and a tailored agent, while evaluation must control scenario coverage, comparison conditions, and complete-package quality.Construction also requires executable dependencies among evidence, states, and actions, independent correctness auditing, and failure localization with targeted repair.Source: Introduction · pp. 1–2
Benchmark
- FinCUABuildBench covers 576 construction requests across 24 financial workflow categories and 192 templates, with state updates, tool failures, and policy changes as runtime variations.It standardizes inputs, resource budgets, and outputs, and qualifies task packages through evidence checks, execution and replay tests, event validation, and verifier testing.Source: Introduction · p. 2
Construction framework
- FinCUABuildAgent compiles capability dependencies into task requirements, independently audits tasks and verifiers, and uses targeted feedback-driven repair to construct tasks, environments, and verifiers.Specialist agents ground financial sources, build environments and runtime events, construct verifiers, and iteratively test and repair episodes.Source: Introduction · p. 2
Results
- 31.3% strict qualification versus 1.3–8.3% for general-purpose construction agents; 144 qualified tasks distinguish downstream CUAs in dynamic adaptation, and held-out qualification is 13.3% on 90 requests.Comparisons use the same model backbone; the held-out result indicates room for improved generalization.Source: Introduction · p. 2
Related Work
Related work covers financial reasoning and workflow benchmarks, computer-use evaluation in supplied environments, and automated benchmark construction. FinCUABuildBench shifts the focus to constructing complete, replayable financial CUA tasks.
Financial benchmarks and workflow agents
- Financial benchmarks span report-based QA and numerical reasoning, financial analysis, executable spreadsheet tasks, and professional workflows; FinCUABuildBench evaluates whether construction keeps episode components jointly consistent.The required components include evidence, application state, tools, post-start event, reset logic, and verifier.Source: Related Work · pp. 2–3
Computer-use agents and stateful tool use
- General CUA benchmarks and stateful-tool benchmarks evaluate agent execution in supplied environments, whereas FinCUABuildBench targets synthesis of the environment, task, event, and verifier.The cited work spans realistic web and desktop interaction, enterprise workflows, controllable application and mobile environments, and stateful or policy-constrained tools (Zhou et al. 2024; Xie et al. 2024b; Drouin et al. 2024; Boisvert et al. 2024; Cao et al. 2024; Trivedi et al. 2024; Rawles et al. 2025; Lu et al. 2025; Yao et al. 2025).Source: Related Work · p. 3
Automated benchmark construction
- Automated construction includes human–model, graph-generated, desiderata-driven, multi-agent, and environment-synthesis approaches; FinCUABuildAgent couples financial grounding, replay semantics, and mutation-tested hidden verifiers.AutoBencher, BENCHAGENTS, and Benchmark Agent are instantiated as matched construction baselines; the release contract also couples cross-artifact state and deterministic post-start events (Kiela et al. 2021; Zhu et al. 2024; Yu et al. 2024; Li et al. 2025; Butt et al. 2024; Xiong et al. 2026; Prabhakar et al. 2025; Song et al. 2026; Xie et al. 2026b; Zhang et al. 2026).Source: Related Work · p. 3
FinCUABuildBench
FinCUABuildBench formalizes financial CUA benchmark construction and evaluates whether methods can generate complete episodes across financial workflows and runtime variation.
Problem Formulation
- Each construction request specifies a financial workflow and target capability, admissible sources, application environments, builder tools, and safety, privacy, resource, and release constraints.A candidate episode packages the user instruction, versioned sources, resettable initial state, goals, evaluated agent’s ToolGraph, runtime-event program, hidden verifier, replay manifest, and permissions.Source: Problem Formulation · p. 3
- A candidate qualifies only if it meets every Γ constraint and passes all seven preregistered episode-level hard gates.H-Acc and CapCov are complementary sample- and portfolio-level diagnostics, not part of the episode-level release indicator.Source: Problem Formulation · p. 3
FinCUABuildBench Construction
- FinCUABuildBench defines 576 preregistered slots from 24 financial workflow categories, eight WorkflowCores per category, and three runtime-dynamic profiles.The profiles cover State Update, Tool Failure, and Policy Change. Each event is replayable, changes at least one action required for success, and preserves a feasible completion path.Source: FinCUABuildBench Construction · pp. 3–4, Figure 2 · p. 4
Automation-Point Verification
- Execution is evaluated with semantic automation-point DAGs that check workflow outcomes and dependencies rather than low-level interface actions.Static completion covers baseline deliverables, while dynamic completion covers event transitions and post-event adaptation; additional checks include artifact, evidence, trace integrity, financial grounding, and safety.Source: Automation-Point Verification · pp. 4–5
FinCUABuildAgent
FinCUABuildAgent constructs dynamic financial benchmark episodes through a capability-guided, validation-gated pipeline that refines grounded workflows into executable tasks. Eleven specialist agents coordinate discovery, episode construction, and independent qualification.
FinCUABuildAgent Architecture
- Eleven specialist agents coordinate workflow discovery and grounding, capability and episode design, tool and environment construction, and task qualification.The Factory Controller routes failed candidates for targeted repair; specialists verify financial provenance, balance taxonomy coverage, specify goal conditions and dependencies, map operations to ToolGraphs, and materialize replayable environments.Source: FinCUABuildAgent Architecture · p. 5
- Episodes are released only after independent oracle, reference-solver, and red-team checks confirm hidden-goal validity, solvability after runtime changes, and resistance to errors or shortcuts.The oracle derives verification conditions before observing candidate solutions; the reference solver uses the same interface as evaluated agents, while red-team tests include stale evidence, ignored events, unauthorized effects, and shortcut solutions.Source: FinCUABuildAgent Architecture · p. 5
Typed Agent–Tool Orchestration
- A typed orchestration graph validates artifact types, source lineage, episode versions, dependencies, and authorized tool traces, triggering repair or rollback when contracts fail.Agents A01–A04 discover and ground workflows, A05–A08 compile and materialize episodes, and A09–A11 conduct separate oracle, reference-execution, and red-team assurance pathways.Source: Typed Agent–Tool Orchestration · p. 5
Experiments
FinCUABuildAgent constructs qualified financial CUA tasks that outperform adapted construction baselines, provide differentiated downstream diagnostics, and transfer partially to held-out workflows. Ablations show that staged coordination and distinct construction components contribute to execution, coverage, and verification quality.
Experimental Setup
- The benchmark spans 576 target slots across 24 workflows and three runtime profiles, with downstream tests on 144 qualified episodes and transfer on 90 held-out requests.Comparisons share target specifications, budgets, and evaluation interfaces, with no access to gold ToolGraphs or reference trajectories; strict Qual requires passing every preregistered episode-level hard gate and mandatory constraint. Methods include rule-based, prompting, tool-augmented, adapted-agent, and single-agent baselines, with multiple model backbones depending on setting.Source: Experimental Setup · pp. 5–6
Main Result
- Qwen3.5 achieves 41.9% Qual with FinCUABuildAgent, versus 1.3–8.3% Qual for adapted agentic baselines; across four shared backbones, Dyn rises 29.0–48.2 points and Dup falls 24.9–34.6 points.The paper reports that FinCUABuildAgent has the strongest overall profile on coupled construction metrics; finance tools alone improve provenance and capability coverage, but effects on dynamicity, verifier reliability, and strict qualification remain uneven.Source: Table 1 · p. 6, Main Result · p. 6
Diagnostic Effectiveness of Generated Tasks
- On 144 qualified tasks, native general agents score 0.004–0.043 Dynamic Completion despite 0.622–0.660 Static Completion, while the external structured configuration reaches 0.676 Dynamic Completion.The frozen reference solver succeeds on every selected task, while evaluated agents obtain differentiated scores; the matched SingleController achieves only 0.258 Dynamic Completion despite near-perfect artifact, evidence, static-completion, and cross-artifact scores. Static and Dynamic Completion use complementary, non-nested automation-point sets.Source: Diagnostic Effectiveness of Generated… · p. 6, Table 2 · p. 7
Generalization to Held-Out Workflows
- Held-out transfer yields 13.3% strict qualification despite V-Acc of at least 86.3%; under P3, 46.7% of episodes are executable but only 6.7% qualify.FP&A achieves 56.7% aggregate Exec and 20.0% Qual, versus AP exception handling at 43.3% and 10.0%, and portfolio risk and compliance at 33.3% and 10.0%; results support partial within-taxonomy transfer, not generalization to entirely unseen workflow categories.Source: Generalization to Held-Out Workflows · pp. 6–7, Table 3 · p. 7
Component and Topology Ablations
- Under matched budgets, staged coordination raises Exec from 18.8% to 73.8% versus the collapsed controller, while removing Grounding & Provenance yields 88.5% V-Acc but only 10.4% Exec.Removing the Dependency Compiler, Tool Architect, or Dynamics Composer produces Dup of at least 96.7% and CapCov of at most 1.6%; removing the Independent Oracle lowers V-Acc from 72.3% to 52.0% and L-Acc from 55.7% to 8.9%, while raising Dup from 12.5% to 43.2%. Red-team repair preserves several structural metrics but reduces L-Acc to 0.5% and raises Dup to 20.8%.Source: Table 4 · p. 7, Component and Topology Ablations · p. 7
Conclusion
FinCUABuildBench and FinCUABuildAgent demonstrate a validation-gated approach to constructing dynamic financial CUA tasks, with measurable gains over matched prompting and downstream diagnostic value. Qualification yield, transfer, production coverage, source licensing, and verifier maintenance remain challenges.
Conclusion
- Against matched tool-augmented prompting across four backbones, FinCUABuildAgent improves Dyn by 29.0–48.2 points and strict Qual by 13.8–29.4 points, while reducing Dup by 24.9–34.6 points.Qwen3.5 achieves the highest Qual of 41.9%.Source: Conclusion · p. 7
- FinCUABuildBench provides 576 construction slots across 24 workflow categories and three runtime-dynamic profiles, alongside an agent framework producing grounded, executable, replayable, and verifiable episodes.Source: Conclusion · p. 7
- Generated qualified tasks yield non-saturated, differentiated automation-point completion scores across existing agents, while Dynamic Completion is lower than Static Completion in 16 of 17 configurations.The dynamic-versus-static gap reveals limited adaptation to post-start changes.Source: Conclusion · p. 7
- Held-out evaluation reaches 13.3% strict Qual and shows partial transfer within the covered taxonomy; ablations support staged multi-agent coordination.Source: Conclusion · p. 7
- The results frame benchmark construction as a systems problem requiring joint control over grounding, state, dynamics, and verification, with challenges in qualification yield, broader transfer, production coverage, source licensing, and verifier maintenance.Validation-gated construction provides a practical basis for dynamic financial CUA evaluation.Source: Conclusion · p. 8
Benchmark-Generation Metric Definitions
The section defines Qualified Task Rate (Qual) as the primary end-to-end construction metric, with the other metrics diagnosing benchmark quality. Strict qualification requires every mandatory hard gate and safety-related constraint to pass.
Benchmark-Generation Metric Definitions
- Metrics are defined per construction run, averaged across independent runs when prescribed, macro-averaged across WorkflowCores, and generally reported on a 0–1 scale or as percentages in Table 1.Invalid, unfilled, malformed, or non-materialized slots receive zero on downstream hard gates.Source: Benchmark-Generation Metric Definitions · p. 10
- H-Acc is a method-blind, stratified human audit of correctness, realism, consistency, and capability alignment, while L-Acc uses weighted rubric scores with a minimum-dimension constraint.H-Acc is reported as an external audit and is excluded from strict episode-level qualification.Source: Benchmark-Generation Metric Definitions · p. 10
- Trace measures validity of required provenance atoms, Exec covers environment boot, ToolGraph execution, resets, reference-solver runs, and event replay, and V-Acc reports continuous verifier performance.A missing verifier or required test set contributes zero and sets the thresholded verifier gate to zero.Source: Benchmark-Generation Metric Definitions · p. 10
- Dyn assesses valid dynamic-task behavior around post-start events, while CapMatch assesses alignment between episodes and their target capabilities.The definitions account for valid plans and goals before and after events, event-ignoring versus event-adaptive outcomes, and capability-removal gaps.Source: Benchmark-Generation Metric Definitions · p. 10
- Dup detects similarity across instruction text, workflow and tool graphs, goal/verifier structure, and source identity, while CapCov measures weighted coverage of capabilities meeting required qualified-task counts.Duplicate comparisons use structure-aware dimensions rather than surface text alone.Source: Benchmark-Generation Metric Definitions · p. 11
- Qual is the primary end-to-end metric, and an episode qualifies only when all preregistered hard gates and mandatory safety, privacy, permission, resource, materialization, and forbidden-effect constraints pass.A missing, malformed, or non-materialized component sets its corresponding gate to zero.Source: Benchmark-Generation Metric Definitions · p. 11