Source-linked AI summary
FinCUABuild: Can Agents Build Reliable Benchmarks for Dynamic Financial Computer Use?
Jingpu Yang, Fengxian Ji, Jinri Guo, Tianhao Li, Qian Jiang, Fan Zhang, Min Peng, Qianqian Xie, Preslav Nakov, Zhuohan Xie
TL;DR
Financial CUA task construction is difficult to scale because existing tasks are largely manually built and construction methods lack unified evaluation. This paper introduces a benchmark and a validation-gated multi-agent framework for constructing dynamic financial episodes, achieving 31.3% strict qualification versus 1.3–8.3% for general-purpose construction agents. The qualified tasks also differentiate downstream CUA capabilities, while broader transfer and production coverage remain open challenges.
Problem
Existing financial CUA tasks are costly to construct, while no unified benchmark evaluates construction across diverse workflows, runtime variations, fair conditions, and complete task quality.
Method
FinCUABuildBench standardizes 576 construction requests, inputs, budgets, outputs, and qualification checks, while FinCUABuildAgent jointly constructs grounded tasks, environments, and verifiers with auditing and repair.
Results
31.3% strict qualification was achieved by FinCUABuildAgent versus 1.3–8.3% for general-purpose construction agents, and 144 qualified tasks differentiated downstream CUAs.
Takeaways & Limitations
The qualified tasks provide meaningful downstream diagnostic value, supporting automated construction as a practical basis for broader dynamic financial CUA evaluation.
Takeaways & Limitations
Remaining challenges include qualification yield, broader transfer, production coverage, source licensing, and verifier maintenance.
Abstract
from arXiv · showhide
Financial scenarios are diverse and complex, spanning varying data conditions, tool configurations, and workflows. Yet existing CUA, Computer-Using Agent, evaluation tasks remain largely manually constructed, limiting scalable coverage of real-world financial scenarios. Then, can agents autonomously construct diverse CUA evaluation tasks for financial scenarios? Evaluating this capability poses three key challenges: scenario coverage of construction requests, fair comparison across construction methods, and reliable assessment of generated task quality. To solve these, we introduce FinCUABuildBench, a benchmark for evaluating financial CUA task construction, featuring: (i) 576 construction requests covering 24 financial workflows and three types of runtime variation; (ii) standardized input, budget, and output specifications; and (iii) a task qualification mechanism based on execution tests and quality checks. We further introduce FinCUABuildAgent, a multi-agent system for automatically constructing dynamic financial CUA evaluation tasks. It consists of three modules that jointly construct tasks, environments, and validators. On FinCUABuildBench, under the same model backbone, existing agent-based construction methods achieve strict qualification rates of only 1.3-8.3%, while FinCUABuildAgent reaches 31.3%. Downstream evaluations further show that the constructed tasks can effectively differentiate CUA task-execution capabilities. These results demonstrate that agents can autonomously construct financial CUA tasks with meaningful evaluation value, offering a practical path toward broader evaluation coverage in financial scenarios. Code: https://github.com/FengxianJi/FinCUABuild
Dynamic Financial Computer Use?
The paper lists Jingpu Yang, Fengxian Ji, and additional coauthors.
- Jingpu Yang and Fengxian Ji are marked as equal contributors.
- The author list includes contributors from affiliations numbered 1 through 4.
- Qianqian Xie and Zhuohan Xie are identified as corresponding authors.
Introduction
The introduction identifies a scalability gap in manually constructed financial CUA tasks and presents FinCUABuildBench and FinCUABuildAgent as a coordinated response. The benchmark standardizes construction and qualification, while the framework jointly builds and repairs task packages.
- Financial workflows span diverse data conditions, tool combinations, and business processes, while prior construction still required over 700 hours of expert annotation.
- Existing financial benchmarks assess predefined task completion, but no unified benchmark evaluates financial CUA task construction.
- Construction requests must cover genuine workflow and runtime variation, standardize comparison conditions, and assess complete executable task packages.
- FinCUABuildBench contains 576 requests from 24 workflow categories and three runtime variations, with standardized inputs, budgets, outputs, and qualification checks.
- FinCUABuildAgent compiles capability targets into requirements, independently audits quality, and repairs cross-stage errors while jointly constructing tasks, environments, and verifiers.
- The qualified-task collection supports downstream differentiation of CUA capabilities but still indicates substantial room for improved generalization.
Related Work
Related work spans financial reasoning and spreadsheet benchmarks, general computer-use environments, and automated benchmark-construction systems. The paper positions its contribution at their intersection: dynamic financial task construction with executable environments and verification.
- Financial benchmarks progressed from report-based question answering and numerical reasoning toward knowledge-intensive analysis and executable spreadsheet or enterprise workflows.
- General CUA benchmarks cover web, desktop, enterprise, mobile, and controllable application interactions, including stateful and policy-constrained tool use.
- Automated benchmark construction includes human–model loops, graph-generated evaluations, desiderata optimization, coordinated generation and verification, and synthesized tool environments.
FinCUABuildBench
FinCUABuildBench defines complete dynamic financial episodes across workflow categories, capability paths, and runtime profiles. Its formal episode contract and automation-point evaluation emphasize grounded execution, replayability, event adaptation, and verification.
- Problem Formulation: Each construction request specifies a financial workflow and target capability within admissible sources, application environments, builder tools, and release constraints.
- Problem Formulation: A generated episode contains an instruction, source bundle, resettable state, goal predicates, ToolGraph, runtime-event program, hidden verifier, replay manifest, and permissions.
- Problem Formulation: Qualification requires every mandatory constraint and hard gate, including grounded accuracy, traceability, execution, verifier passing, dynamicity, capability matching, and nonduplication.
- FinCUABuildBench Construction: The benchmark spans 24 workflow categories, eight WorkflowCores per category, and three runtime profiles, producing 576 preregistered target slots.
- FinCUABuildBench Construction: Runtime profiles cover state updates, tool failures, and policy changes, with deterministic replay and at least one feasible completion path.
- FinCUABuildBench Construction: Downstream evaluation samples 144 qualified episodes, all passing independent execution, replay, verifier, event-validity, and safety audits.
- Automation-Point Verification: Automation-point DAGs represent grounded facts, artifact updates, cross-tool transfers, and runtime adaptation while excluding low-level interface actions.
- Automation-Point Verification: Static completion measures baseline deliverables, whereas dynamic completion measures event-transition and post-event adaptation points.
FinCUABuildAgent
FinCUABuildAgent uses a capability-guided, validation-gated multi-agent pipeline to turn grounded financial workflows into executable dynamic episodes. Specialist agents coordinate discovery, grounding, compilation, materialization, qualification, and assurance through typed handoffs, authorized tools, and repair of failed contracts.
- Pipeline overview: The pipeline first refines a preregistered target slot into a grounded WorkflowCore, then compiles it into an executable episode.This capability-guided process is summarized as the core construction sequence.
- Pipeline overview: Eleven specialist agents coordinate workflow discovery, financial grounding, capability compilation, episode design, tool synthesis, verification, reference solving, and red-team testing.The Factory Controller routes failed candidates to the responsible stage within a generate–audit–repair workflow.
- Validation and repair: Typed handoffs validate artifact types, lineage, episode versions, dependencies, and authorized tool traces, returning failed contracts to the responsible agent for repair.A downstream stage proceeds only when its artifact and tool-use trace satisfy validation.
- Workflow discovery and grounding: Agents A01–A04 identify and ground candidate workflows while enforcing licensing and deduplication constraints before WorkflowCores enter the graph.Their role includes retrieving relevant source entities and retaining only consistent, non-redundant WorkflowCores.
- Episode compilation and materialization: Agents A05–A08 compile executable episodes by assembling goals, states, dependencies, schemas, runtime events, and interface adapters before fixing the initial environment state.Contract, permission, and runtime-conformance checks precede environment instantiation.
- Qualification and assurance: Agents A09–A11 qualify episodes through separate oracle, reference-execution, and red-team pathways that test solvability, stale evidence, shortcuts, leakage, and unauthorized effects.The oracle reasons over hidden goal states, the reference solver uses episode-visible interfaces, and the red-team evaluator probes invalid behavior.
Experiments
The experiments compare construction methods under standardized conditions, test downstream diagnostic value and held-out transfer, and analyze component and topology effects. FinCUABuildAgent achieves the strongest overall construction profile, while generated tasks differentiate runtime adaptation and expose distinct failure modes.
- Evaluation Scope: 576 target slots span 24 financial workflow categories and three dynamic profiles, with downstream evaluation on 144 qualified episodes and held-out transfer on 90 episodes.The dynamic profiles are State Update, Tool Failure, and Policy Change; downstream agents receive three independent runs per task.
- Evaluation Protocol: All methods share target specifications, resource budgets, and evaluation interfaces, while strict Qual requires every preregistered episode-level hard gate and mandatory constraint to pass.Tool access is fixed within each matched comparison, and methods receive neither gold ToolGraphs nor reference trajectories.
- Construction Quality: FinCUABuildAgent raises Dyn by 29.0–48.2 points, reduces Dup by 24.9–34.6 points, and improves strict Qual by 13.8–29.4 points versus tool-augmented prompting across four shared backbones.The paper reports the strongest overall profile on the coupled construction metrics.
- Downstream Diagnostic Value: Generated tasks are feasible and discriminative: the frozen reference solver succeeds on every selected task, while evaluated agents obtain non-saturated and differentiated scores.Native general-agent frameworks achieve only 0.004–0.043 Dynamic Completion despite 0.622–0.660 Static Completion; external structured configuration reaches 0.676 Dynamic Completion.
- Held-Out Transfer: Held-out transfer is partial: V-Acc remains at least 86.3% in every workflow–profile cell, but only 13.3% of episodes strictly qualify.Under P3, 46.7% of episodes are executable but only 6.7% strictly qualify; aggregate Qual is 20.0% for FP&A versus 10.0% for both AP exception handling and portfolio risk and compliance.
- Ablations: 73.8% Exec under staged coordination versus 18.8% for the collapsed controller shows that typed handoffs improve consistency across goals, ToolGraphs, environment states, and verification logic.Removing Grounding & Provenance leaves 88.5% V-Acc but only 10.4% Exec, while removing the Dependency Compiler, Tool Architect, or Dynamics Composer produces Dup of at least 96.7% and CapCov of at most 1.6%.
- Ablations: Removing the Independent Oracle lowers V-Acc from 72.3% to 52.0% and L-Acc from 55.7% to 8.9%, while raising Dup from 12.5% to 43.2%.Removing red-team repair preserves several structural metrics but reduces L-Acc to 0.5% and raises Dup to 20.8%.
Conclusion
FinCUABuildBench and FinCUABuildAgent support automated construction of dynamic financial CUA tasks, while evaluation shows both meaningful diagnostic value and substantial remaining limitations.
- FinCUABuildBench contains 576 construction slots across 24 workflow categories and three runtime-dynamic profiles, while FinCUABuildAgent produces grounded, executable, replayable, and verifiable episodes.
- Remaining challenges include qualification yield, broader transfer, production coverage, source licensing, and verifier maintenance.
Benchmark-Generation Metric Definitions
The benchmark reports macro-averaged construction metrics, with strict qualification requiring every mandatory episode-level gate and constraint to pass. Complementary human, verifier, traceability, execution, dynamicity, capability, and redundancy metrics diagnose distinct aspects of task quality.
- Metrics are computed per construction run, macro-averaged over WorkflowCores, and generally lie in [0, 1] before conversion to percentages in Table 1.
- Human Acceptance Rate is a stratified, method-blind external audit and is excluded from the strict episode-level qualification conjunction.It assesses financial correctness, workflow realism, cross-component consistency, and target-capability alignment.
- Verifier accuracy is reported as a continuous mean, whereas strict qualification uses a thresholded verifier indicator; missing verifiers or required test sets contribute zero.
- Duplicate Task Rate compares instruction text, WorkflowGraph, ToolGraph, goal/verifier structure, and source identity using fixed weighted similarities rather than surface text alone.
- A candidate qualifies only when every mandatory constraint and preregistered hard gate—including L-Acc, Trace, Exec, verifier pass, Dyn, CapMatch, and nonduplication—passes.Missing, malformed, or non-materialized components set their corresponding gates to zero.
- Qualified Task Rate is the primary end-to-end construction metric, while the remaining metrics diagnose individual aspects of benchmark quality.