Source-linked AI summary
StartupBench: Benchmarking General-Purpose Agents on Market-Validated End-to-End Workflows
Liya Zhu, Xin Ma, Tao Liu, Haodong Wang, Ge Zhang, Jingzhe Ding, Qingshui Gu, Yongjie Zhong, Jinxiang Meng, Yuan Gao, Yunqiu Zhou, Hao Zhu, Jifeng He, Yongzhi Liao, Xinyi Zhang, Chaoxin Li, Yi Zhu, Xi Lin, Duju Zeng, Xiang Gao, Wen Zhang, Yunyang Wang, Duo Wang, Huan Zhou, Zuo Wang, Jin Chen, Kaiyuan Zhang, Chuqian Yu, Tianhao Yu, Longxiang Liu, Jianbo Xue, Huimin Che, Jiahao Wang, Yujia Qin, Jiaheng Liu, Shen Yan, Xiaolong Chang, Wenhao Huang
TL;DR
Existing benchmarks often use researcher-selected tasks, leaving uncertain how well they reflect work users seek to delegate to AI. StartupBench derives tasks from adopted AI-product workflows; the strongest model completes only about 30% of benchmark tasks.
Problem
Benchmarks often define tasks from researchers’ perspectives rather than workflows with demonstrated practical demand, and some deliverable evaluations are coarser than the deliverables.
Source: §1 · p. 1Method
StartupBench uses surveys and interviews to derive end-to-end benchmark tasks from adopted AI-native product workflows and real-world user demands across domains.
Source: §1 · p. 2Results
Across tasks, the strongest model completed only about 30% of benchmark tasks, despite meaningful progress on professional work.
Source: §1 · p. 2, §5 · p. 14Takeaways & Limitations
StartupBench offers a realistic measure of practical agent capabilities on workflows users already seek to delegate to AI.
Source: §5 · p. 14Pilot executions randomly select models to make the benchmark challenging and discriminative, rather than to estimate performance over the natural task distribution.
Source: §3.1.4 · p. 5
Abstract
from arXiv · showhide
Recent advances in Large Language Models(LLMs) and agents have substantially improved the ability of AI systems to execute complex tasks. Yet existing benchmarks largely rely on researcher-selected tasks, leaving uncertain whether such progress extends to the work that real-world users actually demand from AI systems. We introduce \textbf{StartupBench}, an E2E agent benchmark grounded in market-validated AI startup products. Rather than defining tasks from pre-defined assumptions about useful agent capabilities, we systematically study AI products with demonstrated adoption, together with their product workflows and users, to identify real-world tasks for which AI has established practical demand across diverse professional domains. We translate these workflows into complete deliverable-oriented tasks and evaluate them with fine-grained rubrics capturing their complex requirements. Across representative models evaluated under a unified agent harness, even the strongest model successfully completes only approximately 30\% of StartupBench, despite making substantial partial progress on many tasks. Further analysis identifies aspects like complex instruction following and domain-specific expertise as major sources of failure. Our results reveal that many market-validated workflows remain beyond the reliable capabilities of current general-purpose agents, establishing StartupBench as an empirical measure of progress toward E2E completions of real-world user tasks.
1 Introduction
StartupBench evaluates whether general-purpose agents can turn their capabilities into professionally usable deliverables by grounding benchmark tasks in adopted AI-product workflows. It addresses limits in task authenticity and evaluation granularity while measuring end-to-end task completion.
1 Introduction
- Existing benchmarks often use researcher-defined tasks and coarse evaluation, limiting evidence that models meet authentic user requirements for professionally usable deliverables.Real-world deliverables can include many functional, structural, formatting, and domain-specific requirements that holistic evaluation may miss.Source: §1 · pp. 1–2
- StartupBench contains 97 tasks across 6 primary domains, sourced from AI-native startup workflows with demonstrated real-world adoption and demand.The authors use startup surveys, product demonstrations, and user interviews to identify workflows and preserve their core objectives and practical requirements.Source: §1 · p. 2
- StartupBench requires complete user-facing deliverables and evaluates them with independent rubrics for functionality, structure, formatting, and domain-specific quality.Its construction combines startup research, enterprise-user interviews, user-oriented task abstraction, expert data construction, and multi-stage quality control.Source: §1 · pp. 2–3
- The strongest model completes only about 30% of StartupBench tasks, with failures linked primarily to complex instruction following and domain-specific expertise.Performance varies across domains, with Finance, STEM & Computer Science, and Education & Humanities particularly challenging.Source: §1 · p. 2
- The paper proposes a survey- and interview-driven method for building benchmarks from real AI adoption scenarios and presents an empirical study of foundation models and general-purpose agents on these workflows.StartupBench is intended as a practical signal for assessing progress from answering questions toward reliably completing real work.Source: §1 · p. 3
2 Related Work
StartupBench connects market-validated AI product workflows with end-to-end agent evaluation and deliverable-centric judging. It distinguishes itself by sourcing tasks from startup products and user needs, then assessing whether agents produce usable work products.
2.1 Commercial AI agents and Vertical Workflows
- Commercial AI products package agent capabilities into vertical workflows, while StartupBench uses market-validated startups and their deep users to discover tasks.These workflows span document processing, data analysis, financial research, enterprise operations, and knowledge work; academic benchmarks rarely use commercial adoption as a task source.Source: §2.1 · p. 3
2.2 End-to-end evaluation of AI agents
- StartupBench extends end-to-end agent benchmarks by deriving tasks from AI-native startup products, ToB user needs, and expected business deliverables.Prior benchmarks progress from tool use and reasoning [10] [12] to web and computer interaction [27] [34], then to professional and long-horizon tasks [7] [11] [18] [35].Source: §2.2 · p. 3
2.3 Agent-as-a-Judge for complex deliverable evaluation
- StartupBench combines expert rubrics with tool-equipped judge agents that inspect artifacts and evidence to assess whether outputs are usable work products.This deliverable-centric approach builds on open-ended judging such as MT-Bench and Prometheus [8] [33], and agent judging work on evidence acquisition and verification [22] [36].Source: §2.3 · p. 3
3 StartupBench
StartupBench turns workflows already delegated to AI products into realistic, answerable, evaluable, and discriminative tasks through a survey-, interview-, expert-, and review-driven pipeline.
3.1.1 Startup Survey for Market-Validated Agent Scenarios
- The startup survey retains agents with more than USD 1M in funding and evidence of adoption through paid usage or substantial user traction.The resulting pool includes 20+ startup agents for subsequent user interviews.Source: §3.1.1 · pp. 4–5
3.1.2 User Interviews for Scenario and Requirement Discovery
- More than 30 deep users describe each agent’s contexts, goals, inputs, deliverables, success criteria, and constraints, with at least one demo case collected per candidate agent.Source: §3.1.2 · p. 5
3.1.3 Expert Construction of Domain Tasks
- Over 50 domain experts reconstruct authentic user requests as reproducible end-to-end tasks with concrete deliverables, clear success criteria, and fine-grained rubrics.Each task is represented by a natural-language request, a workspace of required files and resources, and weighted evaluation rubrics.Source: §3.1.3 · p. 5
3.1.4 Quality Control
- Every task receives independent review by an additional domain expert; pilot executions calibrate difficulty, exclude consistently high-quality tasks, and prompt revision and re-validation when failures stem from ambiguity, incomplete workspace information, or inadequate criteria.Pilot executions use randomly selected models from GPT-5.5, Seed-2.1-Pro, and GLM-5.1 to create a challenging, discriminative benchmark rather than estimate performance over the natural task distribution.Source: §3.1.4 · p. 5
3.2 Statistics for StartupBench
- 97 real-world workflow tasks span six domains, averaging 25.3 fine-grained rubrics per task across six dimensions and three importance levels.Tasks reflect realistic office workflows and support DOCX, XLSX, PPTX, PDF, Markdown, image, and text-based deliverables.Source: Table 2 · p. 6, §3.2 · p. 6, §3.1.4 · p. 6
3.3 Automatic Evaluation of StartupBench Tasks
- StartupBench evaluates each rubric independently with an AgentJudge agent that returns a binary decision and textual justification, then aggregates weighted rubric outcomes into task scores.A lightweight evidence view combines extracted text and rendered page images while preserving access to original deliverable files.Source: §3.3 · p. 6
4 Experiments
StartupBench experiments show that general-purpose agents make substantial progress on market-validated workflows but often fail to deliver artifacts that meet professional acceptance standards; the evaluation framework is reliable, and changing general-purpose harnesses does little to close the gap with specialized agents.
4.1 Experimental Setup
- Evaluation uses nine closed- and open-source models, three runs each, a shared Nanobot harness, and a 200-step cap; success requires a score of at least 90.The judge uses GPT-5.5, and task scores are reported with 95% bootstrap confidence intervals from 10,000 resamples.Source: §4.1 · p. 7
4.2 Main Results
- Kimi-K3 and GPT-5.6-sol average 73.67% and 73.61%, respectively, yet no model completes even one third of tasks at score ≥90.Across models, success rates are substantially lower than average scores, indicating that partial workflow progress often fails to produce a deliverable meeting professional standards.Source: §4.2 · pp. 7–8, Table 3 · p. 8
- Business is the easiest domain and Finance the hardest, while leading models’ strengths differ by domain rather than one model dominating consistently.Finance has the lowest average score at 54.48%; Kimi-K3 leads in Medical and Business success, GPT-5.6-sol in Finance, STEM, and Education success, and GPT-5.5 in Legal success.Source: §4.2 · pp. 8–9
4.3 Effectiveness of Proposed Evaluation Framework
- Rubric-level automatic judging agrees with experts on 92.78% of rubric judgments and 92.84% of success decisions (success requires an aggregated score reaching 90), compared with 83% when one judge scores all rubrics end-to-end.Holistic evaluation requires more than two runs on average before producing a valid result.Source: §4.3 · p. 9
4.4.1 Outcome-level Failure Analysis
- Models score weakest on Domain-Specific Compliance and Calculation Precision, and satisfy Auxiliary rubrics more often than Core ones, often missing requirements central to task completion.Mean satisfaction is 68.67% for Auxiliary, 65.89% for Important, and 63.45% for Core rubrics; no model achieves perfect output-format compliance on 56 tasks, with rates ranging from 97.6% to 86.3%.Source: §4.4.1 · pp. 9–10, Figure 4 · p. 10
4.4.2 Behavior-level Failure Analysis
- Behavioral failures include partial instruction compliance, unverified deliverables, and insufficient domain expertise, with distinct correctness demands across professional fields.Models may mistake confidence in their execution for evidence that the artifact meets requirements; for example, a polished payroll workbook lacked cached values needed for verification. Failure patterns also vary across domains, from finance’s numerical precision and temporal consistency to legal reasoning and medical safety.Source: §4.4.2 · pp. 11–12
4.5.1 Effect of General-Purpose Harness
- Across Nanobot, Hermes, and Claude Code, average best-to-worst framework variation is 1.79 points, and model ordering is unchanged.The largest reported variation is 2.68 points for GPT-5.5, supporting the finding that benchmark performance primarily reflects underlying model capability rather than framework choice.Source: Table 6 · p. 13, §4.5.1 · p. 13
4.5.2 General-Purpose Agents versus Specialized Agents
- General-purpose agents score 64.26 with 19.74% success, versus specialized agents at 83.50 and 39.18%; even oracle selection leaves gaps of 11.75 points and 11.12 percentage points.The oracle keeps each general-purpose model–task pair’s best of three trials, yielding 71.75 average score and 28.06% success, still below specialized agents.Source: §4.5.2 · p. 14, Table 7 · p. 14
5 Conclusion · Appendix
StartupBench evaluates end-to-end professional workflows derived from market-validated AI-native products and real-world user demands. Its results show that meaningful progress on professional work often falls short of deliverables meeting practical acceptance criteria.
5 Conclusion
- StartupBench benchmarks models on end-to-end professional workflows derived from market-validated AI-native products and real-world user demands.Source: §5 · p. 14
- Across diverse domains and multiple tasks, models often make meaningful progress without producing deliverables that satisfy practical acceptance criteria.Source: §5 · p. 14
- The completion gap is driven primarily by limitations in complex instruction following and domain-specific expertise.Source: §5 · p. 14
- Professional conventions and long-horizon workflow execution are also major sources of the gap between progress and acceptable deliverables.Source: §5 · p. 14
- By evaluating workflows users already seek to delegate to AI, StartupBench offers a realistic measure of practical agent capabilities.Source: §5 · p. 14
- StartupBench highlights concrete directions for improving models and agents so they can reliably deliver valuable professional work ready for direct use.Source: §5 · p. 14
A Tutorial of StartupBench Task Annotation · A.1 Task Construction
StartupBench defines each task as a query, a self-contained workspace, and fine-grained evaluation rubrics, preserving realistic workflow structure while enabling systematic assessment.
A.1 Task Construction
- Each StartupBench task is defined as a triple of input query, task workspace, and evaluation rubrics.Experts provide all three elements.Source: §A.1 · p. 19
- The natural-language query reflects authentic user intent, including task background, implicit constraints, and expected deliverables.It is derived from authentic work scenarios of annotators.Source: §A.1 · p. 19
- The workspace gives the agent a self-contained execution environment with necessary source files, resources, and interaction settings.Controlled conditions support reproducible evaluation while preserving realistic workflow structure.Source: §A.1 · p. 19
- Each rubric is a checklist of natural-language scoring points with positive importance weights reflecting their contribution to task completion.Multiple items assess different dimensions of complex, multifaceted workflows.Source: §A.1 · p. 19
- StartupBench categorizes its rubrics into 6 dimensions and 3 importance levels.The dimensions and levels are detailed in Appendix C.Source: §A.1 · p. 19
- Tasks may involve information synthesis, constraint following, structured generation, or domain-specific reasoning and judgment.Together, the task components preserve real-world workflow structure while enabling systematic and objective evaluation.Source: §A.1 · p. 19
A.2 Cross Validation · B Domain Expert Background
Cross-validation by domain experts checks task quality from multiple perspectives, while a 57-person expert pool spans all six StartupBench domains and is assigned according to expertise and experience.
A.2 Cross Validation
- Each task receives independent review by at least one additional domain expert, who checks realism and correctness from complementary perspectives.Review extends beyond annotation consistency.Source: §A.2 · p. 19
- Reviewers verify that task requests and workspaces are authentic, materials are factually correct, and domain-specific knowledge contains no errors.Source: §A.2 · p. 19
- Reviewers assess whether tasks reflect real professional workflows and responsibilities, revising or discarding those that depart from practical procedures.Source: §A.2 · p. 19
- Reviewers check that rubrics assess meaningful workflow capabilities, use clear and unambiguous items, and weight items according to task priorities.Source: §A.2 · p. 19
- Ground-truth deliverables must earn full rubric scores, and reviewers revise the reference solution and criteria until they are consistent.Source: §A.2 · p. 19
B Domain Expert Background
- The 57 recruited experts cover all six StartupBench domains, spanning fields from clinical medicine and law to software, business, and humanities.Experts are assigned according to reported expertise and professional experience to support construction and review in familiar domains.Source: B Domain Expert Background · p. 20
- The expert pool has a median of 5 years’ experience; 68.4% have at least 3 years, 42.1% at least 6 years, and 24.6% at least 10 years.Experts are compensated based on the actual workload of construction and review.Source: B Domain Expert Background · p. 20
C Rubrics of StartupBench
StartupBench evaluates deliverable quality through six cross-domain rubric categories and weights rubric requirements by their importance to task completion.
Rubric Dimensions
- The six domain-agnostic categories include Structure & Completeness, Calculation Precision, and Information Integration, covering deliverable completeness, objective correctness, and structured data workflows.The taxonomy is designed to support interpretable analysis across different task types.Source: C Rubrics of StartupBench · p. 20
- The remaining categories are Domain-Specific Compliance, Engineering & Format, and Output & Presentation, assessing professional requirements, implementation and formatting, and visual communication.Source: C Rubrics of StartupBench · p. 20
- 38.9% of 2,453 rubrics assess Calculation Precision, while Structure & Completeness and Domain-Specific Compliance together account for 42.8%.Calculation Precision is the largest category, reflecting the importance of objectively correct work products alongside complete deliverables and professional domain reasoning.Source: Table 9 · p. 21, C Rubrics of StartupBench · p. 21
Rubric Importance Levels
- Core, Important, and Auxiliary rubrics receive weights of 5, 3, and 1 and average 59.10%, 34.33%, and 6.57% of task weight, respectively.Core requirements directly determine whether user needs are met; Important requirements materially affect quality; Auxiliary requirements mainly affect details, experience, or polish.Source: C Rubrics of StartupBench · p. 21
D Examples of Behavior-level Failure Modes · D.1 Self-Verification Hallucination
A seemingly complete workbook contained acceptance-critical errors because the model treated a successful execution summary as validation instead of independently checking the artifact against the task requirements.
D.1 Self-Verification Hallucination
- The model exhibited self-verification hallucination by trusting its execution summary rather than checking the completed workbook against requirements, leaving incorrect point values and member names unnoticed.The task required filtering level-11 members, simulating route check-ins to the level-12 threshold, and producing an Excel workbook with precise value and formatting constraints.Source: §D.1 · p. 21
D.2 Domain-Specific Failure · Self-Verification Hallucination
The examples show two distinct failure modes: clinically polished plans can violate safety-critical constraints, while plausible workbooks can be declared complete without artifact-level verification.
D.2 Domain-Specific Failure
- Clinically professional formatting does not prevent a treatment plan from violating multiple safety-critical constraints.Source: §D.2 · p. 21, Figure 9 · p. 24
- The clinical failure reflects difficulty translating medical knowledge into actionable, clinically reliable decisions, rather than poor writing.Source: §D.2 · p. 21
- The rubric example separates task evaluation into core, important, and auxiliary criteria.Core criteria assess essential financial reasoning and calculations; important criteria assess supporting analysis, while auxiliary criteria cover presentation and additional quantitative support.Source: Figure 7 · p. 22
Self-Verification Hallucination
- A plausible workbook can lead the model to conclude that it completed the task based on its execution process.Source: Figure 8 · p. 23
- Artifact-level spot checks can expose acceptance-critical errors in the final workbook.Source: Figure 8 · p. 23
- The model mistakes an execution-progress summary for verification of the deliverable against the original task requirements.Source: Figure 8 · p. 23