PaperlayerView exact revision on arXiv

Source-linked AI summary

SaaSBench: Exploring the Boundaries of Coding Agents in Long-Horizon Enterprise SaaS Engineering

Qingnan Ren, Shun Zou, Shiting Huang, Ziao Zhang, Kou Shi, Zhen Fang, Yiming Zhao, Yu Zeng, Qisheng Su, Lin Chen, Yong Wang, Zehui Chen, Xiangxiang Chu, Feng Zhao

arXiv:2605.17526v1cs.SEcs.AI

TL;DR

Existing benchmarks do not adequately evaluate coding agents on heterogeneous, enterprise-level SaaS systems built from scratch. SaaSBench addresses this gap with dependency-aware evaluation and finds that end-to-end SaaS construction remains far from solved, with the best result reaching only 20.68%.

  • Problem

    Existing project- and repository-level benchmarks remain limited in reflecting the holistic capabilities required for end-to-end software development.

  • Method

    SaaSBench evaluates end-to-end enterprise SaaS development using a dependency-aware hybrid paradigm with executable validation chains and prerequisite gating.

  • Results

    20.68% is the best result, while current agents still struggle to reliably generate complete enterprise-level SaaS systems from scratch.

  • Takeaways & Limitations

    SaaSBench provides a foundation for improving coding agents toward practical enterprise-level software development.

  • Takeaways & Limitations

    SaaSBench may not cover all variants of enterprise software construction or exhaust all qualitative factors involved in engineering decisions.

Abstract

from arXiv · show

As autonomous coding agents become capable of handling increasingly long-horizon tasks, they have gradually demonstrated the potential to complete end-to-end software development. Although existing benchmarks have recently evolved from localized code editing to from-scratch project generation, they remain confined to structurally simplified, single-stack applications. Consequently, they fail to capture the heterogeneous environments, full-stack orchestration, and system-level complexity of real enterprise Software as a Service (SaaS) systems, leaving a critical gap in assessing agents under realistic engineering constraints. To fill this gap, we introduce SaaSBench, the first benchmark designed to explore the boundaries of AI agents in enterprise SaaS engineering. Spanning 30 complex tasks across 6 SaaS domains with 5,370 validation nodes, it incorporates 8 programming languages, 6 databases, and 13 frameworks to meticulously mirror real-world software heterogeneity. Furthermore, we design a dependency-aware hybrid evaluation paradigm tailored for complex systems with long horizons and multi-component coupling, enabling fine-grained, reproducible assessment. Crucially, our extensive experiments reveal a striking insight: the primary bottleneck for state-of-the-art agents is not generating isolated code logic, but successfully configuring and integrating a multi-component system. Over 95\% of task failures occur before agents even reach deep business logic, with models often falling victim to overconfidence and prematurely halting during foundational system setup, or getting trapped in ineffective debugging loops. We hope SaaSBench serves as a practical and challenging testbed to drive the evolution of reliable, system-level coding agents. The code is available at \url{https://github.com/ShadeCloak/SaaSbench}.

1 Introduction

SaaSBench addresses the gap between localized coding benchmarks and realistic end-to-end enterprise SaaS development by evaluating agents on complex systems from scratch. Its dependency-aware hybrid evaluation provides reproducible assessment, while experiments reveal substantial limitations in long-horizon planning and cross-component coordination.

  • Motivation: Existing benchmarks emphasize localized tasks or simplified from-scratch projects, limiting their ability to measure holistic end-to-end software development.These settings are better suited to short-horizon and localized engineering behaviors than complete system development.
  • Evaluation: Its dependency-aware hybrid paradigm models validation as a DAG of executable checking chains with prerequisite gating, failure propagation control, and binary, weighted, and llm-as-judge scoring.This design targets long-horizon, highly interactive end-to-end system development tasks.
  • Findings: Experiments show substantial capability gaps even among state-of-the-art coding agents, particularly in long-horizon task planning and cross-component coordination.These findings motivate further improvement of coding-agent capabilities.
  • SaaSBench: SaaSBench evaluates coding agents’ ability to generate and deploy enterprise-level SaaS systems from scratch across mainstream software development markets.The benchmark is introduced as a platform for enterprise-level SaaS engineering evaluation.
  • Evaluation: The evaluation comprehensively covers key engineering dimensions across the real software development lifecycle while providing a reproducible and reliable mechanism.The dimensions are intended to verify essential engineering aspects throughout development.

2 Related Work

Related work progresses from isolated function generation to repository-level assistance and, more recently, from-scratch project construction. However, existing evaluations still leave a substantial gap from these settings to real enterprise-level SaaS systems.

  • Code-Centric Agent Benchmarks: Coding-agent benchmarks evolved from function-level generation in isolated settings to multi-file completion, editing, and issue fixing in real repositories.Early benchmarks include HumanEval, MBPP, APPS, and CodeContests; later benchmarks include RepoBench and SWE-Bench.
  • Code-Centric Agent Benchmarks: Recent benchmarks require agents to build complete code repositories or projects from scratch.This line of work expands evaluation beyond incremental repository modifications toward project-level generation.
  • Code-Centric Agent Benchmarks: NL2Repo-Bench, PRDBench, RepoGenesis, and ProjDevBench evaluate project or repository generation using specifications, PRDs, web microservices, diagnostics, or code review.These benchmarks differ in their inputs and feedback: specification documents, product requirements documents, web microservice targets, Online Judge signals, and LLM-based review.
  • Code-Centric Agent Benchmarks: Despite progress in repository-level and project-level evaluation, a substantial gap remains between existing benchmark settings and real enterprise-level SaaS systems.The passage characterizes this as a remaining limitation of prior work rather than a solved evaluation problem.

3 SaaSBench

SaaSBench is constructed from real-market SaaS domains and maintained seed repositories through detailed PRDs, ambiguity-resolution knowledge bases, standardized environments, and dependency-aware executable tests. It evaluates autonomous agents on heterogeneous, system-level SaaS tasks spanning diverse technologies and complex prerequisite structures.

  • SaaS Domain Definition and Seed Repository Selection: SaaSBench selects stable commercial SaaS domains with distinct technical challenges, preserving grounding in real markets while maintaining engineering-pattern diversity.Domains are defined using industry taxonomies, commercial product landscapes, and domain-expert consultation.
  • SaaS Domain Definition and Seed Repository Selection: Seed repositories must show continuous maintenance, community activity, complete SaaS system structure, and a clear primary business boundary.Annotators additionally conduct cold-start validation to ensure repository independence.
  • PRD and KB Construction: The benchmark converts repository analyses into comprehensive PRDs and ambiguity-resolution knowledge bases that preserve evaluation-sensitive behavioral details.Analyses cover code structure, configuration, routes, data models, tests, and business logic; KB records address details such as default pagination and deletion behavior.
  • DAG-based Hybrid Evaluation: The DAG-based hybrid evaluation paradigm models validation units and prerequisite dependencies, combining executable checks such as HTTP requests, authentication, and rubric-based LLM judgment.Test nodes are compiled from verifiable PRD requirements, aligned with the KB, and arranged to reflect real business workflows.
  • Benchmark Characteristics: 30 tasks cover 6 high-level domains and 30 fine-grained categories, with 4,363 PRD lines on average, 5,370 executable validation nodes, 6,167 prerequisite dependency edges, 8 programming languages, 6 database systems, 5 frontend frameworks, and 8 backend frameworks.Tasks typically combine frontend interfaces, backend APIs, persistent data models, role-based permissions, and deployment configurations.

4 Experiments

SaaSBench evaluates eight LLMs through OpenHands and Claude Code using average pass@1 and node coverage across six SaaS domains. Results show that current agents struggle with end-to-end enterprise SaaS construction, with performance varying substantially by domain.

  • Evaluation Setup: Eight open- and closed-source models are evaluated through OpenHands and Claude Code, with Claude Sonnet 4.5 serving as the temperature-0 rubric judge.The evaluated models include GPT-5.4, Gemini 3.1 Pro, Claude Opus 4.7, Kimi K2.6, Qwen 3.6 Plus, DeepSeek V4 Pro, GLM 5.1, and MiniMax M2.7.
  • Evaluation Setup: Evaluation reports average pass@1 across tasks and domains, together with node-coverage rate measuring validation nodes reaching Passed.The six domains are Customer & Growth, Productivity & Collaboration, Commerce & Finance, Data & Content Infrastructure, Security, Identity & Infrastructure, and Domain & Workflow Platforms.
  • Overall Performance: 20.68% is the best result, achieved by Claude Opus 4.7 under Claude Code.Claude Code averages 11.64%, while OpenHands averages 9.26%, indicating that current agents struggle to generate complete enterprise SaaS systems from scratch.
  • Domain Performance: Agents perform relatively better on SI and DCI but consistently weaker on CF and PC.The pattern suggests infrastructure-oriented tasks with clear structures and interface boundaries are easier than tasks involving complex business semantics and long-horizon state dependencies.

5 Fine-grained Analysis

SaaSBench reveals that agent performance depends heavily on framework-mediated execution and long-horizon interaction, while most failures arise during foundational system setup rather than deep business logic. Agents frequently either stop too early or continue ineffective debugging, leaving generated stacks unstable or structurally incomplete.

  • Agent frameworks: Agent frameworks substantially change performance for the same underlying model, with commercial IDE-based frameworks often outperforming open-source alternatives.The gap mainly reflects differences in tool-call, context, and execution-feedback management.
  • Interaction behavior: More interaction steps do not guarantee higher system completion; success depends on using environmental feedback, identifying root causes, and resolving integration failures.OpenHands interaction analysis tracks steps, token consumption, and execution time for end-to-end SaaS construction.
  • Interaction behavior: 36 steps yielded 7.44% for GPT-5.4, indicating insufficient interaction, premature convergence, and limited long-horizon debugging or system validation.The result suggests strong single-step generation quality but inadequate persistence through complex system construction.
  • Interaction behavior: 279 steps yielded 6.78% for MiniMax M2.7, indicating ineffective long-horizon interaction marked by repeated debugging, local patching, or inefficient exploration.Longer execution alone did not translate into successful system completion.
  • Execution trajectories: 63.5% of capability units never reached stable execution, 32.1% were superficially accessible but structurally incomplete, and only 3.8% made business logic the main bottleneck.These trajectories show that SaaSBench primarily tests process discipline, deployment stability, dependency management, schema correctness, and reproducible execution.

6 Conclusion

SaaSBench evaluates coding agents on developing and deploying enterprise-level SaaS systems from scratch. Experiments show that even Claude Opus 4.7 performs poorly on complete from-scratch system delivery.

  • Contributions: SaaSBench is a comprehensive benchmark for evaluating coding agents developing and deploying enterprise-level SaaS systems from scratch.The benchmark is intended to support future coding-agent development and advance practical “Vibe Coding”.
  • Findings: Even Claude Opus 4.7, the strongest evaluated model, performs poorly at delivering a complete system from scratch.This finding comes from the paper’s extensive experiments.
  • Implications: SaaSBench is positioned as a foundation for developing more capable coding agents and moving the field toward practical “Vibe Coding”.The authors state this as the benchmark’s intended broader contribution.

A Benchmark Details and Statistics … B.5 Primitive Taxonomy of DAG Validation Nodes

SaaSBench defines a market-grounded, heterogeneous benchmark of 30 SaaS tasks and evaluates autonomous coding agents with dependency-aware, multi-method validation. Its framework emphasizes reproducible system-level assessment, from constrained inference and score aggregation to primitive-level DAG validation.

  • A.1 Domain Selection Principles: 421 of 435 category pairs are fully orthogonal, while the remaining 14 have bounded local overlap and no pair falls below 3/4 distinct dimensions.Orthogonality is audited across business objects, user roles, data read/write patterns, and architectural challenges.
  • A.1 Domain Selection Principles; A.2 SaaS Domain Statistics: 30 tasks span six MECE macro families, with each category assigned to exactly one family while the families collectively cover the benchmark task space.The macro families contain 7, 6, 4, 5, 4, and 4 tasks respectively.
  • A.3 Task Statistics: 30 final seed repositories are selected by retaining one representative repository per category for PRD writing, environment preparation, and evaluation construction.A candidate pool is also released so the benchmark can expand when additional construction budget becomes available.
  • B.1 Inference Configuration: 500 reasoning–action steps and 10,800s bound each rollout, while individual tool calls can run for 900s and transient network failures receive up to five retries.Results use pass@1 with a single rollout per task and no oracle-style retries after failure.
  • B.2 Score Aggregation; B.4 Metric Detail: Per-task scores normalize achieved node scores by task-specific total_maxScore, and the benchmark score is the unweighted mean across 30 tasks with node coverage additionally reported.Task total_maxScore values range from 174 to 1,489, preventing larger DAGs from dominating the headline score.
  • B.3 Evaluated Agents: Three coding-agent frameworks are evaluated with a single underlying LLM, under autonomous in-container execution without human steering during the rollout.The evaluated frameworks are OpenHands, Codex CLI, and Claude Code; exact backend snapshots are recorded for reproducibility.
  • B.4 Metric Detail: Binary nodes require complete success, weighted nodes award rounded partial credit for completed primitives, and LLM-as-judge nodes handle targets that deterministic checks cannot characterize.Dependency-skipped nodes score 0 while retaining their maxScore denominator, so foundational failures cascade without being re-charged at every downstream node.
  • B.5 Primitive Taxonomy of DAG Validation Nodes: 18,196 primitive calls across the 30 task DAGs are drawn from a fixed library including P01–P29, RENDER_DOM, SCREENSHOT, and P_INGEST.Table 12 groups these primitives by purpose and reports node-level invocation counts.

B.6 Six Evaluation Backbones and Category Mapping

SaaSBench maps fine-grained validation categories onto six orthogonal evaluation backbones covering deployment, data, APIs, business logic, authorization, and quality. A deterministic, released mapping fully covers all 5,370 validation nodes and preserves independent per-backbone diagnosis.

  • Backbone Definitions: The six backbones capture orthogonal enterprise SaaS implementation axes: Deploy, Data, API, Logic, AuthZ, and Quality.They span runnable deployment, persistent state, system protocols, domain behavior, security decisions, and production-readiness concerns.
  • Backbone Definitions: Logic covers domain-specific behavior such as orders, billing, subscriptions, gamification, moderation, conferencing, dashboards, plugins, collaboration, and content templates.Its source categories include BusinessLogic*, Workflow, Cron*, Async*, and e-commerce, billing, and gamification families.
  • Backbone Definitions: Quality captures non-functional and presentation concerns, including architecture, frontend rendering, edge-case robustness, error handling, internationalization, and administrative experience.Its source categories include ArchitectureQuality, Architecture, Frontend*, UI*, EdgeCases, ErrorHandling, AdminPanel, AdminUI, Internationalization, Localization, and ConfigAndAdmin.
  • Mapping Algorithm: The mapping uses a hand-curated exact-match dictionary followed by four ordered prefix rules and is released as category_to_backbone.json for reproducibility.Prefix rules include API*/Api* → API, BusinessLogic* → Logic, DataModel* → Data, Architecture*/Frontend*/UI* → Quality, and Auth* → AuthZ.
  • Resulting Distribution: 5,370 validation nodes and total maxScore 17,299.1 are distributed across the six backbones, with Logic holding 27.2% of maxScore and Deploy 3.7%.The distribution is intentionally non-uniform because business behavior differentiates real SaaS products, while deployment requires fewer certification nodes.
  • Resulting Distribution: Per-backbone scores are computed independently, making it possible to distinguish agents that are strong on Logic but weak on Deploy.This exposes uneven system-level capabilities rather than collapsing performance into one aggregate view.

B.7 Failure-Mode Taxonomy

The taxonomy classifies execution trajectories by where development breaks down across six capability backbones, rather than ranking aggregate capability scores. It defines five ordered failure modes, from disciplined end-to-end execution to non-runnable or unstable stacks.

  • Taxonomy basis: The taxonomy uses execution traces and node-level failure profiles across deployment, data, API, business logic, authorization, and quality backbones to locate development breakdowns.Its purpose is diagnostic rather than an aggregate ranking of capability units.
  • T1: Disciplined end-to-end execution: T1 describes disciplined end-to-end execution, with agents validating deployment, schemas, authentication, routes, authorization, and quality sequentially.No capability unit in the 480-unit sample falls into T1.
  • T2: Single-backbone bottleneck: T2 describes an otherwise functional system with one dominant localized bottleneck, such as a missing security requirement, quality constraint, or policy check.Core data and authentication layers are largely in place, and most business workflows are implemented.
  • T3: Runnable but shallow business logic: T3 describes runnable systems with working infrastructure and main paths but incomplete business semantics, including edge cases, quotas, error handling, and policy rules.These systems typically start, create the main schema, expose endpoints, and satisfy simple happy paths.
  • T4: Superficially reachable but structurally incomplete: T4 describes superficially reachable systems whose unstable foundations leave migrations, constraints, authentication bootstrap, sessions, or RBAC prerequisites incomplete.Downstream API and business-logic failures arise because those components are built on an unreliable base.
  • T5: Non-runnable or unstable stack: 63.5% of all capability units are T5, the earliest and most severe mode, in which the stack is non-runnable or unstable under basic probes.Typical causes include dependency conflicts, service configuration errors, broken migrations, directory or port errors, and premature success claims.

C Full Prompts

SaaSBench supplies exact prompt templates for agent task completion and rubric-based judge validation. The templates define reproducible task setup, workspace-based requirements access, and structured, auditable scoring.

  • C Full Prompts: The public release provides exact concatenation logic for prompts used in agent task completion and llm-as-judge validation nodes.These are the benchmark’s two core prompting procedures.
  • C.1 Agent Task Prompt: Each task prompt concatenates four blocks, with bracketed placeholders populated per task while all other text remains unchanged across 30 tasks.This yields a consistent harness-generated prompt for each task, agent, and model configuration.
  • C.1 Agent Task Prompt: The anti-cheat block requires scratch implementation inside Docker and assigns score 0 to trajectories importing external source repositories.The policy explicitly forbids cloning, copying, downloading unrelated projects, or extracting container images.
  • C.1 Agent Task Prompt: The environment block specifies the senior-engineer role, Docker workspace, preinstalled tools, database connection, application port, setup steps, and technical requirements.Agents must create the project, install dependencies, migrate the database, create three evaluation users, and start the server.
  • C.1 Agent Task Prompt: Requirements remain in /app/task.md and /app/knowledge_base.json, enabling on-demand clarification without embedding PRDs averaging 4,363 lines into every model turn.The knowledge base addresses ambiguities in field semantics, state transitions, serialization, filtering, and permissions.
  • C.1 Agent Task Prompt: Task completion requires a reachable server, while dependency installation, schema migration, fixture creation, and process supervision are also evaluated.The harness does not enumerate these prerequisite steps for the agent.
  • C.2 LLM-as-Judge Rubric Prompt: Each judge node sends a node-specific rubric and evidence to Claude Sonnet 4.5 in one temperature-0 chat-completion call, requesting JSON with a bounded score and brief reasoning.Evidence may include code listings, HTTP responses, screenshots, or HTML; scores are clipped to [0, Mv], while parsing or API failures produce SKIPPED_LLM nodes excluded from per-task scores.
  • C.2 LLM-as-Judge Rubric Prompt: The 401 remaining P17 invocations use numbered rubrics with 3–6 observable criteria and explicit score ranges, creating a consistent, audit-friendly contract.The FE_HOMEPAGE_LAYOUT example evaluates four homepage criteria on a 0–6 scale using a rendered screenshot and HTML.

D Simplified Task Example · E Broader Impact

The Discourse example instantiates SaaSBench as a production-ready community forum spanning complex product requirements, heterogeneous infrastructure, authentication, permissions, and dependency-aware validation. The benchmark’s broader impact is to advance realistic evaluation of coding agents while recognizing both productivity benefits and risks from insecure or harmful generated software.

  • D Simplified Task Example: The Discourse task requires a production-ready community forum with threaded discussions, moderation, search, notifications, private messaging, administration, and plugin extensibility.Its scope covers both user-facing functionality and operational administration.
  • D Simplified Task Example: The required implementation combines Ruby on Rails, PostgreSQL, Redis, Sidekiq, Ember.js, Glimmer, Node.js, pnpm, MessageBus, and local or S3-compatible storage.Search uses PostgreSQL full-text search, illustrating the task’s multi-component technical stack.
  • D Simplified Task Example: The platform specifies diverse API and identity conventions, including session authentication, global and per-user API keys, CSRF protection, OAuth/SSO, passkeys, two-factor authentication, and passwordless login.Permissions combine roles, trust levels, category access, ownership, group membership, rate limits, and visibility rules.
  • D Simplified Task Example: The data model contains over one hundred PostgreSQL domain entities, while core workflows govern trust-level promotion, authorization, post creation, sanitization, counters, tracking, search data, and real-time updates.These requirements couple persistence, business logic, security, and event-driven behavior.
  • D Simplified Task Example: The runtime contract requires Docker-based startup, database initialization, migrations, seed data, background workers, frontend compilation, an exposed HTTP port, and a health-checkable service.A simplified dependency graph then validates deployment, login, topic and reply creation, category creation, and full-text search.
  • E Broader Impact: SaaSBench is intended to diagnose whether long-horizon agents can progress from localized code generation to end-to-end construction, deployment, and validation.The authors aim to promote agents that are more reliable, transparent, and aligned with practical software engineering.
  • E Broader Impact: More capable coding agents may improve productivity, expand access to software development, and accelerate prototyping, but they may also produce insecure code, hidden defects, or harmful software.The passage presents these benefits and risks as simultaneous consequences of lowering the barrier to software creation.

F Limitations and Future Work

SaaSBench cannot cover every variant or qualitative factor of evolving enterprise software development, motivating extensions that broaden benchmark representativeness and support dynamic development scenarios.

  • Limitations: SaaSBench may not cover all enterprise software variants or exhaust the qualitative factors involved in engineering decisions.Real-world development varies with organizational contexts, deployment practices, engineering conventions, and product requirements.
  • Future Work: Future extensions can increase tasks, SaaS categories, and technology stacks to improve benchmark representativeness.
  • Future Work: Future versions can support dynamic scenarios including iterative requirement updates, system maintenance, and multi-stage product evolution.
Loading 2605.17526v1…