SaaSBench: Exploring the Boundaries of Coding Agents in Long-Horizon Enterprise SaaS Engineering

Benchmark gaps

• Existing benchmarks often lack market grounding, represent simplified systems, and use evaluations that miss dependencies in complex business workflows.Prior benchmarks range from localized code tasks to projects built from scratch, but the latter still face these limitations.Source: §1 · pp. 1–2

Experiments

• Experiments find substantial capability gaps even among state-of-the-art coding agents on SaaSBench.The authors attribute these gaps to limitations in long-horizon task planning and cross-component coordination.Source: Figure 1 · p. 2, §1 · p. 3

SaaSBench

• SaaSBench evaluates agents building enterprise SaaS systems from scratch across 30 tasks in 6 domains.Tasks include a long-context PRD, ambiguity-resolution knowledge base, standardized runtime, and DAG-based test suite; the benchmark has 5,370 executable validation nodes, 8 programming languages, 6 database types, and 13 frameworks, with PRDs averaging approximately 4,363 lines.Source: §1 · p. 2

Evaluation paradigm

• Its dependency-aware hybrid evaluation uses DAG-based validation, prerequisite gating, failure propagation control, and binary, weighted, and llm-as-judge scoring.Validation covers deployment availability, data modeling, API contract consistency, business logic correctness, access control, and engineering quality.Source: §1 · pp. 2–3

Autonomous Coding Agents

• Coding agents include IDE-integrated assistants and autonomous frameworks [18] [45] [48], both using terminal access, execution, dependency installation, and test feedback in longer workflows.IDE-integrated assistants evolved from context-aware completion toward cross-file and repository-level assistance; autonomous frameworks unify terminal, file-system, and runtime access in an agent loop.Source: §2 · p. 3

Code-Centric Agent Benchmarks

• Benchmarks progressed from isolated function generation [5] [6] [17] [25] [47] [50] to repository editing [9] [16] [21] [28] [30] [36], but remain localized and short-horizon.Early benchmarks assess function-level generation, while RepoBench and SWE-Bench require completion, editing, or issue fixing across multiple files.Source: §2 · p. 3
• Project-generation benchmarks include NL2Repo-Bench [10], PRDBench, RepoGenesis [39], and ProjDevBench [31], yet omit enterprise SaaS development and stable evaluation of interactive, multi-dependency systems.These works target complete Python projects, PRD-based projects, web microservices, and projects using Online Judge diagnostics and LLM-based code review, respectively.Source: §2 · p. 4

3.1 Benchmark Construction

• Benchmark construction selects market-grounded, nonredundant SaaS domains and validated repositories, then creates repository-informed long-context PRDs, ambiguity-resolution KBs, and standardized task environments.PRDs capture technical requirements, data models, workflows, API contracts, permissions, boundary rules, and deployment constraints; KBs clarify correctness-sensitive details, while containers preinstall task dependencies and services.Source: §3.1 · pp. 4–5

3.2 DAG Evaluation Protocol

• The hybrid evaluator models independently scored validation nodes and prerequisite edges as a DAG, using binary, weighted, or LLM-judge scoring across six engineering capabilities.Requirements are mapped to nodes, ambiguity is aligned with the KB, and executable checks are chained with business-workflow dependencies; unmet prerequisites are marked “Skipped dependency” rather than repeatedly penalized downstream.Source: §3.2 · pp. 5–6

3.3 Task Quality Validation

• Independent checklist-based PRD review and seed-repository execution against the full test suite gate task inclusion, with revisions until requirements align and tests pass.LLM-as-judge nodes are allowed bounded variance during test-suite validation.Source: §3.3 · p. 6

3.4 Benchmark Statistics

• SaaSBench spans 30 tasks across 6 high-level domains and 30 fine-grained categories, with 4,363-line average PRDs and 5,370 validation nodes linked by 6,167 dependencies.It covers 8 programming languages, 6 database types, 5 frontend frameworks, and 8 backend frameworks.Source: §3.4 · pp. 6–7, Figure 3 · p. 6

4.1 Experimental Settings

• The evaluation covers eight open- and closed-source models across OpenHands and Claude Code, using Claude Sonnet 4.5 as rubric judge at temperature = 0.Reported measures include average pass@1 across tasks and domains, plus node-coverage rate: the proportion of validation nodes that reach the Passed state.Source: §4.1 · p. 7

4.2 Main Results

• 20.68% is the best result, achieved by Claude Opus 4.7 under Claude Code; average performance is 11.64% for Claude Code and 9.26% for OpenHands.The results indicate that end-to-end SaaS construction from natural-language requirements remains challenging; tasks require handling long-context requirements, planning, cross-component implementation, persistent data modeling, permissions, and deployment.Source: §4.2 · p. 8
• Agents perform relatively better on SI and DCI tasks, but weaker on CF and PC tasks involving complex business semantics, long-horizon dependencies, or multi-user collaboration.CF tasks often require billing, order, and financial-state consistency; PC tasks require calendar interactions and shared-state management across frontend, backend, database, and permission logic.Source: §4.2 · p. 8

5.1 Performance by Engineering Capability Dimension

• Table 3 reports per-category scores for Deploy, Data, API, Logic, AuthZ, and Quality.The supplied passage gives the categories but no score values.Source: Table 3 · p. 8

5.4 Error Analysis and Failure Modes

• 95.6% of capability units fail before deep business logic: 63.5% never run stably, 32.1% are structurally incomplete, and only 3.8% reach business-logic bottlenecks.The main limitations are process discipline, deployment stability, dependency management, schema correctness, and reproducible execution—not isolated advanced business logic.Source: Figure 4 · p. 9, §5.4 · pp. 9–10

5.2 Agent Frameworks

• The same model can perform substantially differently across agent frameworks, with commercial IDE-based frameworks often outperforming open-source frameworks.The passage attributes the gap mainly to differences in tool-call, context, and execution-feedback management, which affect coordination costs and long-horizon recovery.Source: §5.2 · p. 9, Figure 4 · p. 9

5.3 Interaction Turns and Performance

• GPT-5.4's 36 steps yield 7.44%, while MiniMax M2.7's 279 steps yield 6.78%, illustrating that more interaction does not ensure higher completion.Successful construction depends on using environmental feedback to diagnose and target fixes; effective reason-act-observe loops matter more than a larger interaction budget.Source: §5.3 · p. 9, Table 4 · p. 9

• SaaSBench is a comprehensive benchmark for evaluating agents developing and deploying enterprise-level SaaS systems from scratch.Source: §6 · p. 10
• Even Claude Opus 4.7, the strongest model in the evaluation, performs poorly at delivering a complete system from scratch.Source: §6 · p. 10
• The authors hope SaaSBench will support future coding-agent development and help move the field toward practical “Vibe Coding”.Source: §6 · p. 10

A.1 Domain Selection Principles

• 421 of 435 category pairs are fully orthogonal; the other 14 have bounded overlap in one dimension, and none fall to 2/4 distinct or below.The audit compares business objects, user roles, data read/write patterns, and architectural challenges; the six families assign each category to exactly one group and collectively cover the benchmark.Source: §A.1 · pp. 14–15

A.2 SaaS Domain Statistics

• 30 tasks span six macro domains: Customer & Growth (7), Productivity & Collaboration (6), Commerce & Finance (4), Data & Content Infrastructure (5), Security, Identity & Infra (4), and Domain & Workflow (4).The category definitions pair each category’s functional scope and engineering challenge with its commercial market, analyst taxonomy, and representative products.Source: §A.2 · p. 15

A.3 Task Statistics

• Each category contributes one selected seed repository, validated by cold-start builds and smoke tests before PRD writing, environment preparation, and evaluation construction.Repositories are selected for representative system form, active maintenance, practical buildability and deployability, and alignment with the category; a candidate pool is also released for possible expansion.Source: §A.3 · p. 16, Table 9 · p. 19, Table 10 · p. 20

B.1 Inference Configuration

• Each configuration receives one rollout per task, with up to 500 reasoning–action steps and a 10,800-second wall-clock budget; failed tasks receive no oracle retries.Individual tool calls can run for up to 900 seconds, while specified network failures are retried up to five times with exponential backoff.Source: §B.1 · pp. 16–17

B.2 Score Aggregation

• Benchmark scores average normalized task scores equally across 30 tasks, while node coverage measures the fraction of validation nodes reaching PASSED.Each task score sums achieved validation-node scores and divides by its task-specific total maximum score before scaling to [0, 100].Source: §B.2 · p. 17

B.3 Evaluated Agents

• Three coding-agent frameworks, each driven by a single LLM, run autonomously from the PRD and knowledge base until completion or budget exhaustion, without human steering.The evaluation scores the final container state to isolate autonomous engineering capability from human guidance.Source: §B.3 · pp. 17, 20

B.4 Metric Detail

• Weighted nodes grant partial credit, rubric-judged nodes assess targets unsuited to deterministic checks, dependency skips score zero, and judge-API skips are excluded from the denominator.The benchmark score is the unweighted mean of normalized task scores; node coverage is also reported.Source: §B.2 · p. 17, §B.4 · pp. 21–22

B.5 Primitive Taxonomy of DAG Validation Nodes

• The DAG validator uses a fixed primitive library spanning 18,196 calls across the 30 tasks, with short identifiers intended to keep DAG files compact and human-auditable.Table 12 groups the primitives by purpose and reports invocation counts; a node may invoke several primitives in sequence.Source: §B.5 · p. 22, Table 12 · p. 22

B.6 Six Evaluation Backbones and Category Mapping

• A deterministic exact-match dictionary and ordered prefix rules map all 289 categories to backbones, covering all 5,370 validation nodes.The released category_to_backbone.json supports re-deriving per-backbone scores; unmatched categories are handled by ordered API, BusinessLogic, DataModel, Architecture/Frontend/UI, and Auth prefixes.Source: §B.6 · pp. 22–23
• Deploy assesses whether the service starts cleanly, including dependency installation, migrations, configuration, fixture seeding, process supervision, and health checks.Its source categories include deployment, setup, build, maintenance, teardown, background jobs, and CLI.Source: §B.6 · p. 23
• Data assesses whether persistent state is correctly modeled and reachable across databases, storage, imports, exports, and event ingestion.Its categories include data models, datasource wiring, files and media, caching, row-level security, and lineage.Source: §B.6 · p. 23
• API assesses correct system protocols, including REST, GraphQL or tRPC, realtime communication, webhooks, search, notifications, and OpenAPI conformance.Its source categories include API variants, CRUD, GraphQL, tRPC, WebSocket, webhooks, search, and validation.Source: §B.6 · p. 23
• Logic assesses domain-specific behavior built on data and API layers, spanning business workflows such as billing, conferencing, gamification, moderation, and collaboration.Its source categories include BusinessLogic, workflow, cron and async tasks, and domain-specific feature families.Source: §B.6 · p. 23
• AuthZ assesses identity, permitted actions, and security auditing across authentication, authorization, access control, identity protocols, and key management.Its source categories include RBAC, audit, password policy, OAuth, OIDC, SAML, and token exchange.Source: §B.6 · p. 23
• Quality assesses non-functional and presentation aspects, including architecture, frontend rendering, edge-case robustness, error handling, localization, and administrative experience.Its source categories include architecture quality, UI, admin panels, internationalization, and configuration and administration.Source: §B.6 · p. 23
• Logic accounts for 27.2% of maximum score, while Deploy accounts for 3.7%; headline Pass@1 does not reweight backbones.Per-backbone scores are computed independently, making differences in an agent’s backbone strengths visible.Source: §B.6 · pp. 23–24, Table 13 · p. 24

T1: Disciplined end-to-end execution

• No capability units in the 480-unit sample are T1, the disciplined trajectory that verifies each layer before building downstream components.T1 proceeds through deployment, data schema, authentication, business routes, authorization policy, and quality checks, yielding a stable, reproducible, well-validated implementation.Source: §B.7 · p. 24

T2: Single-backbone bottleneck

• T2 accounts for only 0.6% of units: the stack is runnable and most workflows are implemented, but one capability backbone has a localized failure.Examples include a missing security requirement, omitted quality constraint, or failed policy-related check.Source: §B.7 · p. 24

T3: Runnable but shallow business logic

• T3 systems run through infrastructure and main execution paths, but leave detailed business semantics such as edge cases, quotas, error handling, and policy rules incomplete.Agents can usually start the application, create the main schema, expose endpoints, and satisfy simple happy paths.Source: §B.7 · p. 24

T4: Superficially reachable but structurally incomplete

• T4 systems appear externally reachable but lack reliable foundations, so downstream API and business-logic failures arise from incomplete setup.Foundations may include migrations, schema constraints, authentication bootstrap, session handling, or RBAC prerequisites.Source: §B.7 · p. 24

T5: Non-runnable or unstable stack

• 63.5% of capability units are T5, where the stack never becomes reliably runnable or fails basic probes because of foundational setup and stability problems.Typical causes include dependency conflicts, service configuration errors, broken migrations, port or volume errors, and premature claims of success.Source: §B.7 · p. 24

C.1 Agent Task Prompt

• The agent prompt concatenates four blocks, with task-specific placeholders and remaining text held constant across all 30 tasks.The blocks cover anti-cheat policy, role and environment, requirements-document and knowledge-base pointers, and the success condition.Source: §C.1 · pp. 25–26
• The prompt forbids importing existing open-source code and assigns a score of 0 to trajectories that fetch external source repositories.Agents must implement the platform from scratch in the provided Docker environment.Source: §C.1 · p. 25
• Agents must build and start the specified application, install dependencies, run migrations, create three evaluation users, and make the server reachable on its designated port.The prompt treats required setup steps, including process supervision, as part of evaluation rather than enumerating them for the harness.Source: §C.1 · pp. 25–26
• The PRD and supplementary knowledge base are stored as workspace files for on-demand reading; PRDs average 4,363 lines, so embedding them each turn would consume the context budget.The knowledge base clarifies ambiguous requirements such as field semantics, state transitions, serialization, filtering, and permissions.Source: §C.1 · pp. 25–26

C.2 LLM-as-Judge Rubric Prompt

• For each llm-as-judge node, P17 sends its rubric and node-specific evidence in one chat-completion call to Claude Sonnet 4.5 at temperature 0.Evidence may include a workspace codebase listing, the last HTTP response body, or a rendered page screenshot or HTML.Source: §C.2 · p. 26
• Judge scores are clipped to [0, Mv]; if JSON parsing fails or the API is unavailable, the node is marked SKIPPED_LLM and excluded from the task-score numerator and denominator.The parser strips Markdown code fences before applying json.loads.Source: §C.2 · p. 26
• The Community Forum homepage-layout rubric scores 0–6 against criteria covering topic-list content, navigation, category navigation, and visual hierarchy and spacing.The example uses a rendered homepage screenshot together with HTML as evidence.Source: §C.2 · p. 26
• The remaining 401 P17 invocations use numbered checklists of 3–6 observable criteria with explicit score ranges, forming a consistent, auditable scoring contract.The judge is instructed to return a JSON object with a score and brief reasoning.Source: §C.2 · pp. 26–27

D Simplified Task Example

• The Discourse task specifies a production-ready forum with extensive community, administrative, and interactive frontend features.Requirements include threaded discussions, moderation, search, notifications, and an administrative dashboard, alongside topic, profile, review, and messaging views and interactive components.Source: D Simplified Task Example · pp. 27–28
• The example fixes a heterogeneous Rails and Ember stack, detailed API conventions, and a PostgreSQL schema with over one hundred domain entities.The stack includes Ruby 3.4, Rails 8.0, PostgreSQL, Redis, Sidekiq, and Node.js 20+; JSON uses shared Rails controllers, while the schema applies ActiveRecord conventions and domain-specific associations.Source: D Simplified Task Example · p. 27
• Forum behavior combines trust-level-based privileges, per-request authorization, validated post and topic lifecycles, and multiple authentication and permission mechanisms.Trust levels TL0–TL3 are automatically promoted and TL4 is manually granted; lifecycle checks include spam, rate limits, and category permissions, while access rules incorporate roles, ownership, groups, and visibility.Source: D Simplified Task Example · pp. 27–28
• Moderation, notifications, and ranked full-text search add review workflows, user alerts, and filtered query capabilities to the forum.The requirements include review scores and threshold actions, warnings and staff logs, notification muting and watched-topic rules, and search filters and advanced query operators.Source: D Simplified Task Example · pp. 27–28
• The runtime contract requires a health-checkable Docker deployment with initialized data, background workers, and compiled frontend assets.The sample validation DAG checks health, then session login, topic creation, and reply creation; later checks depend on prerequisite authentication or created topics and verify returned values and database state.Source: D Simplified Task Example · p. 28
• Separate DAG checks validate category creation through persisted data and full-text search through indexed topic results.The search check creates a topic with a unique keyword, waits for indexing, and queries the JSON endpoint for a result blurb containing that keyword.Source: D Simplified Task Example · pp. 28–29

E Broader Impact

• SaaSBench aims to diagnose whether agents can progress from localized code generation to end-to-end SaaS construction, deployment, and validation.The section presents the benchmark as supporting development of agents that are more reliable, transparent, and aligned with practical software engineering needs.Source: E Broader Impact · p. 29
• More capable coding agents may boost productivity and access to software creation, but can also produce insecure code, hidden defects, or harmful automation.Source: E Broader Impact · p. 29

Limitations

• SaaSBench may not cover every variant of enterprise software construction or qualitative factor in engineering decisions.The diversity and continued evolution of organizational contexts, deployment practices, engineering conventions, and product requirements leave room for refinement.Source: F Limitations and Future Work · p. 29

Future Work

• Future work proposes adding tasks, SaaS categories, and technology stacks, while supporting dynamic scenarios such as requirement updates, maintenance, and product evolution.These are two proposed directions for improving representativeness and covering multi-stage development.Source: F Limitations and Future Work · p. 29