SOP-Bench: Complex Industrial SOPs for Evaluating LLM Agents

1 Introduction

• Existing benchmarks miss the ambiguity, implicit knowledge, branching, and coordinated tool use required by industrial SOPs.These procedures demand contextual interpretation, multi-step synthesis, and judgment under ambiguity, while current evaluations typically isolate tool use, planning, or instruction following.Source: §1 · p. 2
• SOP-Bench covers authentic procedures across twelve domains, including healthcare, hazardous goods, customer service, content moderation, and financial compliance.Industry experts authored procedures reflecting real-world ambiguity, branching logic, and implicit knowledge requirements.Source: §1 · pp. 2–3
• Over 2,000 executable tasks combine structured tool interfaces with grounded outputs that mirror production environments.The benchmark uses AI-generated mock tools, APIs, and datasets that are subsequently validated by human experts.Source: §1 · p. 3
• SOP-Bench enables controlled study of agent architectures, foundation models, and deployment strategies without costly production experimentation.Its contributions include a human–AI construction framework, 2,000+ tasks across 12 domains, and illustrative experiments revealing trade-offs in architecture, reasoning cost, and domain generalization.Source: §1 · p. 3

2 Related Work

• API-centric benchmarks assess isolated tool interactions, whereas SOP execution requires coordinated tools across dependent steps with state tracking and error handling.This distinction separates full workflow execution from single-call tool use.Source: §2 · p. 3
• Existing agent and planning benchmarks use narrow task settings that lack the procedural structure, conditional logic, and ambiguity of industrial SOPs.Embodied environments such as ALFWorld and BabyAI also differ fundamentally from workflow automation tasks.Source: §2 · p. 3
• Earlier SOP automation approaches require manual formalization or target short, narrow descriptions, while some published SOP resources lack tools or ground-truth data.These limitations constrain reproducible evaluation of LLM-native workflow execution.Source: §2 · pp. 3–4
• SOP-Bench differs by combining realistic SOP instructions, diverse domains, structured APIs, and evaluation protocols grounded in real-world complexity.It provides a testbed for robustness in enterprise automation rather than a benchmark of isolated capabilities.Source: §2 · p. 4

3 Generating Industry-grade SOPs

• Human experts define authentic business tasks and procedural context, while AI generates supporting artifacts that experts validate for correctness and realism.The workflow is designed to preserve procedural authenticity while maintaining reproducibility and scalability.Source: §3 · p. 4, §3.1 · p. 4, §3.2 · p. 4

3.1.1 Dataset Schema Generation (LLM-Assisted, Human-Validated).

• Dataset schemas specify inputs, decision points, compliance checks, and outcomes, providing a semantic scaffold that reduces hallucinations and improves consistency.Experts verify that schemas capture implicit knowledge and domain constraints.Source: §3.1.1 · p. 4

3.1.2 SOP Refinement (Human-Authored, AI-Refined).

• AI refines SOPs against the validated schema, then generates contextual datasets containing tool inputs, outputs, decision pathways, edge cases, and failure scenarios.Human experts check domain standards, task logic, data types, and unambiguous procedural behavior.Source: §3.1.2 · p. 4, §3.1.3 · p. 4

3.1.4 API & Tool Specs Generation (AI-Generated, Human- Val-

• The workflow generates APIs, tool specifications, and executable code from datasets, with experts validating interface alignment, relevant data access, and code correctness.These contracts connect procedural logic to reproducible tool execution.Source: §3.1.4 · p. 4, §3.1.5 · p. 4

3.2 Human Authoring and Validation

• Experts validate authoring, schemas, SOP refinements, datasets, APIs, and code for domain accuracy, logical consistency, edge-case coverage, and operational alignment.These five validation steps govern the complete artifact-generation pipeline.Source: §3.2 · p. 4

3.3 SOP Descriptions

• The benchmark contains 2,411 tasks across 12 SOPs, spanning classification, verification, industrial processes, annotation, content flagging, and customer service.Its characterization records task counts, tool counts, token lengths, and human- and model-estimated complexity.Source: §3.3 · p. 5, Table 3 · p. 6

3.3.1 Controlled Complexity Variation.

• Controlled complexity varies branching, redundancy, distractor tools, irrelevant data, ambiguous conditions, and graduated decisions to test filtering, tool selection, and reasoning.For example, one referral-abuse variant expands from 3 tools and 3 outcomes to 6 tools and 7 enforcement actions.Source: §3.3.1 · p. 5, §3.3.2 · pp. 5–6

4 Agents for automating SOPs

• The FC and ReAct agents are complementary baselines, not claimed to be optimal architectures for SOP automation.Both use native LLM tool calling, while ReAct targets branching workflows and multi-step reasoning.Source: §4 · p. 6
• FC iteratively processes native tool calls for up to 10 iterations, whereas ReAct interleaves Thoughts, Actions, and Observations for up to 15 iterations.ReAct also requires at least one tool execution before reaching a conclusion.Source: §4 · p. 6

4.1 Evaluation Methodology

• Standardized scripts provide each agent with a task, SOP, and tool specifications, then log intermediate outputs, final responses, tool traces, and reasoning.Precomputed tool inputs and outputs replace live APIs, enabling stable evaluation without runtime variability.Source: §4 · p. 6, §4.1 · p. 6
• The evaluation reports Execution Completion Rate and Task Success Rate, while future work should add metrics for planning, tool mapping, and tool-calling accuracy.The current metrics distinguish tasks marked complete from tasks completed correctly overall.Source: §4.1 · p. 6

5.1 Results and Insights

• No single agent architecture–model combination consistently dominates across SOP-Bench, so architecture selection must remain task dependent.The results motivate choosing architectures using SOP structure, tool interaction patterns, and latency constraints rather than global averages.Source: §5.1.1 · p. 7

5.1.1 LLM–Agent Architecture Must Be Task Dependent.

• 72% versus 68% TSR favored ReAct over FC with Claude 4 Opus, but ReAct was 33% slower and won on only 8 of 13 SOPs.ReAct took 87.82s ± 10.30s per task versus 66.21s ± 9.57s for FC.Source: §5.1.1 · p. 7, Table 4 · p. 7
• Claude 4.5 Sonnet reached 63.3% TSR with ReAct, below Claude 4 Sonnet at 68.7%, showing that newer models do not guarantee higher performance.FC improved modestly from Claude 3.5 Sonnet v2 to Claude 4.5 Sonnet, whereas ReAct performance declined after Claude 4 Sonnet.Source: §5.1.1 · p. 7
• 65.9% versus 61.1% average TSR favored FC over ReAct in one cross-SOP comparison, while individual SOPs showed reversals, parity, and architecture-specific strengths.Dangerous Goods classification was nearly tied, Customer Service had nearly identical failure rates, and ReAct favored multi-step inspection while FC favored deterministic tool-driven procedures.Source: §5.1.1 · p. 7

5.1.2 Domain Variance Exposes Generalization Limits.

• 100% versus 57% TSR marked the best-case range from Patient Intake to Know Your Business, confirming substantial domain sensitivity and limited cross-domain generalization.Across aggregated results, Claude 4 Opus was strongest for both FC and ReAct, at 68.2% and 73.8% TSR respectively.Source: §5.1.2 · p. 7, Table 5 · p. 7, Table 6 · p. 8
• 94.3% average TSR on Referral Abuse Detection (Easy) versus 27.8% on Video Annotation produced a 3.4× performance range across SOPs.Other difficult SOPs included Aircraft Inspection at 33.4%, Content Flagging at 38.0%, and Warehouse Inspection at 43.5%.Source: §5.1.2 · p. 8

5.2 Ablation studies

• 37% versus 20.8% TSR favored a six-tool minimal registry over a 26-tool bloated registry, indicating that 20 distractor tools can reduce success despite necessary tools remaining available.The difference was 16.2 percentage points on Video Annotation with Claude Sonnet 4.5 ReAct.Source: §5.2.1 · p. 8

5.2.2 Controlled Complexity Variation in Referral Abuse Detection.

• Hard Referral Abuse Detection preserved similarly high TSR but increased execution time by 51.5% for FC and 34.2% for ReAct.FC time rose from 53.99s to 81.79s, while ReAct rose from 66.45s to 89.16s.Source: §5.2.2 · p. 8

6 Conclusions and Next Steps

• SOP-Bench contains 2,000+ executable tasks derived from human expert-authored SOPs across 12 industrial domains.The benchmark is intended to support systematic evaluation under realistic procedural constraints.Source: §6 · p. 8
• Agent performance ranges from 57% to 100% across domains, and Claude 4 averages 73.8% versus Claude 4.5’s 58.8% TSR on ReAct, showing that no model–agent combination consistently dominates.These results motivate evaluating architecture–model interactions before production deployment rather than treating SOP-Bench as a leaderboard.Source: §6 · p. 8
• Future extensions will add controlled SOP-complexity variants, multimodal instructions, hierarchical procedures, and community-contributed domain tasks.The stated aim is to expand SOP-Bench into a shared substrate for studying agent behavior across industries.Source: §6 · p. 8

Appendix Use of Generative AI

• The benchmark’s human–AI workflow uses AI-generated schemas, SOP refinements, datasets, APIs, tools, and executable code, all validated by domain experts.Human experts authored the original procedures and checked the generated artifacts for procedural correctness, consistency, and realism.Source: Appendix Use of Generative AI · p. 9
• The benchmark evaluation used 11 LLMs as agents being evaluated rather than as tools for analysis or writing.The appendix distinguishes evaluation models from models used in benchmark construction and manuscript editing.Source: Appendix Use of Generative AI · pp. 9–10

A.1 FC Agent (Function-Calling Agent)

• The FC agent converts OpenAPI-style specifications into function-calling tools, executes model-triggered calls through a ToolManager, and iterates for up to 10 turns.Final decisions are parsed from XML-style tags into structured outputs, with traceability and minimal supervision as design goals.Source: §A.1 · p. 10

A.2 ReAct Agent

• The ReAct agent alternates Thoughts, Actions, and Observations, enforcing tool use before conclusions and continuing for up to 15 iterations.Its traces record intermediate steps, tool-call status, and iteration counts for workflows involving fallback logic, verification, or multi-phase reasoning.Source: §A.2 · p. 10

B Details of the SOPs in SOP-Bench

• SOP-Bench’s industrial workflows include synthetic data, domain-specific tools, and structured decision procedures designed to mirror operational use cases.The documented examples span diverse business processes and connect benchmark construction to realistic procedural evaluation.Source: B Details of the SOPs in SOP-Bench · p. 10

B.1 Aircraft Inspection

• Aircraft inspection combines mechanical, electrical, and maintenance-record checks through seven API tools, requiring multi-step validation and structured audit outputs.Its conditional branching includes reporting inspection mismatches when they are found.Source: §B.1 · p. 10
• Table 7 reports results for the Aircraft Inspection SOP.The supplied table passage identifies the result scope but does not provide cell values.Source: Table 7 · p. 10

B.2 Content Flagging

• Content flagging combines bot detection, trust scoring, severity assessment, and a rule-based matrix to determine outcomes from content removal to user banning.The workflow incorporates behavioral heuristics, geolocation risk, and multiple scoring dimensions.Source: §B.2 · pp. 10–11
• Four APIs support the workflow, while agents must compute decision outcomes and produce structured audit trails for evaluating planning, execution fidelity, and interpretability.The task reflects conditional logic and real-world signal fusion in trust-and-safety operations.Source: §B.2 · p. 11
• Table 8 reports results for the Content Flagging SOP.The supplied table passage identifies the result scope but does not provide cell values.Source: Table 8 · p. 11

B.3 Customer Service

• Customer Service diagnoses issues using authentication, eligibility, outage, testing, troubleshooting, and escalation steps based on internal system data and predefined thresholds.The workflow produces a JSON Resolution Summary Document containing the complete decision trail for reproducibility and auditability.Source: §B.3 · p. 11
• Tables 9–14 report results for Customer Service, Dangerous Goods, Email Intent, Know Your Business, Patient Intake, and Referral Abuse Detection Easy SOPs.Source: Table 9 · p. 11, Table 10 · p. 12, Table 11 · p. 12

B.4 Dangerous Goods Classification

• Dangerous Goods Classification integrates four document sources and severity scores from 1 to 5 to assign hazard classes A through D.The procedure also requires missing-data handling, validation checks, API integration, document parsing, numerical computation, and structured output.Source: §B.4 · p. 11

B.5 Email intent

• Seller-appeal email intent classification combines message categorization with product, pricing, and inventory checks before generating standardized XML action outputs.The workflow also includes validation, error handling, compliance documentation, and multiple API endpoints.Source: §B.5 · p. 12

B.6 Know Your Business

• Know Your Business verification checks registration, licenses, tax identities, ownership, and sanctions, then either approves the entity or escalates it for human intervention.The agent must assess information completeness and call relevant tools to obtain missing verification data.Source: §B.6 · p. 12

B.7 Patient Intake

• Patient Intake integrates insurance, prescription benefits, clinical risk, and pharmacy checks across six APIs to produce compliant registration outcomes.Evaluation scores completeness and formatting across six output fields plus the final registration decision.Source: §B.7 · pp. 12–13

B.8 Referral Abuse Detection (Version 1)

• Referral Abuse Detection Version 1 counts boolean indicators across four violation categories, applies thresholds and tie priorities, and maps the result to Account Closure, No Action, or Inconclusive.The workflow requires simultaneous scoring, multi-source integration, and structured enforcement decisions under ambiguous evidence.Source: §B.8 · p. 13

B.9 Referral Abuse Detection (Version 2)

• Referral Abuse Detection Version 2 adds temporal patterns, historical violations, weighted indicators, risk severity, and 13 additional input fields to support graduated enforcement.Its two-stage process determines violation type and then risk severity, mapping the combination to seven possible enforcement actions.Source: §B.9 · p. 14

B.10 Traffic Spoofing Detection

• Traffic spoofing detection combines account validation, traffic analysis, source authentication, threshold-based classification, evidence correlation, and graduated enforcement actions.The SOP uses metrics such as user-to-order ratios, engagement scores, conversion rates, and bounce rates, then documents evidence and determines actions from account closure to no action.Source: §B.10 · p. 14

B.11 Video Annotation for self driving

• Video annotation requires checking video specifications, detecting and segmenting objects, enforcing spatial and temporal quality controls, and reporting outputs with human reviewer scores.The required outputs include processing status and paths to object detection and segmentation files.Source: §B.11 · p. 15

B.12 Video Flagging for Content Moderation

• Video content moderation uses fetched review metadata, human escalation, moderator feedback, moderation tags, content warnings, and a final keep-or-takedown decision.The workflow addresses scalable moderation of user-generated video content as platform volume and variety increase.Source: §B.12 · p. 15

B.13 Warehouse Package Inspection

• Warehouse package inspection integrates barcode image comparison, quantity-variance rules, location and damage checks, chargeback calculations, and structured resolution reports.The process immediately returns mismatched items, classifies additional discrepancies, and produces a problem report, chargeback sheet, and resolution status update.Source: §B.13 · p. 15

C.1 SOP Document Generation Prompt

• The SOP-generation prompt converts a task title, context, and sample schema into a detailed, technically structured procedure with explicit steps, substeps, dependencies, edge cases, compliance issues, and deliverables.It requires planning around industry, regulations, terminology, technical details, interdependencies, rare scenarios, and compliance before producing the complete SOP.Source: §C.1 · pp. 15, 17
• The generated SOP follows a formal structure covering purpose, scope, definitions, inputs, detailed main procedures, substeps, and expected results.The prompt also calls for advanced jargon, technical prose, cross-references, and explicit explanations throughout the procedure.Source: §C.1 · p. 17
• The SOP template requires quantitative thresholds and conditions at each stage, including scoring cutoffs, routing rules, and escalation triggers, while remaining self-sustaining.A provided sample schema should be used exhaustively to design thresholds and success or failure logic for decision-making.Source: §C.1 · p. 17

C.2 Data Schema Generation Prompt

• The data-schema prompt derives a synthetic dataset schema from SOP text, task context, and a prescribed output format, defining each field’s type, meaning, and applicable ranges or choices.The example schema includes identifiers, paths, timestamps, anonymized users, languages, regions, metadata, validation flags, and durations.Source: §C.2 · p. 18

C.3 Dataset Generation Prompt

• The dataset-generation prompt extracts decision points, validation steps, escalation conditions, and classifications from the SOP to determine dataset fields and values.It adheres to or infers a complete schema and expands it to reflect the SOP’s operational logic.Source: §C.3 · pp. 18–19
• The synthetic dataset must include valid and invalid examples, edge cases, borderline values, policy violations, and rows representing every step of the SOP’s logic.These requirements are intended to cover the full range of possible cases rather than only typical examples.Source: §C.3 · p. 19
• The dataset output is constrained to a data block containing Python list-of-lists with a header row, complete non-empty rows, valid values, and no malformed strings or embedded newlines.The format is designed for immediate use with CSV or Pandas and excludes commentary outside the data block.Source: §C.3 · p. 19
• Dataset generation requires at least {{n_samples}} valid rows spanning SOP-defined conditions, with consistent field values and uninterrupted output inside the data block.The template also prohibits explanatory interruptions and requires syntactically valid, nonempty rows with parseable booleans and closed brackets.Source: §C.3 · p. 19

C.4 SOP API Generation Prompt

• The SOP API prompt analyzes main-procedure tasks, dependencies, edge cases, parameters, data sources, execution order, errors, and available sample-data columns before generation.This analysis is wrapped in <sop_breakdown> tags and guides comprehensive API documentation.Source: §C.4 · pp. 19–20
• Each distinct SOP operation receives one granular, atomic API that captures external dependencies, derives inputs from SOP data or prior outputs, and is intended to be fully functional.The template also prohibits duplicate APIs for the same task and requires dependencies between API calls to be represented.Source: §C.4 · p. 20
• Generated APIs document names, request bodies, responses, dependencies, and error handling while restricting request and response parameters to fields present in sample data.The complete tool set is enclosed in <TOOLS> tags, with each API enclosed in <API> tags.Source: §C.4 · p. 20
• For each API, the schema prompt produces a toolSpec containing a name, description, and JSON inputSchema aligned with the SOP, restrictions, metadata, and sample data.The schema must include required fields, detailed property descriptions, appropriate data types, and consistent formatting.Source: §C.4 · pp. 20–21
• Schema analysis explicitly quotes relevant SOP text, identifies inputs and constraints, explains workflow alignment, and considers edge cases or tool limitations before emitting the final schema.The final JSON Schema is placed inside <SCHEMA> tags.Source: §C.4 · p. 21

C.5 SOP API Function Code Prompt

• The API function-code prompt supplies API metadata, sample data, and one-shot implementation demonstrations to guide Python functions that simulate the APIs.Its inputs include api, sample_data, api_example, coding_example, and data_example.Source: §C.5 · p. 21
• The SOP API Function Code Prompt supplies API metadata, dataset examples, and code requirements that constrain generated Python implementations to specified interfaces, tests, libraries, datasets, and execution patterns.It requires exact function naming, strict request/response mapping, use of existing dataset columns, CSV loading before field access, demonstrated tool-call handling, and one-pass generation.Source: §C.5 · p. 21

D.1 Function Calling Agent (FC-Agent)

• The FC-Agent template instructs the model to internalize the SOP, analyze ambiguities and edge cases, and execute every step in order without skipping or adding steps.It logs step outcomes, exposes critical-step reasoning, halts on failures or misalignment, inserts checkpoints after every three steps, and returns a final SOP-based summary.Source: §D.1 · p. 22
• The FC-Agent template requires structured output tags for SOP analysis, progress tracking, critical calculations, checkpoint summaries, and the final response.The final response is placed in <final_response> tags after successful completion, while uncertainty or SOP misalignment triggers a clear halt.Source: §D.1 · p. 22

D.2 Prompt Template for React Agent

• The ReAct prompt represents execution as repeated Thought, Action, Action Input, and Observation cycles over an SOP-provided tool set.Tool instructions require careful description review, correctly ordered parameters, valid JSON including required and nested fields, and termination when steps are complete or the answer is unavailable.Source: §D.2 · pp. 22–23
• The ReAct user template embeds the SOP and formatted input, directing the agent to process the input step by step and return a tagged final decision with its rationale.The interaction may repeat tool-use cycles N times before producing the final answer.Source: §D.2 · p. 23

F SOP-Bench: Patient Intake and Registration

• The patient-intake example formalizes registration around demographic intake, insurance and prescription validation, risk stratification, pharmacy-network checks, and HIPAA-compliant handling.Required inputs include insurance and medical-history materials, while system requirements include EHR access, eligibility verification, PBM connectivity, and risk calculation tools.Source: F SOP-Bench: Patient Intake and… · p. 24, §F.1 · p. 24

F.1 SOP: Patient Intake and Registration

• The patient-registration procedure computes lifestyle and clinical risk, confirms eligibility and pharmacy-network status, and produces an archived JSON verification report plus EHR updates.The report includes insurance validation, prescription validation, pharmacy checks, lifestyle and overall risk, and registration status.Source: §F.1 · p. 24

• Table 20 presents the first five records of the synthetically generated patient registration dataset.Source: Table 20 · p. 25

• Tool specifications encode structured inputs and outputs for SOP operations such as lifestyle-risk calculation and pharmacy verification.Source: §F.3 · p. 26
• The schemas define lifestyle inputs including smoking status, alcohol consumption, and exercise frequency for risk assessment.Source: §F.3 · p. 26
• Additional-properties restrictions prevent unspecified fields from being accepted in the tool schemas.Source: §F.3 · p. 26
• Patient identifiers use a specified pattern and example to constrain valid inputs.Source: §F.3 · p. 26
• Pharmacy fields specify names, addresses, and phone numbers with descriptions, examples, and formatting patterns.Source: §F.3 · p. 26