Source-linked AI summary
One Success Isn't Reliability: Thinkingbox, a Sandbox and Benchmark for Agents in Stateful Business Workflows
Zhuochun Li, Youngmin Ko, Ali Keramati, Nicola Ferri, Susana Palmaz Lopez Pelaez, Liang-Chun Tsai, Calvin Wang, Mirco Milletari, Tuhin Kundu, Vadim Smolyakov, Kjartan Olafsson, Tommy Guy
TL;DR
Existing agent evaluations often verify responses or tool calls, but consequential business workflows also require policy-compliant dialogue and correct persistent state changes. Thinkingbox provides an isolated sandbox and builds a 507-task executable benchmark for this setting; the strongest model reaches 65.36% pass@1 but only 25.25% pass^20, exposing a discovery–reliability gap.
Problem
Consequential business work requires more than plausible responses or valid tool calls because agents must follow policies, gather information, and realize correct persistent state transitions without collateral effects.
Method
Thinkingbox provides isolated tool-agent-user sessions, complete trajectories, side-effect extraction, and executable checks over terminal state, side effects, and dialogue.
Results
Across 507 workflows, the strongest model achieves 65.36% pass@1 but only 25.25% pass^20, indicating occasional success is much more common than reliable completion.
Takeaways & Limitations
Thinkingbox-bench separates fluent tool use from dependable work completion by evaluating terminal outcomes and collateral effects rather than surface-level completion alone.
Takeaways & Limitations
Results are bounded by a fixed user simulator and judge: simulated users remain cooperative, do not revise goals, and allow at most ten follow-up turns.
Abstract
from arXiv · showhide
Recent agent benchmarks increasingly ground evaluation in executable environments, from code repair to web navigation, app APIs, and function calling. Yet completing consequential work beyond code requires more than producing a plausible response or valid tool call: agents must gather missing information over multiple turns, follow domain policies, coordinate dependent tools, and realize the correct persistent state transition without collateral effects. In this paper, we introduce Thinkingbox, a sandbox for tool-agent-user interaction that provides isolated MCP-compatible tool sessions, complete execution traces, and outcome evaluation over terminal backend state. Built on this sandbox, Thinkingbox-bench contains 507 policy-conditioned workflows across numerous scenarios, including retail, hospitality, auto insurance, neobank internal IT, and consulting IT/HR support. Each attempt is evaluated by task-specific executable checks that accept valid trajectories while rejecting wrong, missing, or extra effects; designated tasks additionally check required properties of the final response. Across proprietary and open-weight models, the strongest achieves 65.36% pass@1, but only 25.25% pass^20. Moreover, many failed trials show clean termination and valid state-changing actions, showing that response or tool-call-level signals are not clear proxies for end-to-end task completion. Thinkingbox-bench reveals a large gap between occasionally finding a successful trajectory and reliably completing stateful business tasks. We release both Thinkingbox and Thinkingbox-Bench: https://github.com/microsoft/thinkingbox
1 INTRODUCTION
Thinkingbox addresses the difficulty of evaluating consequential, stateful business work that requires coordinated dialogue, tool use, policy adherence, and persistent state changes. Its benchmark makes such work executable and reveals a substantial gap between discovering a successful trajectory and completing one reliably.
- Motivation: Thinkingbox targets multi-turn, stateful business tasks whose correctness depends on user interaction, tool coordination, policy constraints, and backend effects.Examples include changing bookings, processing refunds, updating insurance claims, and routing internal service requests.
- Contributions: Thinkingbox combines isolated tool-agent-user interaction with executable checks over terminal state, side effects, and dialogue.The design accepts different valid trajectories while rejecting wrong, missing, or extra persistent effects.
- Benchmark: 507 tasks span retail, hospitality, auto insurance, neobank internal IT, and consulting IT/HR support.The benchmark evaluates recurring enterprise-assistance patterns across five practical domains.
- Results: 65.36% pass@1 contrasts with 91.12% success at least once and 25.25% success in all 20 attempts for the strongest model.The results show that occasional success is substantially easier than dependable repetition.
- Evaluation: 12 proprietary and open-weight LLMs receive 20 repeated trials per task to expose the discovery–reliability gap.The evaluation reports pass@1 alongside pass@k and pass^k analyses.
2 RELATED WORK
Thinkingbox sits among interactive worlds, tool-use evaluations, and executable professional-work benchmarks. It complements these efforts by focusing on stateful, non-code business workflows with dialogue, policies, backend effects, and outcome-based judging.
- Sandboxes and conversational tool use: Interactive environments such as TextWorld, ALFWorld, WebShop, AgentGym, and Agent World Model support training and evaluating language agents.Related work also studies modular tool use and reasoning–action interfaces.
- Sandboxes and conversational tool use: API-Bank, Gorilla, ToolBench, and BFCL emphasize tool-use and function-calling capabilities.These benchmarks represent the executable tool-use strand of related evaluation.
- Executable and professional-work benchmarks: Executable benchmarks span code repair, interactive settings, web and desktop control, app APIs, enterprise software, CRM, and task utilities.AppWorld notably allows alternative solutions while detecting collateral changes.
- Positioning: Thinkingbox-bench complements prior benchmarks by targeting stateful non-code business workflows with user dialogue, policy-conditioned tool use, backend side effects, and a reusable sandbox verdict.Its scope lies at the intersection of reusable agent environments and executable benchmarks for professional work.
3 THINKINGBOX: A SANDBOX FOR TOOL-AGENT-USER INTERACTION
Thinkingbox is a reusable sandbox that models tool-agent-user interaction as an isolated, finite-horizon environment and judges outcomes through conjunctive executable checks. Its trajectory verdict supports evaluation, failure analysis, and training while preserving credit for different paths to the same correct state.
- Motivation: Function-call correctness alone cannot establish that an agent completed the intended business work or avoided unauthorized state changes.An agent may select the right API but use the wrong entity, skip required confirmation, leave the backend unchanged, or create an extra side effect.
- Task-world model: Each task specifies an initial backend state, user goal, domain tools, simulated-user policy, and hidden executable checks.Tool actions execute against an isolated task-backend session while the orchestrator records the trajectory.
- Task-world model: The induced POMDP contains state, action, observation, transition, observation, reward, initial-state, and horizon components.The latent state includes backend state, user-private state, an evaluator-relevant event log, and episode status.
- Orchestration and isolation: Thinkingbox resets every attempt, creates an isolated tool session, forwards messages, executes calls, and records the trajectory.Isolation prevents attempts from sharing database rows, cached tool state, or side effects.
- Outcome judging: A task passes only when all required conditions hold across final state, side effects, and dialogue.Conjunctive grading detects collateral effects while allowing different valid tool-call paths to receive credit for the same correct world state.
- Evaluation and training: The same verdict supports pass@1 and pass@k evaluation, failure analysis, and outcome-reward training.Individual checks can also provide a reward vector for training.
4 THINKINGBOX-BENCH: VERIFIABLE STATEFUL AGENT TASKS
Thinkingbox-bench instantiates the sandbox as a collection of executable, policy-conditioned business workflows with runnable worlds and validators. Its schema represents both required outcomes and forbidden collateral effects, while construction filters out ill-posed tasks.
- 4.1 SCOPE AND SCALE: 507 executable tasks cover retail, travel and hospitality, auto insurance, neobank support, and consulting IT/HR support.The domains share incomplete information, policy constraints, and the need to update correct records without collateral effects.
- 4.1 SCOPE AND SCALE: Each task is a runnable world with domain tools, an initial state, a simulated user, and executable validators rather than a prompt-only instance.The benchmark uses isolated domain servers with MCP-compatible interfaces.
- Positioning: Thinkingbox-bench focuses on stateful business workflows whose verdict checks backend state, side effects, and dialogue outcome.This scope complements executable benchmarks for code, web, desktop, app, and API settings.
- 4.2 TASK SCHEMA: The task schema specifies policy constraints, expected state changes, forbidden collateral changes, and dialogue requirements.Disclosable facts are released by the simulated user only when the agent asks for them.
- Task construction: Benchmark construction proceeds through workflow design, executable task instantiation, and validation and filtering.Tasks are checked for valid tools, consistent states, clear goals, verifiable outcomes, trajectory-independent checks, and termination.
5 EXPERIMENTS
Thinkingbox-bench evaluates stateful business workflows across domains using repeated trials, executable terminal-state checks, and trace-based failure analysis. Results show substantial domain variation and a gap between fluent tool use and reliable work completion.
- 5 EXPERIMENTS: The evaluation covers 12 proprietary and open-weight models using 20 independent trials per task, with pass@1 based on complete attempts and executable verdicts.The main results use micro-averaged scores over tasks and repeated trials.
- 5.3 DOMAIN-LEVEL ANALYSIS: GPT-5.4 achieves the highest overall pass@1, while performance varies sharply by domain.Retail and booking are substantially easier than auto insurance and neobank support for several models.
- 5.3 DOMAIN-LEVEL ANALYSIS: 52% average retail pass@1 contrasts with about 23% on auto insurance, making retail the easiest and auto insurance the hardest domain.Average pass@1 is about 35% for booking, 30% for neobank support, and 27% for consulting.
- 5.3 DOMAIN-LEVEL ANALYSIS: GPT-5.4 and Sonnet 4.6 are the only models above 50% pass@1 in every domain, but GPT-5.4 drops from 76.33% on retail to 54.60% on consulting.Sonnet 4.6 remains between 51.14% and 68.93% across domains.
- 5.3 DOMAIN-LEVEL ANALYSIS: Open-weight performance is uneven: DeepSeek-V4-Pro reaches 43.26% average pass@1, while Qwen3.6-27B improves from 5.41% to 32.94% over Qwen3.5-9B.The reported rankings suggest nominal parameter count alone does not explain performance.
- 5.4 TRAJECTORY-BASED FAILURE ANALYSIS: Tool Usage dominates failed traces at 77.5% on average, while Wrong State Update, Incomplete User Resolution, and No State-Changing Action average 12.1%, 7.9%, and 2.5%.Tool Usage failures often involve failure to recover from tool feedback; other categories capture omissions, incomplete responses, and incorrect terminal transitions.
- 5.4 TRAJECTORY-BASED FAILURE ANALYSIS: The failure categories distinguish tool-recovery problems, omitted state changes, incomplete resolutions, and incorrect state transitions despite successful tool execution.Complete trajectories expose tool responses, terminal-state differences, and evidence used for deterministic dominant-signature assignment.
6 CONCLUSION
The paper presents Thinkingbox and Thinkingbox-bench as tools for evaluating dependable completion of stateful business workflows. Its results show that strong tool-use performance does not yet ensure reliable work completion, motivating terminal-outcome and collateral-effect checks.
- 6 CONCLUSION: Thinkingbox is a reusable sandbox, and Thinkingbox-bench is an executable benchmark for stateful business workflows.The benchmark evaluates outcomes across domains and repeated attempts.
- 6 CONCLUSION: Strong tool-use performance does not yet translate into dependable work completion because outcomes vary across domains and repeated attempts.Many failures involve poor recovery or incorrect state changes despite plausible responses.
- 6 CONCLUSION: The findings underscore evaluating terminal outcomes and collateral effects rather than surface-level completion alone.The paper positions Thinkingbox as a foundation for developing consistently correct agents in consequential workflows.
A LIMITATIONS
The benchmark’s conclusions are bounded by executable-verdict scope, synthetic task construction, and a constrained simulator. These choices support deterministic evaluation but limit claims about communication, enterprise-work distributions, and real-user interaction.
- Task scope: The tasks are synthetic reconstructions, exclude workflows with multiple defensible resolutions, and depend on fixed harness conventions.The source collection is not claimed to represent the distribution of enterprise work.
- Data provenance: The released benchmark contains no real customer or employee records; identities, references, and database rows are created for execution.The upstream collection protocol for the confidential source collection is outside the released benchmark’s scope.
- Coverage: Synthetic reconstruction preserves linked operational relations, but the benchmark does not claim exhaustive or statistically representative enterprise coverage.Examples include links among orders, customers, fulfillment, payments, and support records.
B.3 COMPOSITION OF THE 507-TASK SET
Thinkingbox-bench consists of individually reviewed executable workflows whose outcomes are checked against task-specific backend states, side effects, and, for selected tasks, final-response requirements. Its evaluation accepts alternative valid trajectories while rejecting wrong, missing, extra, or defective cases.
- Composition: 507 cases include executable backend-state evaluation; 477 are backend-only and 30 additionally impose binary final-response requirements.The response rubrics target communicative conditions such as confidentiality and policy-constrained communication.
- Executable specification: Each case fixes the request, initial records, applicable policy, and expected backend outcome, with the complete specification manually inspected.The benchmark world includes linked records and domain policies that constrain eligible actions and authorization.
- Outcome evaluation: The evaluator compares terminal backend state with a golden state and detects incorrect, missing, duplicate, unrelated, or otherwise extra effects.The principal signal is deterministic and does not require reproducing a prescribed read-write sequence.
- Quality assurance: Cases are individually reviewed for identity, privacy, policy, solvability, execution, reset, golden-state, rubric, and rollout properties.Revision occurs when trace inspection exposes a task or evaluator defect.
- Scope: The retained cases establish synthetic, executable, resettable workflows with checkable outcomes, not exhaustive domain coverage or any company’s request distribution.The benchmark’s conclusions are limited to the workflows represented in the benchmark.
C.2 EVALUATION BENCHMARK
The evaluation benchmark runs agents in isolated MCP sessions with a fixed simulated user and task-specific context, policies, and tool environments. The simulator is cooperative and information-bounded, while the protocol enforces grounding, clarification, and termination constraints.
- Evaluation set: 10,140 independent trials per model result from 507 tasks evaluated over 20 stochastic attempts each.The task set contains 98 retail, 104 travel, 100 auto-insurance, 104 neobank IT, and 101 consulting IT/HR cases.
- Execution: Each case executes in an isolated MCP session where agents inspect or modify scenario-specific environments through structured tools.ThinkingBox records conversations, tool interactions, outcomes, execution time, and token usage after each trial.
- Simulated user: The simulated user supplies follow-ups from task context and conversation history, allows at most 10 follow-up turns, and does not introduce unsupported facts.The simulator is fixed across leaderboard, reliability, ablation, and failure-analysis results.
- Grounding: The user context withholds facts from the opening request, requiring agents to elicit missing information rather than infer it.The simulator must use only permitted context and conversation facts and cannot invent entities or numbers.
- Design scope: The simulator’s constrained cooperative behavior excludes misremembered facts, shifting goals, uncooperative responses, and user actions in the world.Reported pass rates therefore describe performance against an information-bounded, well-behaved interlocutor.
D.1 RELIABILITY AND TASK-DIFFICULTY RESULTS
Repeated-trial evaluation separates discovering a successful trajectory from reliably reproducing one. Thinkingbox-bench reports this distinction through pass@k and passˆk, with a substantial gap for GPT-5.4.
- 91.12% pass@20 versus 25.25% passˆ20 for GPT-5.4 shows that retries can discover success without dependable repeated execution.pass@20 requires at least one successful attempt, whereas passˆ20 requires all 20 attempts to pass.
- pass@k measures whether retries find a successful trajectory, while passˆk measures dependable repeated execution.
- Figures 3 and 4 provide pass@k and passˆk progressions for every evaluated model.
D.2 RETROSPECTIVE EVALUATOR ABLATION
Executable outcome checks reveal failures that response- or action-level completion proxies miss. Many failed trials terminate cleanly and invoke mutating tools, while model–domain interactions remain obscured by aggregate scores.
- 80.88% of failed trials terminate cleanly and invoke a mutating tool, while 67.24% also lack an explicit terminal tool error.These trials nevertheless fail executable outcome checks for wrong, missing, or unintended effects.
- State- and side-effect-based evaluation exposes incorrect executions that response-only or tool-call-only proxies can classify as complete.
- Figure 5 shows model–domain failure interactions that a single overall score obscures.Retail is generally least failure-prone, while auto insurance is particularly difficult for most models.
D.3 FAILURE DIAGNOSTICS
Failure diagnostics classify unsuccessful trajectories by dominant signature and expose domain-specific patterns. Longer traces, more tool calls, or larger contexts do not by themselves imply better completion.
- Failure trajectories receive one dominant signature under a fixed precedence order including Tool Usage, No State-Changing Action, and Incomplete User Resolution.
- Auto insurance has the highest Wrong State Update share, while travel/hospitality and neobank internal IT support are dominated by Tool Usage failures.
- 44.86 messages and 11.99 tool calls for GLM-5.1 do not yield better completion than GPT-5.4’s 29.80 messages and 11.05 tool calls.Low call counts can also reflect premature termination without task resolution.
- Figure 5’s domain heatmap encodes failure rate as 100 −pass@1, with darker red indicating higher failure.
- 12–32K tokens per model turn and accumulated interaction context characterize the benchmark’s average invocation cost.GPT-5.4 processes 14.2K tokens per turn over 8.1 retail turns, approximately 115K tokens per trial.
D.4 REPRESENTATIVE FAILURE CASES
Representative traces show that agents can retrieve relevant information, terminate, or execute tools while still producing wrong, missing, or policy-violating state transitions. The executable checker distinguishes these failure modes from surface-level completion.
- Tool Usage: An unrecovered provisioning error leaves required access and ticket changes missing despite a claimed successful termination.
- No State-Changing Action: A complete lookup sequence still fails when the agent never invokes the required ticket-update action.
- Incomplete User Resolution: An incomplete user-facing resolution determines the dominant signature even when executable checks also find an incorrect ticket status.
- Wrong State Update: A policy-violating extension creates an invalid third payment arrangement even though every mutating tool executes without an API error.
- Prompt Template Example: Thinkingbox assembles episodes from system messages, the task request, MCP tool schemas, domain policy, online tool observations, and simulated-user replies.
E.3 TRAVEL / HOSPITALITY: RAW QUIET-ROOM TRAJECTORY
The quiet-room request trajectory verifies the booking, consults relevant policies, updates the reservation and ticket, and communicates that fulfillment is complimentary, best-effort, and prioritized for VIP customers.
- The customer requested a quiet room for upcoming booking BKG-44935348.
- Policy searches retrieved guidance on pre-stay support and customer segmentation before the booking was modified.
- The verified reservation was confirmed for October 7–9, 2025, with two adults and a standard room including breakfast.
- The agent added “quiet room” to the booking’s special requests and updated the associated open ticket.
- The final response disclosed that the request was complimentary and best-effort rather than guaranteed, while noting VIP priority fulfillment.