Source-linked AI summary
ShowTellArena: Evaluating Business Workflow Understanding from Demonstrations
David Garg, Ritobrata Sarkar, Ehsan Azarnasab, Siddhartha Borah
TL;DR
Agents may reproduce demonstrated clicks without understanding business rules, exceptions, or the correct answer for a particular business. ShowTellArena provides a public workflow dataset and native-interface assessment protocol, then examines selected pilot attempts. The pilot illustrates answer errors and failures to reach the quiz, but its selected sample is not a controlled product ranking.
Problem
Showing an agent a process does not establish that it understood the rule, exception, or business-specific answer, so the paper asks what it can explain and apply after a demonstration.
Source: §1 · p. 1Method
Results
The selected pilot illustrates both answer errors and failures to reach the quiz, while its excluded rows and uneven selection do not support an unbiased product ranking.
Source: §6.2 · p. 4, §7 · p. 5Takeaways & Limitations
The authors invite extensions to the inspectable dataset and assessment workflow, including new workflows with verified evidence and predeclared repetitions.
Source: §7 · p. 5The tasks are authored scenarios with related variants rather than a representative sample of business work, and independent replication is needed before interpreting pilot differences as a product ranking.
Source: §7 · p. 5
Abstract
from arXiv · showhide
We often teach a colleague by showing the work and explaining the decisions as we go. How can we check what an agent understood from the same lesson? We introduce ShowTellArena, a benchmark protocol and public dataset for comprehension after narrated business demonstrations. The v1.0 release contains 50 business workflow tasks, with recordings, screenshots, narration, fixture seeds, and 502 questions. Tasks span finance, hiring, procurement, customer decisions, inventory, and logistics. The protocol holds the business scenario and quiz fixed while allowing each product to capture the lesson through its own teaching interface. Questions test operational rules, boundaries, exceptions, and errors in proposed automations. We analyze 218 selected pilot attempts across 39 workflow cases, including 28 cases attempted by all three evaluated systems. These exploratory results expose both answer errors and failures to complete the teaching experience. We describe the release's verification gaps and the pilot's uneven coverage, exclusions, and grading provenance. The contribution is an inspectable dataset and assessment workflow that others can extend; the selected pilot is not a controlled product ranking.
1 Introduction
ShowTellArena asks whether an agent can explain and apply a narrated business demonstration, rather than merely reproduce its clicks. It contributes a task format, a 50-task public dataset, an evaluation harness, and an exploratory analysis of selected pilot attempts.
1 Introduction
- Showing an agent a process does not establish that it understood the rule, exception, or business-specific answer.The benchmark aims to check understanding before the agent handles a next case.Source: §1 · p. 1
- ShowTellArena combines a repeatable application state, demonstrated actions and narration, and a fixed comprehension quiz.Products encounter the intended lesson through native teaching interfaces, with capture, processing, and permitted learned state included in evaluation.Source: §1 · p. 1
- The paper contributes an evidence-linked task format, a public dataset of 50 workflow tasks and harness, and an exploratory analysis of selected pilot attempts.The code, dataset, and research site are public; mapping case names to released tasks does not establish which recording or quiz revision historical runs used.Source: §1 · p. 1
2 Related Work
ShowTellArena sits alongside work on process documentation, tutorial learning, and agents executing enterprise tasks, but focuses on evidence-linked business rules and exceptions after teaching. Its immediate comprehension measure does not establish execution, learning gain, transfer, or delayed retention.
2 Related Work
- Prior work already studies business-process documentation and learning from demonstrations, while ShowTellArena focuses on evidence-linked business rules and exceptions across native teaching interfaces.WONDERBREAD and VideoWebArena are direct precedents for process documentation and tutorial question answering or retention.Source: §2 · p. 1
- Execution benchmarks complement this immediate comprehension diagnostic: explaining a refund rule does not establish that an agent can carry out the refund safely.The cited work evaluates enterprise-app or environment actions, tool use and policy following, or consequential simulated-company work.Source: §2 · p. 1
- Each product captures and uses its own learned state, so the comparison includes capture and memory rather than isolating foundation-model capability.The protocol diagram does not establish identical historical run conditions.Source: §2 · p. 2, Figure 1 · p. 2
- ShowTellArena stops after teaching and questioning, so it does not establish learning gain, transfer to executed work, or delayed retention.ApprenticeBench studies learning an accounts-payable job across a sequence of tasks.Source: §2 · p. 2
3 Task Format and Protocol
The protocol packages a seeded application state, demonstration, narration, quiz, and grading contract, while distinguishing evaluator materials from what the learner may access. It assesses a fixed intended lesson through native interfaces, with attention to capture, access, grading, and whether answers were supported by what the learner saw or heard.
3 Task Format and Protocol
- Each task has an initial application state, demonstration, narration, quiz, and grading contract, with evaluator materials separated from learner-permitted observations.Publishing an answer key makes task design inspectable but does not authorize learner access to it.Source: §3 · p. 2
3.1 Native Teach–Comprehend
- The controlled input is the intended scenario and scripted lesson, not an identical internal representation across products.Processing delays, capture failures, or omitted audio may change what a product actually receives; repetitions should reset task state and learner context under a declared policy.Source: §3.1 · p. 2
- At quiz time, products may use policy-permitted learned state and notes, but must not recover answers from evaluator questions, demonstration source, or fixture records.The protocol calls for technical access controls and an audit of learner-available tools; a prompt alone is not an access control.Source: §3.1 · p. 2
3.2 Questions and grading
- Questions may be multiple-choice, closed-answer, or rubric-based; closed answers use normalization and possibly semantic-equivalence judging, while rubrics permit bounded partial credit.The current runner averages item grades and records judge configuration and overrides.Source: §3.2 · p. 2
- An unavailable grader is ungraded, distinct from an incorrect answer or an incomplete teaching flow; the pilot's incomplete-attempt convention is separate.The released runner's convention should not be retroactively attributed to the historical pilot.Source: §3.2 · p. 2
3.3 Evidence-linked answers
- Evidence references connect expected claims to recordings, screenshots, or source records, but source records can verify facts without proving they appeared in the lesson.Table 1 describes counts verified against released bundles; screenshots and questions are within-task observations, not independently sampled workflows, and evaluation covers a subset.Source: §3.3 · pp. 2–3, Table 1 · p. 3
4 Public Dataset
The public v1.0 release contains workflow tasks, teaching artifacts, narration, and evidence-linked questions spanning several business areas. Its returns example tests rules and exceptions through questions about new cases and proposed automations; the paper also reports evidence-audit findings and bounds what the dataset and pilot can support.
4 Public Dataset
- The v1.0 release contains 50 workflow tasks spanning finance, hiring, procurement, customer decisions, inventory, and logistics across five named applications.Numbered variants are separate tasks, not necessarily independent workflow families.Source: §4 · p. 3
4.1 One inbox, different decisions
- The returns task tests 30-day eligibility, inclusive £50 direct approval, a damage exception, and diagnosis of incorrect automation rules as question answering, not executed transfers.It asks learners to use delivery rather than email dates and apply the demonstrated rules to new cases.Source: §4.1 · p. 3
4.2 Construction and evidence
- Each bundle includes demonstration or replay artifacts, narration, seed material, and questions with answers or rubrics; the release has 857 question-evidence references.These links support review of whether an answer was taught.Source: §4.2 · p. 3
- All referenced evidence paths and seed-asset references resolve, but 16 of 857 supplied evidence hashes disagree and 68 omit hashes.Hash agreement establishes file identity, not whether a lesson supports an answer.Source: §4.2 · p. 3
- A corrected returns quiz repairs 15 hashes and accepts displayed and replay-fixture address forms, but does not validate the whole upstream dataset or justify reinterpreting historical scores.Its 23 evidence references verify; one mismatch also belongs to Job Requisition Triage.Source: §4.2 · p. 3
4.3 Availability and intended use
- The code is MIT-licensed and authored benchmark content is CC BY 4.0, while application-derived content and third-party software retain upstream terms.Running tasks requires documented fixture applications and product access; the published index lacks a complete fixture-image manifest for bit-for-bit environment reproduction.Source: §4.3 · p. 3
- The release's authored scenarios do not establish general business competence, and public answer keys make it unsuitable as a hidden test against future training contamination.The dataset supports inspecting task construction, exercising adapters, and studying comprehension across workflows.Source: §4.3 · p. 3
5 Writing and Administering the Exam
The exam should test decision-changing details, especially exceptions, and answerability checks should distinguish lesson-based comprehension from prior knowledge or wording cues. Native teaching interfaces introduce different access routes and failure modes, so prospective evaluations should document the full process and retain unsuccessful attempts.
5 Writing and Administering the Exam
- Good questions target details that change a decision; development questions sometimes tested generic knowledge or omitted a consequential exception.This is a development observation, not a quantified annotation study.Source: §5 · p. 4
- An attentive person answering from the lesson and a question-only control can test answerability and prior-knowledge or wording-cue effects, but the revised sample has not completed these controls.Source: §5 · p. 4
- Native teaching can expose different answer routes, including local code or fixture inspection, empty artifacts, refusals, and interrupted processing; the workbook does not establish a complete historical leakage audit.Source: §5 · p. 4
- Prospective runs should freeze task and system settings, repetition and stopping rules, and grading, while reporting failure stages and retaining unsuccessful attempts and human interventions.Source: §5 · p. 4
6 Exploratory Pilot
The exploratory pilot covers selected attempts on released workflow cases, with coverage and grading that vary across systems. Its aggregation and incomplete historical records constrain interpretation: the selected sample is not an unbiased ranking or a basis for causal claims about teaching.
6 Exploratory Pilot
- 218 selected attempts cover 39 tested case entries, with three additional untested entries; the cases span six business domains and map by name to released tasks, not necessarily historical artifact revisions.The workbook snapshot is dated 20 September 2026; the release also contains 11 tasks beyond the evaluated cases.Source: §6 · p. 4
6.1 Coverage and aggregation
- Attempt coverage is uneven: Brackett has three selected attempts on 29 cases and two on 10, while Claude has two on 30 cases.Codex has three attempts on four cases, two on seven, and one on 25.Source: §6.1 · p. 4
- Scores average selected attempts within each case and weight the 28 cases attempted by every system equally; incomplete NA attempts count as zero, while blanks are untested and excluded.Numeric zero is a graded answer, and the authors recompute from attempt cells rather than spreadsheet summary formulas.Source: §6.1 · p. 4, Table 2 · p. 4
6.2 What the records support
- Historical system labels and grading are incomplete: Brackett is reported as fused/multiple-model, Codex’s model is unrecorded, and grades mix manual and model-assisted assessments without common independent calibration.Imported grades are identified for Brackett and Codex, but exact model or adapter versions at each run are not established.Source: §6.2 · p. 4
- Uneven exclusions and grading provenance make Table 2 a description of the selected workbook sample, not an unbiased ranking; no matched cold condition supports a causal teaching-benefit claim.The revised public quiz is not established as the version used in every historical attempt, and missing artifacts prevent complete reruns or independent regrading.Source: §6.2 · pp. 4–5
- Brackett has 107 numeric grades in 107 selected attempts, Codex 51 in 51, and Claude 12 in 60; score differences may also reflect capture, processing, answer production, or grading.Newer attempt notes document demonstration and replay problems.Source: §6.2 · pp. 4–5
7 Limitations and Conclusion
ShowTellArena contributes a format, evidence-linked questions, native adapters, and 50 released workflow tasks; its pilot illustrates the relevance of tracking both answers and failures to reach the quiz. The authors caution that the scenarios and pilot do not support a representative or controlled product ranking, and identify directions for further evaluation.
7 Limitations and Conclusion
- ShowTellArena contributes a task format, evidence-linked questions, native adapters, and 50 released workflow tasks; the pilot illustrates why both answers and failures to reach the quiz matter.Source: §7 · p. 5
- Authored scenarios with related variants are not representative of all business work, and the pilot has selected repetitions, incomplete version and access records, and mixed grading.The authors develop one evaluated system, so they call for independent replication before interpreting differences as a product ranking.Source: §7 · p. 5
- Future studies can add workflows with verified evidence and predeclared repetitions, then examine execution, new cases, and retention separately.Source: §7 · p. 5
A Release and Reproduction
The release and code are pinned separately, with revision details and evidence-file hashes recorded in a manifest. A sanitized export and script support reproducing the aggregation, while new native evaluations require their own recorded configuration and are not reconstructions of the historical pilot.
A Release and Reproduction
- The public dataset v1.0 and inspected code revision are pinned separately, with full identifiers and evidence-file hashes in the release manifest.The dataset resolves to e8dc213 and the inspected code revision is ad59046, both inspected on 20 September 2026.Source: A Release and Reproduction · p. 6
- A sanitized pilot-row export and aggregation script reproduce equal-case weighting without distributing the private workbook, artifact links, or raw evaluator logs.Source: A Release and Reproduction · p. 6
- A new native evaluation uses the pinned instructions, public task, seeded fixtures, configured judge, and product prerequisites, with new versions preserved; it is not a reconstruction of the historical pilot.Source: A Release and Reproduction · p. 6
B Selection and Missing Evidence
The pilot selection record distinguishes scored, untested, and excluded attempts, while a zero-imputation sensitivity check illustrates how one selection decision changes Codex’s mean. Prospective comparisons should freeze repeat selection and report failures by stage.
B Selection and Missing Evidence
- The pilot selection record separates numeric scores, selected NA outcomes, untested cells, earlier excluded Brackett attempts, and explicitly excluded Codex attempts.Excluded incomplete attempts treated as zero are a sensitivity convention, not evidence of zero comprehension.Source: B Selection and Missing Evidence · p. 6
- 55.4% to 54.5%: Codex’s mean on the 28 shared cases after including one all-answers-empty attempt as zero, a sensitivity analysis rather than corrected performance.Assigning zeros to recording failures would conflate missing teaching evidence with measured comprehension; neither convention resolves earlier exclusions or unlisted retries.Source: B Selection and Missing Evidence · p. 6
- Prospective comparisons should freeze which repeats count and report failures by stage, retaining raw answers, item grades, revisions, judge settings, capture evidence, interventions, and stopping rules.Related takes and variants should remain grouped in uncertainty analyses; converting the historical workbook cannot supply missing records.Source: B Selection and Missing Evidence · p. 6