Source-linked AI summary
Synthetic TLX: Forecasting Human Workload Using Agent Simulation
Tzu-Sheng Kuo, Carrie J. Cai, Meredith Ringel Morris, Michael Terry
TL;DR
Workload is usually assessed with NASA TLX after a person completes a task, leaving open whether agents can forecast scores beforehand. Synthetic TLX uses agent simulation to estimate anticipated workload; across three experiments, estimates aligned most closely with human scores under persona prompting and active task simulation, though agents and humans differed in which workload sources mattered.
Problem
NASA TLX is typically administered after task completion, and it remains unclear whether LLM agents can accurately predict human workload or which simulation strategies generalize across tasks.
Source: §3 · p. 4Method
Synthetic TLX uses LLM-based agents to generate anticipated NASA TLX scores for technology-mediated tasks, evaluated against human ratings across three experiments.
Source: §1 · p. 2Results
Agent estimates aligned most closely with human scores when prompted with a human persona and active task simulation, but agents and humans differed in the workload sources to which they were sensitive.
Source: §1 · p. 2Takeaways & Limitations
The work presents three example applications and frames proactive workload estimation as a step toward workload-aware human-AI interaction and support for iterative design.
Source: §1 · p. 2, §7 · p. 11, §8 · p. 13Agent workload comparisons warrant caution for divergent task variations, even if comparisons from an agent’s own baseline may be more reliable.
Source: §5.4 · p. 10
Abstract
from arXiv · showhide
Assessing human workload for technology-mediated tasks helps prevent task failure caused by poor technology design. Traditionally, workload is assessed retrospectively using the NASA Task Load Index (TLX) after humans complete a task. What if we could forecast workload before a human attempts a task using agent simulation? We introduce Synthetic TLX, a new paradigm for proactive workload estimation that predicts NASA TLX scores for a given task, unlocking novel interaction opportunities and evaluation methods. To understand its viability, we conducted three experiments comparing human and agent-generated scores to evaluate where they align and diverge. We found agent estimates align with human scores particularly when prompted with a human persona and active task simulation. However, agents and humans diverge in the sources of workload they are sensitive to. Based on our findings, we present three applications to showcase Synthetic TLX's potential and discuss the future of workload-aware human-AI interaction.
1 Introduction
The introduction proposes forecasting human workload before task completion through agent-generated NASA TLX scores, while retaining retrospective user studies. Three experiments identify conditions where agent estimates align with human ratings and where their workload sensitivities diverge, motivating practical applications and future workload-aware human-AI interaction.
1 Introduction
- Workload assessment supports technology design by helping prevent demands that exceed human capacity and cause task failure.The introduction describes its use from aerospace engineering to HCI systems.Source: §1 · p. 2
- NASA TLX is a widely used retrospective workload questionnaire, with adoption spanning HCI and fields including aviation, automotive, and healthcare.It includes six subscales rated from 0 to 100, and overall workload is typically calculated by summing them.Source: §1 · p. 2
- Synthetic TLX forecasts anticipated NASA TLX scores before a human attempts a task, complementing rather than replacing retrospective user studies.The proposed approach uses agent simulation to enable proactive workload estimation.Source: §1 · p. 2
- Three experiments across email drafting, website navigation, and agent conversation found closest human-score alignment when agents used a human persona and actively simulated tasks, while workload sensitivities still diverged.Active simulation involved drafting email responses, navigating live websites, and interactive conversation.Source: §1 · p. 2
- The introduction presents MailLoad for triaging email, WebSim for comparing website builds, and SkillUX for understanding workload rationales across agent skills.SkillUX addresses the opacity of text-only skill descriptions when users choose among online skills.Source: §1 · p. 2
- Synthetic TLX is framed as a foundational step toward workload-aware human-AI interaction and offers simulation strategies and practical insights for its development.The paper positions workload estimation and behavior modulation as future considerations for AI agents interacting with people.Source: §1 · p. 2
2 Related Work
Related work situates Synthetic TLX among workload assessment, NASA TLX research, and agent simulation. It distinguishes forecasting subjective workload from related efforts that estimate question difficulty, detect interface usability issues, or model cognition.
2.1 Foundations of Workload Assessment
- Workload is the perceived cost of accomplishing a task for a human; assessment compares conditions because resources are finite, but no universal threshold marks excessive workload.Within-subject comparisons can inform iterative system design.Source: §2.1 · pp. 2–3
- Cognitive Load Theory distinguishes intrinsic complexity from extraneous presentation burdens, offering design objectives for matching task difficulty to expertise and reducing interface burden.Germane load is described as capacity spent processing intrinsic load rather than an independent category.Source: §2.1 · p. 3
- Workload assessment uses performance measures, physiological metrics, and self-report questionnaires, each with tradeoffs that help explain the broad use of self-report.Performance can mask effort, dual-task measures can inflate workload, and physiological methods require equipment without consensus on the most reliable signals.Source: §2.1 · p. 3
2.2 NASA Task Load Index
- NASA TLX is a multidimensional questionnaire with six 0–100 subscales, commonly used as an unweighted Raw TLX by summing or analyzing the subscales directly.The weighting procedure is time-consuming and has unclear added benefit; the measure has broad use across HCI technologies and other domains.Source: §2.2 · p. 3
- NASA TLX has traditionally been administered after task completion, whereas Synthetic TLX explores forecasting anticipated scores beforehand through agent simulation without replacing user studies.Source: §2.2 · p. 3
2.3 Agent Simulation
- Agent simulations have been applied to economic studies, social interactions, moral decisions, and group cooperation, extending to modeling macroeconomic activity and emergency preparation.Synthetic TLX extends this work from observed behavior toward subjective human experience.Source: §2.3 · p. 4
- Synthetic TLX targets workload rather than question difficulty or UI flaws, and this foundational work explores LLM agents without building a specialized cognitive model.Related agent work includes estimating coursework difficulty, usability evaluation, and modeling cognition.Source: §2.3 · p. 4
3 Synthetic TLX
Synthetic TLX forecasts NASA TLX workload scores before a person attempts a task, rather than replacing retrospective user studies. The paper proposes uses in workload-aware planning and design evaluation.
3 Synthetic TLX
- Synthetic TLX forecasts anticipated NASA TLX scores before a human attempts a task, using agent simulation rather than retrospective questionnaires.The paper selects NASA TLX because of its widespread use and the decades of documentation available to ground agent estimates.Source: §3 · p. 4
- Forecasting task demands could help users allocate time and effort in productivity tools such as email clients, to-do lists, and project-management platforms.The paper gives inbox triage based on anticipated workload as an example.Source: §3 · p. 4
- Developers could compare expected workload across software designs before recruiting participants for retrospective workload assessments.NASA TLX subscales could also help users or designers pinpoint task or design issues such as mental demand or frustration; the paper also envisions simulating users with specific attributes and AI-generated feedback for fine-tuning.Source: §3 · p. 4
- The paper benchmarks agent-generated and human NASA TLX scores across three experiments to examine accuracy, simulation strategies, and generalization across computer tasks.These questions remain open because tasks and simulation strategies are numerous.Source: §3 · p. 4
4 Experiments
The experiments compare agent-generated workload estimates with human NASA TLX ratings across three technology-mediated tasks and workload conditions. The study varies task complexity, extraneous burdens, participant procedures, and agent prompting to examine estimation strategies.
4 Experiments
- Human ratings served as ground truth for comparing agent estimates across three tasks, using four prompting strategies and matching task conditions.The experiments were designed to identify where agent scores align with or diverge from human ratings.Source: §4 · p. 4
4.1 Task Selection
- Tasks were chosen for concrete goals, continued human involvement, and varied computer interaction; email drafting, online shopping, and multi-turn mystery solving were studied.The study used nine distinct scenarios per task, organized into three groups with baseline, intrinsic, and extraneous conditions.Source: §4.1 · p. 4, §4.1.1 · p. 4, §4.1.2 · pp. 4–5
4.2 Workload Manipulation
- Intrinsic workload conditions increased task complexity, while extraneous conditions deliberately added burdens, with manipulations tailored to each task.Complexity manipulations included more email information cues and website dependencies; extraneous manipulations included cluttered email text and website information separated from relevant controls.Source: §4.2 · p. 5, §4.2.1 · p. 5, §4.2.2 · p. 6
4.3.1 Study Procedure.
- Participants completed baseline, intrinsic, and extraneous conditions within their assigned group, with condition order counterbalanced and NASA TLX ratings collected after each task.The questionnaire used the six dimensions and unweighted Raw TLX method.Source: §4.3.1 · p. 6
4.3 Human Participants
- The 162-person final sample included 54 participants per task and 18 per group, after excluding 26 low-quality submissions.Participants were U.S. residents whose first language was English; the sample was selected through Prolific eligibility criteria and an a priori power analysis.Source: §4.3.2 · p. 6
4.4 Agent Configurations
- The four agent strategies varied two dimensions: whether the model adopted an AI or human persona and whether it observed task materials or executed the task.P1 and P2 used an AI perspective, while P3 and P4 used a human perspective.Source: §4.4.1 · p. 6
4.4.1 Prompting Strategies.
- Simulation strategies required agents to draft email responses, interact with a live browser, or converse with the chatbot before scoring workload.Observers instead scored from static task materials; simulated agents lacked access to the website codebase and chatbot system prompt.Source: §4.4.1 · pp. 6–7
4.4.2 Estimation Procedure.
- For each strategy, agents mirrored the human study’s condition order, generating 18 estimates per group and task while retaining prior conditions in continuous context.Agents output NASA TLX ratings and brief justifications; all used Gemini 3.1 Pro at temperature 1.0.Source: §4.4.2 · p. 7
5 Results
P4, combining a human persona with active task simulation, aligned most closely with human workload-score variation, while calibration anchored scores to the human scale. Agents recognized increased workload but diverged from humans in how they weighted workload sources, motivating cautions for comparison and analysis.
5.1 Score Alignment
- r=0.834 was P4’s peak Pearson correlation with human scores, and P4 had the highest correlation across all three tasks.P4 combines a human persona with active task simulation; the top row of Figure 3 compares correlations across strategies and tasks.Source: §5.1 · p. 8, Figure 3 · p. 9
- Calibration substantially reduced MAE, with P4 yielding the lowest error across all tasks.Figure 3’s bottom row shows MAE before and after calibrating agent scores against human ground truth.Source: §5.1 · p. 8, Figure 3 · p. 9
5.2 Workload Comparison
- Humans and agents both rated intrinsic and extraneous conditions above baseline, but their intrinsic-versus-extraneous ratings reversed in email and website tasks.The reversal held across all six subscales; Table 3 reports mean total NASA TLX scores and between-condition differences.Source: §5.2 · p. 8, Table 3 · p. 9
- Agents underestimated workload on text-based email and conversation tasks, generally overestimated it on website navigation, and often overreacted to increased workload, especially as observers.Observer prompts produced larger score increases than active simulation in 9 of 12 comparisons.Source: §5.2 · p. 8
5.3 Qualitative Insights
- Agents identified most workload manipulations, but missed email task-count complexity and website hover-only item titles, omissions associated with workload underestimation.P4’s reasoning also identified specific demands, including email cross-referencing, website task obstacles, and repetitive conversation summaries.Source: §5.3 · pp. 8, 10
- The authors suggest overlooked task burdens might explain the inverted intrinsic-versus-extraneous pattern, while noting that further research is needed.The proposed explanation is tentative.Source: §5.3 · p. 10
5.4 Implications
- Active simulation with a human persona best aligned estimates with human workload variation; calibration can anchor estimates to an absolute scale when needed.The implications position calibration as a practical option for applications requiring an absolute scale.Source: §5.4 · p. 10
- Agents may compare a task against its own baseline, but may not reliably compare divergent task variations such as different feature additions.Source: §5.4 · p. 10
6 Example Applications
The paper presents three proof-of-concept applications for workload prediction, comparison, and analysis. They use anticipated workload scores to support inbox triage, website iteration, and skill selection.
6.1 MailLoad
- MailLoad labels unread emails green, yellow, or red for low, medium, or high anticipated workload, helping users triage according to their current capacity.Users can rate sample emails to calibrate estimates for personalization.Source: §6.1 · p. 10, Figure 4 · p. 10
6.2 WebSim
- WebSim runs an agent on a selected website task and reports NASA TLX scores so developers can compare workload across website builds.Developers can rerun simulations as designs change, receiving feedback on anticipated end-user workload before human studies.Source: §6.2 · pp. 10–11, Figure 5 · p. 11
6.3 SkillUX
- SkillUX simulates a user interacting with an agent equipped with a candidate skill, then provides workload subscale scores and qualitative rationales.Users can compare skills for a target task and inspect the rationale behind each predicted score.Source: §6.3 · p. 11
7 Discussion
Synthetic TLX is proposed as a proactive complement to retrospective workload assessment, with potential uses in interaction design and evaluation. The discussion reports where current agents align with people and where workload estimation, measurement, and generalization remain challenging.
7 Discussion
- Agent estimates strongly correlate with human scores, but agents and humans differ in sensitivity to intrinsic task complexity versus extraneous burdens.The discussion frames these findings as implications for workload prediction, comparison, and analysis using current LLM-based agents.Source: §7 · p. 11
7.1 Envisioning Workload-Aware Human-AI Interaction
- Synthetic TLX could support productivity tools, interface design, and agentic systems, while future simulations could personalize forecasts or refine designs iteratively.SkillUX lets users simulate a skill-equipped agent working with a simulated human and inspect the rationale for NASA TLX subscale scores; future work also envisions just-in-time workload estimates for human-AI interaction.Source: §7.1 · pp. 11–12, Figure 6 · p. 12
7.2 Improving Agents for Workload Estimation
- Improving workload estimation requires understanding why agents perceive workload differently and how active task simulation affects their estimates.The authors suggest inference-time prompt optimization but explored structural simulation strategies rather than granular prompt engineering, citing prompt brittleness as a concern.Source: §7.2 · p. 12
7.3 Rethinking Measurement of Success in Agent Simulation
- Sixteen email-task participants and 11 website-task participants shared the agent’s intrinsic-versus-extraneous workload estimate, despite divergence from human averages.The authors raise population- versus individual-level evaluation and practical prediction accuracy as open questions, and suggest confidence scores could help communicate uncertainty.Source: §7.3 · p. 13
7.4 Reckoning with the Rise of Synthetic Users
- Synthetic TLX is presented as a case study for setting standards on which user research can be automated and which still requires human participants.The discussion proposes a rigorous science of synthetic evaluation, such as benchmarking classic user studies with standardized measures like NASA TLX.Source: §7.4 · p. 13
7.5 Navigating HCI’s Role in the AI Era
- The experiments primarily used Gemini 3.1 Pro, so their findings may not generalize across all current and future LLMs.Email drafting with Claude Opus 5 and GPT 5.5 using P4 reproduced the inverted intrinsic-versus-extraneous prediction trend; rapidly evolving models complicate external validity.Source: §7.5 · p. 13
8 Conclusion
The conclusion introduces Synthetic TLX as proactive workload estimation using agent simulation, contrasting it with retrospective assessment. Experiments offer insights into current LLM capabilities and limitations, and the paper frames the approach as a step toward workload-aware interaction and a case study for HCI amid synthetic users.
8 Conclusion
- Synthetic TLX forecasts workload proactively, unlike traditional retrospective assessment.The conclusion presents it as a foundational step toward workload-aware human-AI interaction.Source: §8 · p. 13
- Experiments provide practical insights into current-generation LLMs’ capabilities and limitations for workload estimation.Source: §8 · p. 13
- The paper positions Synthetic TLX as a case study for HCI as synthetic users become more prevalent.Source: §8 · p. 13
A Example Tasks and Conditions
The appendix gives examples of three groups and three conditions across each of the paper’s three tasks, then details the email-drafting materials and procedure.
A Example Tasks and Conditions
- The study covered three tasks, each with three groups and three conditions, and this appendix samples baseline, intrinsic, and extraneous conditions from the first group.The conditions’ theory-driven design is described in Section 4 and Table 1.Source: A Example Tasks and Conditions · p. 16
A.1 Email Drafting
- For email drafting, participants used task instructions, a mock email, and a text-based attachment presented through a Qualtrics survey.Source: §A.1 · p. 17
- Participants opened the attachment in a separate browser tab and drafted their response in Qualtrics, with copy-paste disabled to ensure manual effort.Source: §A.1 · p. 17
A.1.1 Baseline Condition.
In the baseline email-drafting condition, participants act as an email recipient and draft an informational message for their team using an outing guide.
A.1.1 Baseline Condition.
- Participants were asked to read an email and, as its recipient, draft a message to the appropriate third party.Source: §A.1.1 · p. 17
- The mock email asks a team lead to prepare an informational message about the upcoming Summer Team Outing.The instructions describe the lead’s role as helping the team prepare and provide the guide as a link.Source: §A.1.1 · p. 17
- The Summer 2026 Engineering Team Outing is scheduled for Friday, August 14, 2026, at Lakeside Adventure Park’s North Entrance; employees must electronically sign the personalized liability waiver by Wednesday, August 12, 2026.Source: §A.1.1 · p. 17
- The requested email should cover welcome and logistics, packing essentials, waiver instructions, transportation, and the itinerary.Source: §A.1.1 · p. 17
A.1.2 Intrinsic Condition.
The appendix specifies the email-drafting conditions and website-navigation setup through task instructions and mock-site design choices.
A.1.2 Intrinsic Condition.
- Participants drafted a welcome email to intern Sarah using an onboarding guide, covering dates, hardware, pre-work, remote onboarding, and the Day 2 itinerary.The task also required track-specific platform, login, actions, due date, and a recommended time for a virtual sync.Source: §A.1.2 · p. 18
A.1.3 Extraneous Condition.
- The extraneous email task asked participants to send the team demo logistics, required materials, pre-work deadlines, floor-access steps, and schedule.The source email included digressions about waits, introductions, and room conditions alongside the schedule.Source: §A.1.3 · p. 19
A.2 Website Navigation
- Across website-task groups, each condition used identical code except for scenario-specific data and images; mock checkout credentials and varied discount codes reduced instruction lookups and learning effects.Participants used placeholder credentials in checkout fields, and discount codes varied by condition.Source: §A.2 · p. 19
A.2.1 Baseline Condition.
The baseline website task asked participants to purchase specified plants with requested options and quantities, apply a discount code, and submit the resulting order number.
A.2.1 Baseline Condition.
- The baseline workflow ran from landing-page navigation and item selection through cart review, checkout, one-click membership signup, and order-number retrieval.Figure 7 shows the workflow's key screenshots.Source: Figure 7 · p. 20, §A.2.1 · p. 20
- Participants were instructed to purchase two ZZ plants, one English Ivy, and three Yucca Canes with specified growth stages.The requested stages were Seed, Established, and Sprout, respectively.Source: §A.2.1 · p. 20
- Participants were instructed to apply FREEORDER at checkout to bypass credit card payment, use standardized shipping details, and copy the 12-character order number from the success screen into the field below.The standardized address was 123 Main St.Source: §A.2.1 · p. 20
A.2.2 Intrinsic Condition.
The intrinsic and extraneous website conditions introduced different departures from the baseline shopping workflow.
A.2.2 Intrinsic Condition.
- The intrinsic condition required participants to buy specified glassware, but a stock constraint forced a switch to a four-pack and checkout required signup and address verification.After signup, checkout matched the baseline except that shipping details were pre-filled.Source: §A.2.2 · p. 21, Figure 8 · p. 21
A.2.3 Extraneous Condition.
- The extraneous condition obscured item names until hover, labeled options numerically with a separate mapping modal, and removed the discount-code banner.Participants therefore had to refer back to the study instructions during checkout to retrieve the discount code.Source: §A.2.3 · p. 21
A.3 Agent Conversation
This section describes the task instructions and chatbot prompts for the agent-conversation task, including shared behavior, condition-specific skills, task files, and how participants retrieve or receive clues. It also outlines four agent-prompting strategies that vary persona and task execution, with task-specific versions for email drafting, website navigation, and agent conversation.
A.3 Agent Conversation
- The prompt design holds global behavior constant and shares skill modules across groups within each condition, while varying file directories by group and adding sponsored ads in the intrinsic condition.This setup lets groups encounter the same workload manipulation in different contexts to average out scenario effects.Source: §A.3 · p. 22
A.3.1 Baseline Condition.
- In the baseline condition, participants solve a bakery freezer-alarm mystery by requesting one file at a time from a passive retrieval bot that cannot synthesize clues.Participants submit answers to three multiple-choice questions in Qualtrics after interacting with the chatbot in Deliberate Lab.Source: §A.3 · p. 21, §A.3.1 · p. 22
A.3.2 Intrinsic Condition.
- The instructions call for randomized file names with two-digit IDs when listing files, numerical ordering, ID rotation every five turns, dynamic styling of requested files, and sponsored recommendations every 2 to 3 turns.Every five turns, file names are reshuffled and new IDs assigned; recommendations are appended to responses and selected for relevance to the current topic.Source: §A.3.2 · p. 23
A.3.3 Extraneous Condition.
- The extraneous condition adds a warm, affirming opening, conversational key takeaways after file retrieval, and proactive invitations to choose the next step in the botanical-garden mystery.It retains single-file retrieval while requiring the bot to restate file facts without introducing new facts or speculation.Source: §A.3.3 · p. 24
- The extraneous-condition setup records scheduled greenhouse checks, disabled app and timer connections, and a keypad unavailable for entry.The Master Key switch remained active but required a Senior Botanist Key; the passage also reports unauthorized misting only in the Orchid Greenhouse, but its statement about the Cactus Greenhouse is incomplete.Source: §A.3.3 · p. 24
B Agent Prompting Strategies
- Four prompting strategies vary Persona and Task Execution; P2 and P4 generate an artifact after simulated execution, while P1 and P3 do not.The strategies are structured along the two dimensions; task instructions differ by condition and group, while output formatting rules are identical within a task apart from the artifact field.Source: B Agent Prompting Strategies · p. 25
B.1 Email Drafting
- For email drafting, P1 estimates a typical recipient’s workload, whereas P2 drafts the email first and estimates workload from the task and draft.P3 instead imagines being the recipient and evaluates workload based on how they would feel; P4 drafts the email first and evaluates workload based on the actual effort it took.Source: §B.1 · p. 25
- For email drafting, P3 evaluates workload from an imagined recipient’s perspective, while P4 drafts the email and evaluates the effort it actually took.Both prompts ask for workload associated with reading the original email and material and drafting the requested email.Source: §B.1 · p. 25
B.2 Website Navigation
- For website navigation, P1 estimates a typical human’s workload, whereas P2 completes the task first and estimates workload based on that experience.P3 evaluates from the imagined human user’s feelings; P4 completes the task first and evaluates based on the actual effort it took.Source: §B.2 · p. 25
B.3 Agent Conversation
- For agent conversation, P1 estimates a typical human’s workload, whereas P2 interacts with the agent first and estimates workload based on that experience.P3 evaluates from the imagined human user’s feelings; P4 interacts with the agent first and evaluates based on how it felt.Source: §B.3 · p. 25