Source-linked AI summary
Plan-Then-Execute: An Empirical Study of User Trust and Team Performance When Using LLM Agents As A Daily Assistant
Gaole He, Gianluca Demartini, Ujwal Gadiraju
TL;DR
Research has limited evidence about how LLM agents can support everyday assistance through planning and sequential decision making, especially across tasks with different risks. This study evaluates plan-then-execute agents with human involvement in planning and execution across six simulated scenarios. User involvement can improve task performance when it repairs plans or execution, but plausible errors can still produce uncalibrated trust.
Problem
Research has limited understanding of how LLM agents support daily assistance through planning and sequential decision making, including how trust and team performance develop across everyday tasks.
Method
The study examines plan-then-execute LLM agents with human involvement in hierarchical planning and step-by-step execution across six simulated everyday scenarios.
Results
User involvement can improve task performance by repairing imperfect plans and execution errors, but plausible wrong plans can mislead users and leave trust uncalibrated.
Takeaways & Limitations
Effective daily assistance requires high-quality plans and appropriate user involvement, while plausible agent outputs require careful attention because they can be convincingly wrong.
Takeaways & Limitations
The study covers a limited set of daily scenarios and uses a simulated execution environment, so broader daily-assistance cases and real-world adoption remain boundary conditions.
Abstract
from arXiv · showhide
Since the explosion in popularity of ChatGPT, large language models (LLMs) have continued to impact our everyday lives. Equipped with external tools that are designed for a specific purpose (e.g., for flight booking or an alarm clock), LLM agents exercise an increasing capability to assist humans in their daily work. Although LLM agents have shown a promising blueprint as daily assistants, there is a limited understanding of how they can provide daily assistance based on planning and sequential decision making capabilities. We draw inspiration from recent work that has highlighted the value of 'LLM-modulo' setups in conjunction with humans-in-the-loop for planning tasks. We conducted an empirical study (N = 248) of LLM agents as daily assistants in six commonly occurring tasks with different levels of risk typically associated with them (e.g., flight ticket booking and credit card payments). To ensure user agency and control over the LLM agent, we adopted LLM agents in a plan-then-execute manner, wherein the agents conducted step-wise planning and step-by-step execution in a simulation environment. We analyzed how user involvement at each stage affects their trust and collaborative team performance. Our findings demonstrate that LLM agents can be a double-edged sword -- (1) they can work well when a high-quality plan and necessary user involvement in execution are available, and (2) users can easily mistrust the LLM agents with plans that seem plausible. We synthesized key insights for using LLM agents as daily assistants to calibrate user trust and achieve better overall task outcomes. Our work has important implications for the future design of daily assistants and human-AI collaboration with LLM agents.
1 Introduction
LLM agents offer flexible, tool-enabled assistance, but their use as daily assistants raises unresolved questions about trust, user control, and task performance. This study examines plan-then-execute collaboration across everyday tasks with varying risks.
- Motivation: High-risk tasks can trigger hesitation because users worry about loss aversion and algorithmic aversion when AI outcomes are imperfect.The introduction contrasts these concerns with lower-risk tasks such as playing music or reporting weather.
- Motivation: LLM agents combine dynamic planning, conversational interaction, and external tools to support flexible daily assistance.Their step-wise plans can be understood by users, while conversation reduces manual control effort over specialized tools.
- Research gap: Existing research leaves unclear how LLM agents function as general daily assistants and how interaction shapes user trust and team performance.Prior work emphasized specific use cases or expert interviews rather than everyday tasks with varying stakes.
- Approach: The study uses a plan-then-execute workflow in which the agent first generates a hierarchical step-wise plan and then converts it into sequential actions.Separating planning from execution is intended to improve task clarity, support user control, and provide a representative abstraction of agent operation.
- Study questions: N = 248 participants completed six everyday scenarios, while the study assessed how involvement in planning and execution affected trust and task performance.Scenarios included credit card payment and itinerary planning.
- Findings: User involvement improved performance by helping fix imperfect plans and execution errors, but it did not reliably calibrate trust and could impose cognitive costs.Plausible but incorrect plans could mislead users, while involvement sometimes reduced confidence or plan quality.
2 Background and Related Work
Prior human-AI collaboration research explains trust, reliance, delegation, and AI-assisted decision making, but LLM agents introduce more flexible planning and execution patterns. The literature therefore leaves their trust and collaboration dynamics under-explored.
- Human-AI collaboration: Human-AI collaboration research studies delegation, trust, reliance, and the effects of user, task, and system factors.Relevant factors include expertise, AI literacy, risk perception, task complexity, uncertainty, feedback, transparency, and advice confidence.
- LLM-agent collaboration: LLM-agent collaboration remains under-explored because prior studies mainly examined fixed advice systems, specific use cases, or structured interviews.LLM agents can instead provide logical step-wise plans and step-by-step execution.
- Trust and reliance: Trust concerns arise because AI systems may appear competent through persuasive outputs even when their actual trustworthiness differs from user perception.Researchers have proposed confidence scores, explanations, trustworthiness cues, and uncertainty communication to support calibration.
- Trust and reliance: Trust calibration alone may not ensure complementary team performance; users must appropriately rely on AI when it is capable and override it when it is not.Under-reliance means disusing useful AI assistance, whereas over-reliance means misusing incorrect or inferior assistance.
- LLM challenges: LLMs can reduce human cognitive effort through coherent generation, but hallucinations may produce plausible yet incorrect outputs with serious consequences.The risk of failure depends on the application context, including high-stakes domains such as medical diagnosis.
3 Method
The study uses a plan-then-execute LLM agent that generates hierarchical step-wise plans and executes them one action at a time through backend APIs. User involvement enables plan editing and step-level checking or correction during execution.
- Plan-Then-Execute: The agent first generates a hierarchical step-wise plan, then translates each step into one backend API action during execution.The plan contains up to three levels of sub-steps, while execution proceeds primary step by primary step.
- User-Involved Planning: User-involved planning lets participants add, delete, edit, or split plan steps before execution.These controls support correcting grammar errors, missing information, or steps containing multiple actions.
- User-Involved Execution: During user-involved execution, the agent predicts an action for each primary step and presents it for user review before execution.The interface separates action prediction from action execution and provides a conversational presentation.
- Experimental Comparison: Automatic planning and execution provide the comparison baseline in which the agent directly generates task outcomes without user involvement.The experiment varies user involvement independently across planning and execution stages.
- User-Involved Execution: Users can discover and correct execution mistakes by checking predicted actions and execution results at each primary step.The proposed hypotheses were that user involvement would improve calibrated trust and overall task performance in both planning and execution.
4 Study Design
The study evaluates plan-then-execute LLM assistants in a factorial experiment spanning everyday tasks, trust measures, performance measures, and participant controls. The procedure uses simulated API-based tasks and compares different levels of user involvement.
- Experimental Conditions: The experiment uses a 2 × 2 factorial design crossing automatic versus user-involved planning and execution.The four conditions are AP-AE, AP-UE, UP-AE, and UP-UE.
- Tasks: Six simulated everyday scenarios cover currency transactions, credit card payments, repair appointments, alarm setting, flight booking, and itinerary planning.Tasks come from the UltraTool planning dataset and use backend APIs in a simulation environment.
- Measures: Calibrated trust measures whether users trust high-quality plans and correct execution results while distrusting lower-quality plans and wrong execution results.Subjective trust is measured with four trust-in-automation subscales, alongside familiarity, propensity to trust, and expertise covariates.
- Measures: Plan quality is rated on a five-point scale, while execution performance compares generated action sequences with ground-truth sequences and correctness criteria.Task performance is assessed from planning and execution outcomes.
- Participants and Procedure: The study recruited 347 participants to obtain the required sample of 244 after accounting for exclusions and multiple-hypothesis testing.Participants were screened for English proficiency, age, prior platform experience, and attention-check performance.
5 Results
This section presents the study’s main experimental results and exploratory analyses.
- The results section presents the main experimental results from the study.
- The section also reports exploratory analyses alongside the primary findings.
- The reported analyses concern the empirical study described in the preceding sections.
5.1 Descriptive Statistics
Across 248 balanced participants, planning and execution showed moderate calibrated trust and differing levels of action-sequence performance. User involvement did not change subjective trust, but involvement at both stages increased cognitive load.
- Participants: 248 participants were balanced across AP-AE (63), AP-UE (64), UP-AE (61), and UP-UE (60) conditions.Edited plans in user-involved planning conditions were evaluated using the study’s plan-quality criteria.
- Trust and Performance: Calibrated trust averaged M= 0.50, SD= 0.13 for planning and M= 0.64, SD= 0.19 for execution.
- Trust and Performance: Action-sequence accuracy averaged M= 0.48, SD= 0.17, while recall of ground-truth actions averaged M= 0.77, SD= 0.11.The successful rate for correct execution was M= 0.52, SD= 0.18.
- Subjective Trust: Two-way ANOVA found no significant impact of planning or execution involvement on subjective user trust across conditions.
- Cognitive Load: User involvement in planning affected Mental Demand, Temporal Demand, and Frustration, while execution involvement affected Performance and Effort.Post-hoc Tukey HSD confirmed higher cognitive load when both planning and execution involved users.
5.2 Hypothesis Verification
Across task-specific tests, user involvement did not consistently improve calibrated trust or task performance, although execution involvement often improved execution accuracy after controlling for plan quality.
- Planning and calibrated trust: Only task-4 showed a significant planning-trust difference, with automatic planning producing higher calibrated trust than user-involved planning; H1 was not supported.The result held after a two-way ANOVA controlling for execution-stage involvement.
- Planning and calibrated trust: Calibrated trust in planning was low for high-risk tasks with imperfect plans, whereas low-risk tasks with high-quality plans produced much higher calibrated trust.User-involved execution yielded slightly higher planning calibration in task-1 and task-4.
- Planning and calibrated trust: User involvement in planning decreased plan quality overall, especially for tasks with perfect initial plans, but improved task-1’s grammar-error plan.The study analyzed plan quality, action-sequence accuracy, and execution accuracy separately.
- Task performance: UP-UE achieved the best execution accuracy overall, but task-4 was an exception where planning involvement significantly worsened both action-sequence and execution accuracy.AP-AE produced relatively higher action-sequence accuracy, while UP-UE performed better or comparably in most tasks.
- Task performance: Automatic planning and execution produced very low success in tasks 1 and 6 because imperfect plans or execution choices caused missed transactions or incorrect itinerary selection.In task-1, an uneditable step bundled two actions; in task-6, automatic execution often selected the wrong itinerary.
- Execution involvement: User involvement in execution produced no significant difference in calibrated execution trust, so H3 was not supported.The analysis excluded tasks where user-involved planning had reduced plan quality.
5.3 Exploratory Analysis
Exploratory analyses linked plan quality and user factors to trust and performance, while risk perception showed broadly negative correlations; execution accuracy rose sharply with plan quality but remained vulnerable to action-prediction errors.
- Plan quality and risk perception: Plan quality positively correlated with most trust, calibration, and performance measures, whereas risk perception negatively correlated with most measures and with plan quality.These relationships appeared in task-level and broader covariate analyses.
- Plan quality and risk perception: 1.8% execution accuracy occurred for low-quality plans, compared with 59% for potentially misleading plans and 66.7% for high-quality plans.The categories were plan quality = 1–2, 3–4, and 5, respectively.
- Plan quality and risk perception: Among 717 tasks with high-quality plans, 235 still produced wrong execution results, mainly because action parameters were predicted incorrectly.Wrong action parameter prediction accounted for 48.9% of these failures; wrong action names accounted for 2.6%.
- Confidence dynamics: AP-AE produced the highest confidence in both planning and execution, while planning involvement significantly lowered planning confidence.The comparison included AP-AE, UP-AE, and UP-UE conditions.
5.4 Analysis of Open Feedback
Open feedback was predominantly positive about LLM agents as daily assistants but also identified trust, usability, cognitive-load, and reliability concerns requiring design attention.
- Participant opinions: More than 80% of analyzed comments expressed positive opinions toward LLM agents as daily assistants, alongside negative comments about difficulty, expertise, and trust.Feedback was categorized as positive, negative, mixed, or neutral after excluding non-informative responses.
- Design suggestions: Participants requested drag-and-drop plan reordering and undo operations to make plan editing more convenient and tolerant of mistakes.These suggestions specifically addressed plan-editing interactions.
- Design suggestions: Some participants found the detailed plans cognitively burdensome, while many described execution as smooth.The feedback distinguished plan complexity from execution-stage usability.
- Design suggestions: Participants suggested verifying each execution step to further improve daily-assistant reliability.The proposed verification was tied to validating each requirement during execution.
6 Discussion
User involvement can improve task performance by correcting plans and actions, but it does not consistently calibrate trust and can sometimes harm outcomes. The discussion therefore favors selective oversight and more flexible collaboration workflows, while noting limits on transferability and user control.
- Trust calibration: User involvement does not consistently calibrate trust, especially when plausible plans are low quality.Plan quality positively correlates with calibrated trust, whereas plausible but wrong plans can mislead users.
- Task performance: User involvement can correct imperfect plans and wrong action predictions, improving task performance and execution accuracy.Planning involvement can fix errors, while execution involvement can control risky or invalid actions.
- Trust calibration: Convincingly wrong outcomes may push users toward over-reliance before sharply reducing trust once errors become noticeable.The discussion connects these opposing reactions to algorithm appreciation and algorithm aversion.
- Implications: Effective collaboration requires balancing oversight against cognitive load and the risk that users damage already high-quality plans.The authors recommend minimizing involvement while preserving control over risky or uncertain actions.
- Implications: Flexible workflows should let humans adjust planning and execution together because plan steps may not map cleanly onto actions.The proposed design would expose action predictions earlier and support more direct correction.
- Caveats and limitations: The findings may not transfer broadly because the study used selected daily tasks, a simulation environment, and relatively fixed collaboration modes.Real-world tasks may involve more dependencies, complexity, and multiple actions per goal.
7 Conclusion
This work empirically examines how user involvement with plan-then-execute LLM agents affects trust and task performance in everyday scenarios. It finds that plausible but wrong outputs can undermine trust calibration, while execution involvement more consistently improves performance than planning involvement.
- Conclusion: The study examines human-AI collaboration with plan-then-execute LLM agents across everyday scenarios, focusing on trust and task performance.Users could interact with agents during planning and execution to correct plans or outcomes.
- Conclusion: User involvement fails to calibrate trust, while plausible but wrong plans can produce uncalibrated trust in planning and execution outcomes.The conclusion identifies plausible wrong outputs as detrimental to trust calibration and task performance.
- Conclusion: Execution involvement has a more stable positive impact on task performance than planning involvement.Planning involvement can improve outcomes when plans are imperfect but harm quality when the original plan is already good.
- Conclusion: The paper proposes flexible collaborative workflows and future interventions for detecting and correcting plausible but imperfect LLM outcomes.These directions aim to improve control and collaboration with plan-then-execute agents.
A Details about Tasks
The appendix details six simulated daily-assistance tasks, pairing hierarchical text plans with tool-based action sequences. Examples show correct automation alongside plans requiring user correction because of missing steps, unnecessary actions, or wrong parameters.
- Task setup: The study represents planning as step-wise text plans and execution as generated action sequences for each selected task.The appendix provides both artifacts for the simulated daily-assistance scenarios.
- Task 1: Task 1 requires two foreign-exchange transactions, but its imperfect plan must be split so the agent can perform both transactions.The reported correction adds a primary step before execution.
- Task 2: Task 2 combines debt inquiry, repayment, and future-bill checking, but missing the correct credit-card search step prevents complete execution.The correct card ID is absent from the flawed plan and action sequence.
- Task 3: Task 3 plans a Sony TV repair appointment, but its final primary step is misaligned with the available actions and should be removed or overridden.The first three primary steps are described as clear, while the last step creates unnecessary actions.
- Tasks 4–5: Tasks 4 and 5 are reported as having correct automatic planning and execution, covering a recurring alarm modification and a round-trip flight booking.The task descriptions specify the alarm schedule and London-Amsterdam travel requirements.
- Task details: Another task has a correct plan but an incorrect route selection, causing wrong parameters that affect a later hotel-booking step.The expected route is route-2 rather than route-1.