How to Steer Your Multi-Agent System: Human-LLM Collaborative Planning

• LLM plans may misalign with human intent, miss domain-specific constraints, or hallucinate, motivating systematic user intervention.Source: §1 · p. 1
• Outcome-based supervision obscures intermediate states, making failures harder to diagnose and targeted corrections difficult in complex multi-agent plans.Source: §1 · p. 1
• AMBIPOM represents plans as directed acyclic graphs whose subtask nodes and data-dependency edges can be inspected and modified before and during execution.Source: §1 · p. 1
• The framework formalizes mode, scope, and level, distinguishing semantic from structural, global from targeted, and low-level from high-level interactions.AMBIPOM makes these co-planning strategies explicit, operationalizable, and comparable.Source: §1 · p. 2

2.1 Multi-Agent Systems

• Centralized orchestration assigns subtasks through a controller and represents plans as DAGs, providing explicit data flow and globally coherent planning for human-in-the-loop settings.Source: §2.1 · p. 2

2.2 Human-LLM Co-Planning

• Prior co-planning systems range from high-level chat to direct plan editing, while AIPOM combines chat and graph editing but covers the mode axis only.The paper claims its design space is the first to formalize mode, scope, and level and enable controlled comparison.Source: §2.2 · p. 2

3 Problem Statement

• A plan update is characterized by interaction mode, scope, feedback or graph manipulation, and a selected subgraph when the interaction is targeted.Source: §3 · p. 2

3.1 Research Questions

• The research questions examine how users steer plans and how feedback scope and revision strategy affect LLM-generated revisions.Source: §3.1 · pp. 2–3

3.2 Design Goals

• The design goals call for safe targeted revisions, support for complex structural edits, and transparency through inspectable plan states and interaction history.Source: §3.2 · p. 3

4.1 LLM-Based Planner

• The LLM planner generates the initial plan and supports revisions across semantic and structural modes, global and targeted scopes, and low- and high-level edits.The initial plan is generated from the task query and agent descriptions.Source: §4.1 · p. 3, §4.1.1 · p. 3

4.1.2 Semantic Interaction: Global Feedback

• Global semantic feedback applies free-form text to the entire plan, prompting holistic revisions to reflect changes in overall strategy or task goals.The planner regenerates the plan from the existing plan and the user's text feedback.Source: §4.1.2 · p. 3

4.1.3 Semantic Interaction: Targeted Feedback

• Targeted semantic feedback revises a selected subgraph while preserving compatibility with the rest of the plan.Replanning uses the selected subgraph and its boundary I/O specification rather than the full plan and conversation history.Source: §4.1.3 · p. 3

4.1.4 Structural Interaction: Direct Manipulation

• Low-level direct manipulation lets users add or delete graph elements and edit task descriptions, agent assignments, I/O fields, or agent configurations without involving the LLM.These operations are applied deterministically to the plan.Source: §4.1.4 · p. 3

4.1.5 Structural Interaction: High-Level DM and LLM-Assistance

• High-level structural edits merge interface-closed subgraphs or split a node, manually or with LLM assistance.Merging preserves external I/O interfaces; splitting maintains the original connectivity, and automated edits are applied deterministically and reduce user effort while maintaining structural validity.Source: §4.1.5 · pp. 3–4

4.2 Interface (Supporting DG3)

• A shared plan state connects the dual-panel interface: the chat panel handles semantic interaction, while the plan panel supports visual inspection and structural editing.Updates from either panel propagate immediately to keep the views consistent.Source: §4 · p. 3, §4.2 · p. 4

4.2.1 Chat Panel (Fig. 1A).

• The chat panel supports plan generation and global or targeted replanning, and logs plan changes and direct-manipulation activity for traceable interaction history.Replanning summaries report node and edge changes; logs for LLM-assisted merge and split also summarize resulting node-level changes.Source: §4.2.1 · p. 4

4.2.2 Plan Panel (Fig. 1B).

• The plan panel presents an editable DAG of subtasks and data dependencies, with node cards, execution logs, direct edits, and manual or LLM-assisted merge and split.Users can execute nodes individually or run the full plan and view aggregated results.Source: §4.2.2 · pp. 4–5

5.1 Study Design

• The within-subject study compared AMBIPOM with a similar-interface baseline, focusing on targeted semantic feedback and high-level structural edits among 13 convenience-sampled participants.Participants completed two sessions, one per system, to compare efficiency, effort, and success.Source: §5 · p. 5, §5.1.1 · p. 5

5.1.2 Tasks.

• Tasks covered four multi-agent planning patterns—stepwise math, multi-hop computation, listed retrieval and aggregation, and top-K retrieval and aggregation—with participants revising initial plans to produce correct answers.Each participant completed eight questions across two sessions, with one question from each pattern per session.Source: §5.1.2 · p. 5

5.1.3 Procedure.

• The counterbalanced procedure used demonstrations, tutorials, four tasks per session, and post-session questionnaires, while performance measures covered effectiveness and efficiency.Measures included task success, plan completeness, completion time, interaction frequency, and conversation turns.Source: §5.1.3 · p. 5, §5.1.4 · p. 5

5.2 User Study Results

• Basic interactions produced slightly higher-quality plans, while advanced interactions reduced cognitive load or conversational turns and semantic interactions reduced effort as structural interactions better matched user intentions.DMℎ𝑖𝑔ℎ reduced cognitive load relative to manual DM𝑙𝑜𝑤, and targeted feedback reduced conversational turns required to refine a plan.Source: §5.2.1 · pp. 5–6

5.2.1 RQ 1: Performance with Advanced Interaction Types.

• Within advanced interactions, DMℎ𝑖𝑔ℎ was associated with better plans and higher accuracy than targeted feedback, which required users to select subplan boundaries and formulate textual instructions.The authors also report this pattern as consistent with their benchmark comparison of targeted feedback and more constrained revision strategies.Source: §5.2.1 · p. 6

5.2.2 RQ 2: User Strategy and Collaboration Patterns.

• Users dynamically combined interaction types, often using text feedback for broad changes before applying manual direct manipulation for local repairs.Structural modification favored LLM-assisted edits; semantic preferences were more mixed, with targeted feedback frequently paired with manual edits to finalize changes.Source: §5.2.2 · p. 6
• Users balanced effort, control, and risk across interaction types, and both review-first and execute-first participants converged on broad text edits, local direct-manipulation patches, and re-execution.Ten of 13 participants preferred inspecting and cleaning the plan before execution; three executed early and used failures to guide edits.Source: §5.2.2 · p. 6
• Text feedback was usually prescriptive (169 of 178 messages), while diagnostic feedback appeared mainly during debugging; users’ verification habits also declined later in sessions as trust increased.Participants shifted from checking intermediate results early to accepting plausible-looking outputs without verification later.Source: §5.2.2 · p. 6

6.1.1 Direct Plan Regeneration (GF, TF, TF+P).

• Conditions vary feedback scope and full-plan context for regeneration, while GF-to-DM instead produces structural edit operations from full-plan context.GF replans globally; TF revises a selected subplan without full-plan context and freezes boundaries; TF+P adds full-plan context while retaining frozen boundaries.Source: §6.1.1 · p. 6, §6.1.2 · p. 6, §6 · p. 6

6.2 Benchmark Dataset

• The benchmark contains 200 gold plans and 1,150 broken-plan items, spanning four task subsets and seven plan-operation types.Items pair an initial plan and feedback with target nodes and the gold plan; plans meet a minimum threshold of five nodes and five edges.Source: §6.2 · pp. 6–7

6.3 Experimental Procedure

• The procedure uses ID-anchored feedback for global conditions and deictic feedback for targeted conditions, then evaluates graph distance, semantic similarity, stability, and applicable execution accuracy.Targeted subplans are reintegrated with boundary checks; invalid integrations are excluded from quality-metric calculations.Source: §6.3 · p. 7, §6.4 · p. 7

6.5 Experiment Results

• Global feedback produced valid revisions in all cases, while targeted feedback and edit-sequence revisions had more integration failures.Full-plan context slightly improved targeted integration success; auto-merge/split variants under targeted feedback showed near-perfect integration.Source: §6.5 · p. 7, Table 1 · p. 7
• Targeted feedback improves plan stability but often yields larger graph edit distance than global feedback; GF-to-DM falls between them structurally.GF-to-DM can achieve perfect GED for localized agent-assignment changes, while targeted conditions preserve more non-target nodes.Source: §6.5 · p. 8, Table 2 · p. 8
• Auto-merge tends to preserve structure relative to targeted replanning, whereas auto-split typically increases structural deviation.The authors report the asymmetry as larger topology changes from split operations than from merge operations.Source: §6.5 · p. 8
• 0.840 average execution accuracy for global replanning versus 0.553 for structural operation generation (0.287 difference) on Stepwise Math Reasoning.The authors attribute the gap to incomplete recovery from the initial to gold plan, which can propagate to incorrect final answers.Source: §6.5 · p. 8, Table 3 · p. 8
• Edit-sequence revisions provide an auditable operation trail and can suit localized edits, but direct regeneration is more reliable for complex structural changes.GF-to-DM exposes stepwise operations for inspection, unlike GF's single-pass revised plan.Source: §6.5 · pp. 8–9

7.1 Design Suggestions

• Users combined targeted feedback with direct manipulation, while advanced interactions reduced perceived effort without improving final plan quality.The authors identify a shift in effort from authoring toward verification and integration; global feedback performed strongest in the LLM revision experiments but was less preferred by participants.Source: §7.1 · p. 9
• Systems should preview edits at reintegration points and validate compatibility, with lightweight checks for orphan nodes and input/output mismatches.These supports are proposed to reduce verification burden and help users confirm plans without heavy manual inspection.Source: §7.1 · p. 9
• Interactive planners should guide targeted prompts and surface likely follow-up fixes after replanning, including missing bindings, orphan nodes, and boundary mismatches.The authors also propose one-click repairs and suggestions for next actions to reduce iteration overhead and verification fatigue.Source: §7.1 · p. 9
• Domain-agnostic planning shells should pair with domain-specific output panels for code, images, tables, or long-form text.The current interface assumes node outputs can be summarized as short text, which may be difficult for richer workflows to verify within a node card.Source: §7.1 · p. 9

7.2 Limitations

• The study covers one user working with an orchestrated system of fixed agents; its 13 participants came from one lab and were mostly LLM experts.The authors also flag constrained curated tasks, declining verification effort over time, and synthetic initial errors as limitations; larger validation across models and failure distributions remains future work.Source: §7.2 · p. 9

8 Conclusion

• AMBIPOM operationalizes co-planning across mode, scope, and level, and the evaluations find hybrid user workflows alongside a trade-off between global robustness and targeted stability.The authors frame these contributions as steps toward more transparent, controllable, and trustworthy multi-agent systems.Source: §8 · p. 9

A Implementation Details

• The prototype uses a React frontend, FastAPI communication, and a Python backend, with GPT-4o for planning and GPT-4o-mini for execution.The planner and execution agents are specialized for different tasks.Source: A Implementation Details · p. 11

A.1 Planner Prompts

• Planner prompts specify plan generation, text-feedback replanning, targeted subgraph replanning, and automatic splitting and merging, with plans structured as atomic, self-contained tasks in a valid dependency graph.Nodes use one agent each, and dependencies represent data flow between named outputs and inputs.Source: §A.1 · p. 11
• Subgraph replanning regenerates only selected nodes while preserving the outside nodes and boundary inputs and outputs, allowing internal tasks and edges to be restructured.New nodes use negative IDs, and the replanned subgraph must preserve its boundary interface.Source: §A.1 · p. 12

A.2 Execution Agents

• Four execution agents use task-specific LLM reasoning and structured tools, including Python execution for code, SymPy calculation for math, and Google Custom Search or Brave Search for search; the design is a proof of concept.The authors state that agents producing structured outputs can be integrated with minimal modification.Source: §A.2 · p. 12

A.2.1 Code Agent.

• The Code Agent prompt asks for Python that solves the current plan step rather than directly answering the overall question, and specifies JSON-formatted output.For math questions, it directs the agent to use SymPy representations for pi and fractions and simplify fractions and square roots without decimal conversion.Source: §A.2.1 · p. 13

A.2.2 Math Agent.

• The Math Agent converts a subtask into an arithmetic expression using only numeric inputs and basic operators, returning null when an expression cannot be fully evaluated.Its specified output includes a reasoning field and results keyed to expected output variables.Source: §A.2.2 · p. 13
• The Math Agent prompt says not to include unknown variables in the output expression.The passage gives “a” and “b” as examples.Source: §A.2.2 · p. 13

A.2.3 Search Agent.

• The Search Agent generates concise, informative Google queries and rewrites search results into JSON without outside knowledge, removing irrelevant redundancy and using empty strings for missing information.The response format includes a thought field and output_format key-value entries.Source: §A.2.3 · p. 13

B.1 Task Questions

• The eight questions used in the user-study tasks are provided in Table 4.Source: §B.1 · p. 13

B.2 Metrics

• Effectiveness combines task-specific final-answer accuracy with a manual plan-completeness rubric scoring up to three essential steps; a second author independently verified grades.Task success criteria and completeness rubrics are provided in Tables 5 and 6.Source: §B.2 · p. 13, Table 5 · p. 15, Table 6 · p. 15
• Efficiency is measured by task completion time, individual user operations, and interaction–execution cycles, using interaction logs.Operations include feedback and plan edits; cycles end with a node execution or an “Execute All” run.Source: §B.2 · p. 14

B.3.2 Baseline System Questionnaire (1 = Strongly Disagree, 7 = Strongly Agree).

• The baseline questionnaire asks about ease, intent alignment, and effort for textual feedback and basic direct manipulation, plus satisfaction and perceived control.The listed items cover text feedback and add/edit/remove node or edge operations.Source: §B.3.2 · pp. 14–15

B.3.3 AMBIPOM Questionnaire (1 = Strongly Disagree, 7 = Strongly Agree).

• The AMBIPOM questionnaire covers general and targeted text feedback, manual and LLM-assisted direct manipulation, intent alignment, effort, trust, satisfaction, and control.The appendix also lists post-experiment questions about preferred interaction modes and when participants preferred text versus direct manipulation.Source: §B.3.3 · p. 15

B.3.4 Post-Experiment Survey.

• The post-experiment survey asks participants to choose preferred interaction modes for structural, semantic or ambiguous, and overall plan modifications, and to explain their choices.It also asks about helpful or frustrating aspects and possible use in daily or work planning.Source: §B.3.4 · p. 15

B.4 Detailed Results

• Task success: 0.712 vs. 0.635; plan completeness score: 2.904 vs. 2.808, favoring basic over advanced interaction types.The comparisons do not support the hypothesis that advanced types produce better-quality plans.Source: §B.4 · p. 16
• DM_low was slightly faster (8.163 vs. 8.256 minutes) and better aligned (6.385 vs. 6.231), while DM_high required less effort (3.923 vs. 4.385).These descriptive comparisons partially support H2: DM_high reduced cognitive load, but was slightly slower and marginally less aligned with intent.Source: §B.4 · p. 16
• Across both systems, semantic feedback took less mental effort but felt less aligned with user intent than structural edits; LLM-assisted DM_high was AMBIPOM’s easiest option.In the baseline, general feedback was also rated easier than DM_low.Source: §B.4 · p. 16
• Baseline had more conversation turns (3.615 vs. 3.404), while AMBIPOM required more editing interactions (45.357 vs. 29.750).The authors attribute the extra operations to AMBIPOM’s richer feature set.Source: §B.4 · p. 16

B.4.2 RQ 2: How do users choose among interaction types for different plan modifications, and what recurring patterns emerge in collaborative refinement?

• Participants assembled hybrid workflows: review or execute, make broad changes through global or targeted text feedback, patch local details with direct manipulation, then re-execute.Type selection primarily served as a sequencing strategy rather than a stable personal preference.Source: §B.4.2 · p. 16
• Participants preferred LLM-assisted direct manipulation for structural changes (7), but preferences for semantic or ambiguous changes were evenly distributed across four options.For semantic changes, users often paired targeted feedback with direct manipulation for fine-grained structural adjustments.Source: §B.4.2 · p. 16
• Direct manipulation offered high control but high effort; global feedback required the least authoring effort but the most verification, while targeted feedback and assisted edits bounded scope or structural risk.Participants used direct manipulation for local repairs and avoided global replanning when they feared it could disrupt the plan.Source: §B.4.2 · pp. 16–17
• Ten of 13 participants followed a review-first rhythm and three an execute-first rhythm; both groups commonly made broad edits, patched locally, and re-executed.Decomposition-heavy tasks often elicited text feedback, while multi-hop tasks drew users toward direct manipulation for duplication, edge wiring, and intermediate verification.Source: §B.4.2 · p. 17
• Of 178 feedback messages, 169 were prescriptive; global text tended toward plan-wide revisions, targeted text toward localized changes, while diagnostic feedback appeared chiefly during debugging.In AMBIPOM, participants used targeted feedback slightly more often than global feedback (42 vs. 39).Source: §B.4.2 · p. 17
• Trust in LLM-assisted DM_high rose from 3.620 to 5.538, while verification declined later in sessions as fatigue or time pressure led some participants to accept plausible outputs without checking all dependencies.Participants also used manual recomputation and inspection of intermediate outputs or execution traces to verify and diagnose results.Source: §B.4.2 · p. 17

C.1 Detailed Integration Results

• Table 7 supplements the main results with per-operation integration success rates across conditions, including boundary-flexible variants.Source: §C.1 · p. 17

C.2 Boundary Analysis

• Integration success increased from TF to TF+B, TF+P, and TF+B+P, with TF+B+P highest among targeted replanning conditions.Full-plan context provided a slightly larger benefit than boundary flexibility alone.Source: §C.2 · p. 17, Table 7 · p. 18
• Boundary-flexible parallel split edits had GED values of 17.180 for TF+B and 17.220 for TFsplit+B.The authors note that these revisions can propagate beyond the intended subgraph and diverge sharply from the gold structure.Source: §C.2 · p. 18

C.3 Naturally Occurring Faulty Plans

• Among 200 generated plans per model, GPT-3.5 Turbo produced 51 faulty plans and 16 error plans, versus 28 faulty plans for GPT-4o-mini and 5 for GPT-5-mini.Manual inspection found failures concentrated in edge-linking errors and input/output variable mismatches.Source: §C.3 · p. 18
• Global feedback outperformed targeted feedback on three of four faulty-plan subsets, while both failed on Multi-Hop Computation.Targeted feedback recovered almost no plans elsewhere; faults spanning the plan structure limited even global repair.Source: §C.3 · p. 18