PaperlayerView exact revision on arXiv

Source-linked AI summary

ARGUS: An Agenda-Driven Framework for Reliable User Simulation

Hang Zheng, Nan Jiang, Zichen Zhu, Zixu Yang, Zhiyang Tang, Shuai Fan, Lu Chen, Xiaobao Wu, Kai Yu

arXiv:2610.21346v1cs.AI

TL;DR

LLM-based user simulators can omit goals or violate policies, limiting behavioral fidelity in interactive environments. ARGUS uses a symbolic agenda stack to separate state tracking from response generation and distills the approach with evaluator-guided SFT and online RL. The paper reports superior behavioral fidelity on τ 3-Bench and MultiWOZ and transfer of the framework to compact models.

  • Problem

    LLM-based simulators can struggle to maintain multi-turn goal progression and policy adherence, even though reliable simulation is needed for evaluating and training agents.

    Source: §1 · p. 1

  • Method

    ARGUS separates symbolic updates to a dynamic LIFO agenda stack from language generation, then distills the framework through evaluator-guided SFT and online RL.

    Source: §1 · p. 2

  • Results

    ARGUS demonstrates superior behavioral fidelity on τ 3-Bench and MultiWOZ, and its training pipeline transfers neuro-symbolic tracking to compact models while advancing the cost-quality Pareto frontier.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The paper presents ARGUS as a scalable solution for synthetic data generation and rigorous evaluation of autonomous agents.

    Source: §8 · p. 11

    Errors in goal parsing or agenda updates may propagate, and applicability to open-ended conversations without clear target states remains unstudied.

    Source: §8 · p. 11

Abstract

from arXiv · show

High-fidelity user simulators are essential for evaluating and training agents in interactive environments. However, current LLM-based simulators struggle with reliability issues, such as goal omission and policy non-compliance. We propose ARGUS (Agenda-dRiven Goal-oriented User Simulator), a neuro-symbolic framework that maintains the user's goal state using a dynamic agenda stack manipulated via symbolic operations. By rigorously decoupling state tracking from natural language generation, ARGUS facilitates strict goal progression and constraint-aligned interactions. For cost-efficient deployment, we distill ARGUS into compact models using a unified pipeline: evaluator-guided SFT followed by online RL with dense step-level rewards. To rigorously evaluate behavioral fidelity, we introduce a rubric-based metric Composite Behavioral Score (CBS). Evaluations on $\tau^3$-Bench and MultiWOZ demonstrate the significant superiority of the ARGUS architecture and the efficacy of our training pipeline. Code and data will be released soon.

1 Introduction

ARGUS addresses goal omission and policy non-compliance by separating agenda-based state tracking from language generation. The introduction also presents evaluator-guided training for compact models and CBS as contributions, and reports superior behavioral fidelity on two benchmarks.

1 Introduction

  • Reliable simulators must follow complex, multi-step goals and interaction protocols without hallucination or arbitrary deviation.The introduction motivates this need for both agent evaluation and synthetic data generation.Source: §1 · p. 1
  • ARGUS uses a dynamic LIFO agenda stack and symbolic operations to separate goal-state tracking from natural-language response generation.This design is intended to mitigate goal omission and structural failures in complex interactive tasks.Source: §1 · p. 2
  • Evaluator-guided SFT followed by online RL distills ARGUS into compact 4B/8B models and advances the cost-quality Pareto frontier.The pipeline uses a shared step-level evaluator for SFT filtering and RL reward assignment.Source: §1 · p. 2
  • Composite Behavioral Score (CBS) uses rubric-based behavioral subcriteria to expose procedural violations that outcome-centric metrics miss.Source: §1 · p. 2
  • ARGUS demonstrates superior behavioral fidelity on τ 3-Bench and MultiWOZ.Source: §1 · p. 2

2 Problem Formulation

The section frames user simulation as a sequential task in which an agent message and dialogue history inform the next user response. It contrasts stateless generation with explicit state tracking, while noting the overhead and reliability issues of text-based state rewriting.

2 Problem Formulation

  • The simulator receives a user goal, dialogue history, and the latest agent message, then generates the next user utterance.Source: §2 · pp. 2–3
  • Stateless LLM simulators combine goal understanding, intent reasoning, condition verification, and language generation in one pass, burdening the model with implicit progress tracking.This can produce state confusion and premature termination as task complexity grows.Source: §2 · p. 3
  • Stateful simulators separate state transition from response generation, but rewriting flat textual states for nested conditions adds inference overhead and risks parsing errors, semantic drift, and temporal confusion.Source: §2 · p. 3

3 ARGUS

ARGUS maintains explicit agenda state and separates state updates from response generation. Its training pipeline uses evaluator-guided SFT and online RL to transfer this approach to compact models.

3 ARGUS

  • ARGUS addresses stateful simulation's need for precise, reliable state transitions with a neuro-symbolic agenda stack and deterministic tool calls, then distills the design for lower-latency models.Source: §3 · p. 4

3.1 Neuro-Symbolic User Simulation

  • The runtime state combines a static user profile with a dynamic LIFO agenda of main goals and dialogue-introduced subtasks.A parser converts the textual goal into the profile and task set; stack frames pair a task with its type.Source: §3.1 · p. 4
  • PUSH, POP, MODIFY, and WAIT respectively add a needed interaction, remove a satisfied need, update a pending frame, or leave the agenda unchanged.The controller can execute consecutive operations based on dialogue history and the previous state; symbolic updates reduce semantic drift and parsing errors.Source: §3.1 · p. 4
  • The response generator conditions on the updated agenda, keeping lower-priority goals visible while the top frame sets the default immediate focus.The two phases assign state-transition decisions to the controller and linguistic realization to the generator.Source: §3.1 · p. 4

3.2 Training Pipeline

  • A shared Step-Level Evaluator provides behavior-aware supervision for filtering SFT data and assigning online RL rewards.It scores each user step across Soundness, Faithfulness, and Adherence using binary atomic checks.Source: §3.2 · p. 5
  • Evaluator-guided SFT retains only teacher-generated user steps scoring perfectly on all three dimensions; adapted online DAPO RL then explores broader interaction distributions.RL discards rollout groups with identical rewards, balances Stack and NLG groups and domains, and uses harmonic-mean step rewards while randomly sampling a rollout to advance the environment.Source: §3.2 · p. 5

4 Experimental Setup

The experiments evaluate simulators on Retail, Airline, and MultiWOZ, with Airline held out for out-of-distribution evaluation. CBS is the primary behavioral-fidelity metric, while TSR is treated as an auxiliary measure of agent task completion.

4 Experimental Setup

  • The evaluation covers τ^3-Bench Retail and Airline plus an adapted MultiWOZ environment, with Airline held out for out-of-distribution evaluation.MultiWOZ is normalized into writable domain states with transactional APIs and final-state checks; its test set is manually verified.Source: §4 · pp. 5–6
  • Simulators are compared against four stateless and one stateful baseline across frontier LLMs and trained Qwen3 4B/8B variants.The interacting agent and CBS judge are fixed as Qwen3-MoE-235B, and each task is run five times.Source: §4 · p. 6

4.1 Evaluation Metrics

  • TSR measures whether the agent completes its goals but can reward simulator shortcuts that violate realistic user protocols, so it is only an auxiliary reference.Shortcuts include prematurely revealing constraints or hallucinating details.Source: §4.1 · p. 6
  • CBS is the primary fidelity metric, evaluating faithfulness, policy adherence, goal completion, and format/protocol compliance.Each dimension passes only when all its binary subcriteria pass; CBS combines the four pass rates with a harmonic mean.Source: §4.1 · pp. 6–7
  • Expert validation reports that CBS aligns strongly with human judgment and GPT-5.4, while its harmonic mean penalizes critical weakness in any dimension.Source: §4.1 · p. 7

5 Empirical Results

Across prompt-based and trained evaluations, ARGUS leads on behavioral fidelity, including as task complexity rises. Its trained variants approach much larger simulators at lower inference cost, while CBS reveals distinctions that TSR can miss.

5.1 Prompt-Based Simulators

  • Prompt-based ARGUS achieves the highest CBS in controlled comparisons and outperforms baselines across most backbones and domains, with a wider gap in Retail.The margin follows an inverse U-shape across model capability: it is narrow at 8B, largest at intermediate capability, and remains clear at 235B.Source: §5.1 · p. 7

5.2 Trained Simulators

  • Retail CBS rises from 83.3% with ARGUS-SFT to 85.8% with ARGUS-RL; trained ARGUS variants outperform their baseline counterparts, and ARGUS-RL (8B) approaches much larger simulators.Table 2 reports trained-simulator CBS and TSR, including the held-out Airline environment; ARGUS-RL approaches 235B Vanilla and 32B prompt-based ARGUS.Source: §5.2 · p. 7, Table 2 · p. 7
  • UGST-SFT has competitive MultiWOZ TSR but significantly lower CBS; case studies identify unrealistic shortcuts, including hallucinated task conditions.The reported behavioral gap between UGST-SFT and ARGUS-SFT highlights explicit agenda-based state management and evaluator-guided filtering as complementary supervision.Source: §5.2 · p. 8
  • Trained ARGUS shows robust zero-shot generalization on held-out Airline, which the paper interprets as domain-agnostic goal-compliance capability.Source: §5.2 · p. 8

5.3 Robustness Across Task Complexity

  • +10.3% CBS over Vanilla on Expert tasks yields only +2.8% TSR; ARGUS leads CBS across all four complexity tiers, while TSR declines as tasks grow harder.The 114 Retail tasks are partitioned into Easy, Medium, Hard, and Expert tiers by database-modifying actions and conditional behaviors; TSR is proposed as secondary to CBS.Source: §5.3 · p. 8

5.4 Inference Cost Analysis

  • Trained ARGUS advances the low-cost, high-quality Pareto frontier, matching massive 235B configurations at a fraction of inference cost.Figure 4 plots inference cost per dialogue against CBS; prompt-based baselines cluster in low-quality/cheap or high-quality/expensive regions.Source: §5.4 · p. 8

6 Analytical Insights

ARGUS’s behavioral gains are concentrated in structural control, while TSR alone can mask simulator violations. The RL stage outperforms SFT even as additional SFT data plateaus.

6.1 Fine-Grained Error Pattern Analysis

  • ARGUS reduces Goal Misunderstanding, Protocol Failure, and Mixed Generation errors to 1.1%, 1.8%, and 2.3%, respectively, and Constraint Violation to 21.7%.The corresponding baseline errors are 3.9–14.4% for the first three criteria and 39.3–46.6% for Constraint Violation.Source: §6.1 · p. 8
  • ARGUS’s smaller gains on State Confusion (3.2% vs. 3.3–4.7%) and Conditional Logic Failure (11.7% vs. 12.6–14.7%) indicate gains concentrated in structural control.The authors distinguish format compliance and state tracking from a universal improvement in reasoning.Source: §6.1 · p. 9

6.2 The Illusion of Success

  • Baselines have False Positive Rates of 48.1%–60.6%, compared with ARGUS’s 33.6%, for agent successes despite simulator violations.The paper defines Illusory Success as TSR = 1 despite CBS < 1; shortcuts include premature disclosure and unconditional compromise.Source: §6.2 · p. 9, Figure 6 · p. 10

6.3 Advantage of RL over SFT

  • ARGUS-SFT CBS improves through 20k training samples, then plateaus and slightly regresses at 30k, never surpassing ARGUS-RL.The comparison averages scores across Retail and MultiWOZ.Source: §6.3 · p. 10, Figure 7 · p. 10

7 Related Work

Related work spans agenda-based, LLM-based, and stateful user simulators. ARGUS distinguishes its stateful approach through explicit stack operations rather than descriptive state summaries or traces.

Agenda-Based User Simulators

  • Classical agenda-based simulators use explicit goals and agenda-based dialogue acts for controllable state updates and policy learning.Source: §7 · p. 10

LLM-Based User Simulators

Stateful User Simulators

  • UGST tracks goal progress to guide responses (Mehri et al., 2025), whereas ARGUS operationalizes state through a LIFO stack updated by constrained symbolic operations.Other stateful work adds persona control or structured reasoning over dynamic goals and trajectories (Karthikeyan, 2025; Zhang et al., 2026).Source: §7 · p. 10

8 Conclusion

ARGUS separates agenda-based state tracking from language generation and distills the framework into compact models. Its limitations include unverified intermediate state and an assumption that goals can be represented as structured tasks.

8 Conclusion

  • ARGUS combines a dynamic LIFO agenda stack with evaluator-guided SFT and behavior-aware online RL to transfer simulation capabilities to compact 4B and 8B models.The conclusion reports that evaluations using CBS show ARGUS advances the cost-quality Pareto frontier.Source: §8 · p. 11
  • Errors in goal parsing or agenda updates may propagate because intermediate state is not explicitly verified or revised; extension to open-ended conversations without clear target states remains unstudied.Source: §8 · p. 11

AI use statement; Ethics statement

  • The authors used generative AI for manuscript editing and synthetic task and dialogue data, and state that they reviewed the writing and manually checked generated data.The evaluated materials contain no personally identifiable information, and no external participants or crowdworkers were recruited.Source: AI use statement · p. 12, Ethics statement · p. 12

Reproducibility statement

  • The paper says its main-paper setup, results, and appendices provide information needed to reproduce the reported evaluations.The appendices cover implementation, data construction, benchmarks, CBS evaluation, evaluator reliability, and prompts.Source: Reproducibility statement · p. 12

A Initial Goal Extraction and Agenda Initialization; A.1 Extraction Pipeline

  • Goal extraction converts each natural-language scenario into a user profile and task list; a verifier checks JSON faithfulness, and failed outputs are revised for up to five attempts.The input includes background, private constraints, task instructions, and the reason for contacting the agent.Source: A Initial Goal Extraction and Agenda… · p. 15, §A.1 · p. 16

A.2 Structured Goal State

  • The global profile stores persona, communication policies, and static facts, while tasks initialize the agenda in scenario order with requirements and conditional behaviors.Conditional behavior activates only when its specified agent-message condition is met, allowing the task to be updated while preserving other requests.Source: §A.2 · pp. 16–17

B Step-Level Evaluator

  • The step-level evaluator judges stack operations and generated responses separately; SFT retains turns only when both steps pass all three evaluation dimensions.For RL, dimension-level scores from the two stages provide dense rewards.Source: B Step-Level Evaluator · p. 17

B.1 Stage 1: Stack Logic Evaluation

  • Stack-operation evaluation checks lifecycle correctness, grounding in goals or dialogue, and adherence to conditional behaviors and persona constraints.Its operations include push_task, pop_task, modify_top_task, and wait.Source: §B.1 · p. 17

B.2 Stage 2: NLG Consistency Evaluation

The NLG evaluator checks responses against the updated agenda and user profile. Its step scores closely align with trajectory-level CBS, and the section also describes an executable MultiWOZ environment.

B.2 Stage 2: NLG Consistency Evaluation

  • The NLG consistency evaluator checks response soundness, faithfulness, and adherence against the updated agenda, profile, dialogue, and latest agent message.It checks complete fulfillment without scope creep, grounding of factual content, and consistency with persona, policy, privacy, and realism.Source: §B.2 · p. 17
  • Mean step score is strongly associated with mean CBS across eight trajectory buckets (Spearman’s ρ = 0.9762, p < 0.001; Pearson’s r = 0.9463, p < 0.001).CBS broadly increases with step score, with a small decline in the highest-score bucket; the analysis covers 300 Retail trajectories.Source: §B.2 · p. 17, Table 3 · p. 18

C Interactive MultiWOZ Environment

  • The adapted MultiWOZ test set contains 155 tasks: 125 single-domain and 30 multi-domain.The environment adds writable domain state, transactional records, and executable APIs across hotel, restaurant, train, taxi, attraction, and public-service domains.Source: C Interactive MultiWOZ Environment · p. 18

C.1 Data Cleaning and State Construction

The converted environment supports grounded, state-changing tasks across domains, with cases constructed against its actual state and checked for executable completion. CBS evaluates simulator behavior across four dimensions using strict subcriteria and weakest-dimension-sensitive aggregation.

C.1 Data Cleaning and State Construction

  • The environment adds a seven-day transactional state window with domain-specific inventory and booking state, plus shared contact and payment services.This supports cross-domain dependencies while limiting unintended interference between single-domain tasks.Source: §C.1 · pp. 18–19

C.2 Tool Design

  • Tools enforce closed-loop search, feasibility checking, booking, modification, cancellation, and identity and payment constraints.State-changing operations update domain state, inventory, calendars, contacts, and payment information when applicable.Source: §C.2 · p. 19, §C.3 · pp. 19–20

C.3 Closed-loop Behavior and Verification

  • State updates let tasks include realistic fallback paths, with success judged from final environment state rather than surface-form matching.Examples include choosing another room type, reservation time, train seat, or taxi fare when the preferred option is unavailable.Source: §C.3 · p. 20

C.4 Cross-domain Support

  • Geographic, temporal, contact, and payment state support cross-domain tasks with distance, scheduling, balance, and refund constraints.The converted environment is intended for interactive user simulation and agent evaluation involving both information retrieval and valid state changes.Source: §C.4 · p. 20

D.1 Case Construction

  • Case construction grounds high-level intents through controlled database resolvers, then uses the resulting trajectory to generate the task, initial state, solution, and expected final-state check.Precomputed common-case and corner-case pools support feasible ordinary and difficult scenarios.Source: §D.1 · pp. 20–21

D.2 Case Validation

  • Generated cases must pass a one-attempt solo execution check and an LLM audit of grounding, constraints, completion checks, and ambiguity; MultiWOZ test cases also receive manual review.Cases failing either validation stage are not retained.Source: §D.2 · p. 21

D.3 Human Evaluation Setup

  • Human evaluations were conducted by the authors as expert annotators, without external participants or crowdworkers.The evaluated materials were synthetic or benchmark-derived task descriptions and dialogue transcripts without personally identifiable information.Source: §D.3 · p. 21

E.1 Evaluation Protocol

  • CBS scores binary behavioral subcriteria across four dimensions, uses strict AND aggregation within each dimension, and combines dimension-level pass rates with a harmonic mean.The harmonic mean emphasizes the weakest dimension rather than allowing strong dimensions to compensate for a severe weakness.Source: §E.1 · pp. 21, 23, Table 5 · p. 22

E.2 Human Alignment Validation

CBS shows strong human agreement and close alignment between its primary LLM judge and human labels, with a second judge supporting robustness of method rankings. The appendix also details ARGUS implementations, baselines, training settings, and Telecom results.

E.2 Human Alignment Validation

  • ARGUS ranks first under the alternative DeepSeek judge with 95.49%–99.92% probability across four backbones, and judges agree on 91.67% of within-backbone method orderings.The cell-level scores correlate at Pearson’s r = 0.873, Spearman’s ρ = 0.886, and Kendall’s τb = 0.746.Source: §E.2 · pp. 23–24, Table 6 · p. 24

F.1 ARGUS Implementations

  • Prompt-based ARGUS uses separate state-transition and response-generation phases with stack tools, while trained variants internalize the operations and use shorter prompts.Malformed tool calls are ignored; the prompt-based implementation does not use an exception-and-retry mechanism.Source: §F.1 · p. 24

F.2 Baseline Implementations

  • ReAct, Verify, and Reflection are prompt-only baselines without explicit external state, whereas UGST tracks structured goal state before response generation.Verify retries rejected responses up to three times, and Reflection adds self-correction before re-verification.Source: §F.2 · pp. 24–25

E.2 Human Alignment Validation

  • Human annotators reach AC2 = 0.861, and Qwen3-MoE-235B aligns with human labels at AC1 = 0.812; GPT-5.4 provides an independent check.Agreement is strongest on objective structural criteria and weakest on Conditional Logic Failure.Source: §E.2 · p. 23, Table 7 · p. 25

F.3 Training Details

  • ARGUS-SFT filters teacher trajectories step by step, while ARGUS-RL collects on-policy rollouts and rewards from the live τ 3-Bench environment.SFT uses 10,000 samples and one epoch; ARGUS-RL uses four rollouts per sample and 25 training steps.Source: §F.3 · p. 26

G.1 Additional Prompt-Based Results

  • Prompt-based ARGUS results on Telecom provide additional evidence of effectiveness in a user-simulation setting that requires tool-use interaction.Source: §G.1 · p. 26, Table 8 · p. 26

G.2 Ablation Study

The ablations find higher CBS for ARGUS architecture variants than for matched alternatives, while sample-filtering choices trade off CBS and TSR. ARGUS also retains the highest CBS across input controls and extractor comparisons, with strong results reported for 4B models.

G.2 Ablation Study

  • ARGUS-SFT variants achieve substantially higher CBS than Naive-SFT and UGST-SFT counterparts, indicating that architecture is the main advantage rather than sample filtering.Step-level filtering generally yields higher CBS, while trajectory-based filtering tends to yield higher TSR; rejected step-level samples can include shortcut behavior.Source: §G.2 · p. 27, Table 9 · p. 27
  • ARGUS achieves the highest CBS in every tested setting, while structured-goal and prompt controls improve stateless baselines variably and are not consistently additive.TSR follows a less consistent pattern than CBS.Source: §G.2 · p. 27, Table 10 · p. 28
  • Replacing UGST’s GPT-4o goal extractor with GPT-5.1 does not produce consistent gains, while ARGUS retains the highest CBS in every comparison.The extractor-model difference therefore does not account for the observed behavioral advantage.Source: §G.2 · p. 27, Table 11 · p. 28

G.3 Additional Trained Model Results

  • Trained ARGUS variants also achieve strong performance with Qwen3-4B, indicating that the training pipeline is not specific to the 8B setting.Source: §G.3 · p. 27

H Task Complexity Stratification

Across Retail and MultiWOZ, weaker user-model backbones increase TSR false positives, while ARGUS has the lowest reported FPR. The appendix defines task-complexity tiers, illustrates shortcut behavior, and compares inference-cost patterns for prompt-based and trained simulators.

H Task Complexity Stratification

I.1 FPR Across User LLM Backbones

  • 20.6% is ARGUS’s lowest FPR across the tested user LLM backbones, while baselines exceed 40% even with Qwen3-MoE-235B and reach 64.7% for UGST + Qwen3-32B.The comparison aggregates Retail and MultiWOZ; weaker backbones produce more simulator errors that can still yield task success.Source: §I.1 · p. 29, Figure 8 · p. 30

I.2 Examples of Shortcut Behavior

  • A correct final environment state can result from a shortcut that violates the intended interaction process, so TSR alone does not establish simulator quality.Source: §I.2 · p. 30, Table 14 · p. 31

J Inference Cost Details

  • Prompt-based stateful simulators cost substantially more than stateless methods, while trained ARGUS variants reduce per-turn input to a level comparable with Vanilla.The cost analysis uses per-turn and per-dialogue costs for prompt-based models and actual token-usage records for trained simulators.Source: J Inference Cost Details · p. 32, Table 15 · p. 32, Table 16 · p. 32

K SFT vs. RL: Per-Dimension Analysis

RL performs better on Faithfulness & Consistency and Policy Adherence, while SFT leads on Goal Completion; both approaches are near ceiling on Format & Protocol. CBS’s harmonic mean keeps SFT’s overall score below RL because its shortfalls on the two semantically demanding dimensions outweigh its Goal Completion lead.

Per-Dimension Results

  • RL scores 95.6 vs SFT’s 90.2 (30k data) on Faithfulness & Consistency and 82.8 vs SFT’s 78.3 (20k) on Policy Adherence.The gaps persist across SFT data scales.Source: K SFT vs. RL: Per-Dimension Analysis · p. 32, Figure 9 · p. 33
  • SFT scores 80.1 vs RL’s 74.9 on Goal Completion at 20k, primarily driven by lower Premature Termination rates.The authors hypothesize that dense step-level RL rewards can induce a risk-averse, myopic effect that encourages premature dialogue termination.Source: K SFT vs. RL: Per-Dimension Analysis · p. 33

Future Work

Per-Dimension Results

CBS Aggregation

  • CBS’s harmonic mean keeps SFT’s overall score below RL because SFT’s FC and PA shortfalls outweigh its Goal Completion lead.The aggregation rewards balanced performance across dimensions rather than allowing strength in one dimension to compensate for weaknesses in others.Source: K SFT vs. RL: Per-Dimension Analysis · p. 33

L Benchmark Data Composition

This appendix section documents benchmark-split usage and prompt templates for structuring and auditing simulated-user goals and interactions. The prompts cover agenda-stack transitions, faithfulness, policy compliance, goal completion, factual grounding, and output protocol across ARGUS phases and baselines.

L Benchmark Data Composition

  • The benchmark split table summarizes dataset sizes; learned simulators use pipeline-constructed SFT and RL instances, while prompt-based simulators use test splits, and Airline and Telecom use only test data.The SFT and RL entries for Airline and Telecom are empty.Source: L Benchmark Data Composition · p. 33, Table 17 · p. 34

M Prompts

  • The initial extraction prompt converts a scenario into a JSON user profile and task list, separating happy-path requirements from conditional behaviors and grouping actions by transaction context.It assigns personality, communication policy, and static facts to the profile, while fallbacks and other conditional actions go in conditional behaviors.Source: M Prompts · p. 35
  • The structured-goal verification prompt checks task grouping, separation of requirements from exceptions, exact data integrity, profile accuracy, and preservation of source information.Any critical rule violation yields a score of 0.Source: M Prompts · p. 36
  • The stack evaluator’s soundness checks require well-formed operations on the current top frame, lifecycle evidence for changes, and appropriate use of wait.A pop requires direct completion or abandonment evidence in agent or tool history; wait is appropriate while the agent is working and no user action is due.Source: M Prompts · p. 37
  • The stack evaluator’s faithfulness and adherence checks preserve grounded arguments, unresolved goals, and active constraints while applying each fired conditional exactly once.Conditions must be classified from conversation evidence and mutations must use the supplied condition index.Source: M Prompts · pp. 37–38
  • The turn-level NLG rubric uses binary checks for actionable progress, goal consistency, termination mechanics, factual grounding, dialogue-state response, privacy, constraints, conditional policies, and persona instructions.It requires auditors to judge only the current actionable turn, treat candidate reasoning as non-evidence, and inspect the raw response for STOP placement and content after STOP.Source: M Prompts · pp. 39–40
  • The faithfulness-and-consistency prompt checks for fabricated details, entity-attribute confusion, ungrounded confirmation, and withholding known scenario information.It instructs evaluators to verify each claim against the scenario and each agent question against available scenario information, while not penalizing user turns for agent-only task failures.Source: M Prompts · p. 42
  • The policy-adherence prompt focuses on privacy leaks, premature disclosures, and missed conditional triggers, treating any restricted-information leak or missed trigger as failure.Its framing distinguishes being overly revealing from failing to act when a conditional policy fires.Source: M Prompts · p. 43
  • The policy audit fails users for premature disclosure, hard-constraint breaches, or missed or false conditional triggers, while distinguishing user compliance from agent failure.The workflow extracts scenario rules, traces whether they were triggered, logs the user’s action, and assigns a zero for any logged violation.Source: M Prompts · pp. 43–44
  • The goal auditor checks whether users explicitly state every scenario requirement, regardless of the agent’s ability to fulfill the request.It marks incomplete checklists or premature termination as failures, while allowing the user to stop if the agent definitively blocks further progress.Source: M Prompts · p. 45
  • The format audit flags missing or malformed termination tokens, mixed text and commands, and mechanical repetition, while not penalizing mere insistence on a goal.The prompt directs evaluators to inspect every user turn and score syntax violations as failures.Source: M Prompts · pp. 46–47
  • The stack operator updates agenda and condition state through symbolic calls, leaving conversational response generation to a separate module.Its workflow checks task completion, agent inquiries, and condition triggers in order; subtasks are removed upon user fulfillment, while main tasks require explicit evidence of agent execution.Source: M Prompts · pp. 47–48
  • The stack rules preserve unresolved goals, enforce privacy and grounding, and trigger conditions only when the agent’s latest turn semantically matches them.Main tasks require definitive execution evidence, and ungrounded identifiers from the agent must not be added to task requirements.Source: M Prompts · p. 48
  • The response-generation prompt makes the NLG component use dialogue history, the updated agenda stack, user profile, and active condition pool while leaving state and tool management to other components.It requires focus on the top agenda task, preservation of pending goals, persona alignment, non-fabrication, ordered fallback handling, and a stop token only after transfer or task completion.Source: M Prompts · p. 49
  • Trained ARGUS prompts separate stack-operation output from natural-language response generation and inject agenda and condition state into each phase.Phase 1 requests reasoning and tool calls from the pre-update stack and filtered pool; Phase 2 supplies the updated stack, task update, filtered pool, and recent triggered conditions.Source: M Prompts · pp. 50–51
  • The ReAct baseline prompts private brief reasoning before a grounded, progressive-disclosure user response and requires the exact stop string when the task is complete.Its output format requires one thought block and one response block, with no additional content.Source: M Prompts · p. 52
  • The Verify baseline classifies a candidate response as true or false, while Reflection revises an unsatisfactory response using a reflection and replacement-response format.Verify requests a single classification token; Reflection asks for a grounded diagnosis and a stepwise correction without overloading the agent.Source: M Prompts · pp. 52–53

Related papers

Loading 2610.21346v1…