PaperlayerView exact revision on arXiv

Source-linked AI summary

Preemptive Detection and Correction of Misaligned Actions in LLM Agents

Haishuo Fang, Xiaodan Zhu, Iryna Gurevych

arXiv:2407.11843v4cs.CLcs.AI

TL;DR

Misaligned LLM-agent actions can have serious consequences, yet preemptive detection and correction remain underexplored. InferAct uses LLM belief reasoning grounded in Theory of Mind to detect misalignment, and outperforms baselines across tasks and settings. Its collaborative evaluation also reports improved agent success and reduced human oversight.

  • Problem

    Misaligned critical actions can cause harmful or irreversible outcomes, while effective pre-execution detection and correction methods remain underexplored.

    Source: §1 · p. 1

  • Method

    InferAct uses LLM belief reasoning grounded in Theory of Mind to infer agent intent and compare it with the user’s instruction.

    Source: §1 · p. 2

  • Results

    InferAct outperforms baselines in 11 of 12 settings across three tasks, with up to 20% improvement on Macro-F1 for misaligned-action detection.

    Source: §1 · p. 2

  • Takeaways & Limitations

    With InferAct assistance, human oversight can be reduced by 50% with a 3% performance drop versus fully manual inspection.

    Source: §1 · p. 2

    The study’s combined false-positive and false-negative cost metric may not capture the complexity of real-world consequences.

    Source: §7 · p. 10

Abstract

from arXiv · show

Deploying LLM-based agents in real-life applications often faces a critical challenge: the misalignment between agents' behavior and user intent. Such misalignment may lead agents to unintentionally execute critical actions that carry negative outcomes (e.g., accidentally triggering a "buy-now" in web shopping), resulting in undesirable or even irreversible consequences. Although addressing these issues is crucial, the preemptive detection and correction of misaligned actions remains relatively underexplored. To fill this gap, we introduce InferAct, a novel approach that leverages the belief reasoning ability of LLMs, grounded in Theory-of-Mind, to detect misaligned actions before execution. Once the misalignment is detected, InferAct alerts users for timely correction, preventing adverse outcomes and enhancing the reliability of LLM agents' decision-making processes. Experiments on three widely used tasks demonstrate that InferAct achieves up to 20% improvements on Marco-F1 against baselines in misaligned action detection. An in-depth evaluation of misalignment correction further highlights InferAct's effectiveness in improving agent alignment.

1 Introduction

Misaligned critical actions can cause operational failures, erode trust, or produce irreversible outcomes, while manual action checks burden users. InferAct uses LLM belief reasoning to detect misalignment before execution, and the paper reports strong detection and collaboration results.

1 Introduction

  • Misexecuted critical actions can cause operational failures, erode user trust, or lead to irreversible outcomes, while action misalignment remains underexplored.Manual validation can prevent unintended errors but imposes cognitive burden and limits agent autonomy.Source: §1 · p. 1
  • InferAct applies LLM belief reasoning to infer an agent’s intent, compare it with the user instruction, and abstract action trajectories into high-level goals.It is triggered only for pre-identified critical actions with negative consequences.Source: §1 · p. 2
  • Across three tasks, InferAct outperforms baselines in 11 of 12 settings and improves Macro-F1 by up to 20%.Evaluations span web shopping, household, and search-based question answering tasks.Source: §1 · p. 2
  • The paper proposes a collaborative workflow in which InferAct alerts users to verify and correct detected misalignment through feedback.The workflow is intended to improve agent performance while reducing adverse outcomes from misaligned actions.Source: §1 · p. 2
  • With InferAct guidance, the Actor agent’s success rate improves by 10.4% over alternative methods using natural-language feedback.Human oversight falls by 50% with only a 3% performance drop versus fully manual inspection.Source: §1 · p. 2

2 Related Work

Related work addresses agent trustworthiness, evaluation methods, and machine Theory of Mind. InferAct is positioned as an evaluator of real-time alignment that uses LLM ToM capabilities to detect misaligned behavior.

2 Related Work

  • InferAct evaluates real-time alignment between agent behavior and user intent, unlike sandbox emulation methods that struggle to model complex real-world scenarios such as web shopping.The paper distinguishes its evaluator role from work on deciding task delegation between humans and agents.Source: §2 · p. 2
  • Prior LLM-as-a-judge work includes pairwise comparison, single-answer grading, and reference-guided grading; InferAct uses single-answer grading as a baseline.Agentic tasks often lack standardized references or process annotations, limiting some evaluation methods.Source: §2 · pp. 2–3
  • In a Webshop example, InferAct detects the mismatch between selected custom-sized shades and the user’s requested 66 × 66 inch shades while other methods fail.Source: Figure 2 · p. 3
  • Prior studies evaluate LLM Theory-of-Mind abilities, whereas InferAct leverages those abilities to help detect misaligned agent behavior.The paper identifies this shift from evaluating ToM to applying it as its distinction from related work.Source: §2 · p. 3

3 Approach

InferAct infers an agent’s intent from its observed behavior and checks it against the user’s task. Its collaborative workflow monitors selected critical actions and alerts a human to verify and correct flagged behavior.

3.1 InferAct

  • InferAct uses belief reasoning to infer the Actor’s task from its observed actions and observations, then verifies that inferred task against the user’s task.Unlike self-reflection, belief reasoning takes a third-person perspective based on observable behavior.Source: §3.1 · pp. 3–4
  • A two-stage verification checks whether the inferred task fulfills the user’s task and, if not, whether it represents valid progress toward that task.The progress check avoids treating an intermediate subgoal as misalignment; InferAct detects misaligned actions rather than terminating actions that may contribute to solution exploration.Source: §3.1 · p. 4
  • The setup assumes the user’s task is clear and unambiguous; handling ambiguous instructions is outside the study’s scope.Source: §3.1 · p. 4

3.2 Collaborative workflow between InferAct, Actor and the User

  • Users define which consequential actions to monitor; when InferAct flags behavior, a human makes the final judgment and provides corrective feedback.Selective monitoring is intended to balance safety with the Actor’s autonomy and efficiency.Source: §3.2 · pp. 4–5
  • The workflow may not scale seamlessly to open-domain settings, while the paper describes InferAct as adaptable to different LLM-based agent environments, particularly high-stakes setups.The authors say domain experts remain indispensable in high-stakes domains.Source: §3.2 · p. 5

4 Experimental Setup

The experimental setup evaluates InferAct on three interactive agent benchmarks, using detection metrics and prompting-based baselines across several detector backbones.

4.1 Tasks

  • Evaluation covers WebShop, HotPotQA, and ALFWorld, with critical actions defined for each benchmark.Examples include Buy Now in WebShop, final answers in HotPotQA, and Clean, Heat, or Cool in ALFWorld.Source: §4.1 · p. 5

4.2 Evaluation Metrics

  • The detector is evaluated with Macro-F1, Total Detection Errors, Effective Reliability, and PR-AUC.The metrics respectively address balance between detection and usability, combined false negatives and false positives, reliability of flagged detections, and performance across thresholds.Source: §4.2 · p. 5

4.3 Baselines and Backbone LLMs

  • Prompting-based baselines include direct judgment, five-sample self-consistency, token probability, token entropy, and multi-step evaluation; InferAct has verbal and probability-output variants.Probability-based methods require development data to tune decision thresholds.Source: §4.3 · pp. 5–6
  • The Actor uses gpt-4-1106-preview, while detectors are built on Llama-3 (70B), gpt-3.5-turbo-0613, gpt-4-1106-preview, and Qwen3-4B.Source: §4.3 · p. 6

5 Experiment Results and Analysis

InferAct achieves the strongest results in most evaluated benchmark settings. Further analyses examine model pairing, risk and calibration, coding scenarios, and collaboration between the detector, Actor, and users.

5.1 Overall Performance

  • InferAct also outperforms alternatives on PR-AUC in 11 of 12 settings; InferAct-verb scores 0.624 versus 0.655 for InferAct-prob.InferAct-verb is identified as the best choice when no validation set is available for threshold tuning.Source: §5.1 · p. 6
  • Average Macro-F1 scores are 0.576 for Multi-step, 0.563 for Token Probability, 0.524 for Token Entropy, 0.485 for Direct Prompt, and 0.448 for Self-Consistency.Probability-based methods outperform direct prompting but require a development set for threshold tuning; Self-Consistency fluctuates across models.Source: §5.1 · p. 6

5.2 Analysis

  • Model pairings vary by task, and mixing models can improve performance at lower cost; GPT3.5-Turbo inference with GPT4-Turbo verification scores 0.662 on HotpotQA.This pairing is cheaper than using GPT4-Turbo for both components; InferAct is also described as cheaper than Self-Consistency and Multi-Step while delivering better performance.Source: §5.2 · pp. 6–7, Figure 4 · p. 7, Table 2 · p. 7

5.1 Overall Performance

  • InferAct leads in 11 of 12 settings on Macro-F1, outperforming the strongest baseline by an average of 8%.Table 1 covers three tasks and four back-end LLMs.Source: §5.1 · p. 6, Table 1 · p. 7

5.2 Analysis

  • InferAct-verb has the lowest false negative rate across risk levels; InferAct-prob achieves the best average calibration, with ALFWorld ECE 0.116 versus Token Probability’s 0.583 using GPT-35-Turbo.WebShop high-risk actions are purchases over $60, while ALFWorld high-risk actions are heating or cooling.Source: §5.2 · pp. 7–8, Table 3 · p. 8, Table 4 · p. 8
  • In 30 selected high-risk R-Judge coding tasks, InferAct effectively flags potentially dangerous operations and supports safe human-in-the-loop intervention.The selected tasks involved high-risk bash commands or API calls and were evaluated with Qwen-3-4B.Source: §5.2 · p. 8, Table 5 · p. 8

5.3 Synergy Between InferAct, Actor, and Users

  • Across three iterations, the Actor guided by InferAct achieves the highest success rates with both binary and natural-language feedback, surpassing Token Entropy by 5% on WebShop with natural-language feedback.Figure 5 and Table 6 report the feedback-guided Actor results.Source: Figure 5 · p. 9, Table 6 · p. 9, §5.3 · p. 9
  • InferAct-prob lags Full Validation by an average of 3.5% with binary feedback while supporting up to 50% fewer human interventions; in a three-user WebShop study, the guided Actor performs best with human feedback.The user study is described as small-scale.Source: §5.3 · p. 9

6 Conclusion

The conclusion presents InferAct as a Theory-of-Mind belief-reasoning approach for detecting deviations from user intent. It reports results across environments and LLMs and describes collaboration with the Actor and user.

6 Conclusion

  • InferAct uses belief reasoning grounded in Theory of Mind to detect whether an agent deviates from user intent.The conclusion frames preemptive detection and correction as important for real-world deployment of LLM-based agents.Source: §6 · p. 9
  • Experiments show InferAct outperforms alternative methods across different environments and LLMs.Source: §6 · p. 9
  • The paper reports that collaboration among InferAct, the Actor, and the user prevents misaligned actions and enhances the Actor’s performance.The conclusion identifies automatic detectors as potential proxies for human oversight that can reduce cognitive burden.Source: §6 · p. 9

7 Limitations

The authors identify limitations in cost modeling, the approach’s target, and the scope of risky-action definitions.

7 Limitations

  • Summing false positives and false negatives simplifies real-world costs and omits long-term and indirect effects.The authors cite different consequences for missed unsafe actions and incorrectly flagged safe actions, motivating more fine-grained cost modeling.Source: §7 · p. 10
  • InferAct addresses misalignment with user intent, not harmful user intent such as making a bomb.Source: §7 · p. 10
  • Risky actions are manually defined for the relatively small tested action spaces; automatic discovery in open domains remains a research direction.The authors specify that such discovery would be under human control.Source: §7 · p. 10

A Anslysis (Cont.)

This appendix examines prompt variation and model scale, reports a Micro-F1 result and a self-reflection comparison, and describes prompts used for baselines, InferAct, and feedback.

A Anslysis (Cont.)

  • Robustness testing varied InferAct prompts through rephrasing and synonym substitution, including removal of the explicit Theory-of-Mind capability sentence.The appendix caption identifies the corresponding robustness table but supplies no outcome in these passages.Source: A Anslysis (Cont.) · p. 13, Table 7 · p. 13
  • Increasing Qwen2.5 model size from 3B to 72B does not guarantee better performance for InferAct or Direct Prompt.The figure reports Macro-F1 across tasks and model sizes.Source: A Anslysis (Cont.) · p. 13, Figure 6 · p. 14
  • InferAct-verb outperforms other methods in Micro-F1, an alternative metric for users who prioritize the major class.The section notes that Macro-F1 is used when undetected errors and false alarms are equally important.Source: A Anslysis (Cont.) · p. 14, Table 8 · p. 14

B Comparison between self-reflection and InferAct

C.1 Prompts Used for Baseline Methods

  • Baseline prompt materials cover Direct Prompt, Multi-step Evaluation, and Token Probability and Entropy.The appendix points to Figures 7–15 for these prompts.Source: §C.1 · p. 14

C.2 Prompts for InferAct

  • InferAct prompts infer a task from an action-observation sequence, then check task alignment and whether the agent remains on track.The task-inference prompt includes task background and belief reasoning; examples appear in Figures 16–21.Source: §C.2 · p. 14

C.3 Natural Language Feedback from AI

  • Natural-language feedback prompts are provided separately for WebShop, ALFWorld, and HotpotQA.Source: §C.3 · p. 14

D Details of Experiments

This section details InferAct’s experimental prompts, controls, trajectory handling, threshold tuning, oracle inspection, and ground-truth labeling, including observations that inferred intents were usually unambiguous and invalid-format outputs rare. It also distinguishes the approach from prior evaluation and offline-learning work by its focus on real-time detection and correction of misaligned actions using LLM Theory-of-Mind to help humans identify off-track behavior in critical decisions.

Prompt Details

  • Prompts assess step correctness with probabilities and justifications, infer intended tasks from behavior, verify task entailment, and request constructive feedback without revealing the answer or product.The prompts cover WebShop, ALFWorld, and HotPotQA; household-task feedback also asks the oracle to point out errors rather than provide the correct action chain.Source: D Details of Experiments · pp. 15, 17, 19

Data Statistics and Threshold

Thresholds, Oracle Inspection, and Ground Truth

  • Probability-based Llama-3-70B thresholds were tuned on a 50-example development set, oracle inspection per iteration was limited, and ground truth used task-specific labels.WebShop and HotPotQA use gold labels for terminal critical actions; ALFWorld trajectories are labeled from human demonstrations in the original dataset. The number of trajectories available for oracle inspection is specified in Table 12.Source: D Details of Experiments · p. 20, Table 12 · p. 20, Table 11 · p. 20

Edge Cases due to Output Format

  • Invalid Llama-3-70B outputs were flagged for human review, and fewer than 2% failed to follow the required format.Invalid outputs could not provide a judgment and were therefore flagged as positive for human involvement.Source: D Details of Experiments · p. 20

Reliability of Inferred Intent

  • Only 2 of 100 WebShop intents inferred by InferAct were ambiguous, supporting the reliability of its inferred intent.The intents were manually examined.Source: D Details of Experiments · p. 20

E Related Work

  • Existing safety approaches include tool-execution emulation and a static-trajectory benchmark, but emulation may not match complex real-world execution.ToolEmu was proposed by Ruan et al. (2024), Agent constitution by Hua et al. (2024) adds preplanning, in-planning, and post-planning evaluation, and R-Judge was proposed by Yuan et al. (2024).Source: E Related Work · p. 20
  • The study adopts single-answer grading and token entropy, while meta-ranking is limited by the lack of standardized agent-task references or process annotations.Zheng et al. (2023) outlines grading approaches; Han et al. (2024) and Lin et al. (2024) examine uncertainty metrics; Liu et al. (2024b) proposes meta-ranking.Source: E Related Work · pp. 20–21
  • Prior work typically assumes feedback is available after execution or unavailable during inference, whereas this work seeks human feedback during online operations.Prior offline approaches use gold labels or human feedback to learn from past trajectories, including Co-learning, ExpeL (Zhao et al., 2024), and contrastive fine-tuning.Source: E Related Work · p. 21
  • Unlike Pan et al. (2024)’s whole-trajectory evaluation for web-agent performance, this work prioritizes real-time misaligned-action detection and correction, with evaluator–Actor–user collaboration in critical decisions.It contrasts with offline learning from past trajectories and targets preventing negative consequences from irreversible actions during online operations.Source: E Related Work · p. 21
  • This work takes a preliminary step toward using LLMs’ Theory-of-Mind to help humans detect agents’ off-track behavior in critical decisions.Prior work mostly evaluates LLM ToM; reported capabilities include strengths on false beliefs and non-literal expressions, but difficulty recognizing faux pas.Source: E Related Work · p. 21

F Results for Multi-Step Evaluation

The Multi-step Evaluation comparison identifies Product as the most effective aggregation method across all tasks.

F Results for Multi-Step Evaluation

G Task Description

The section describes three agent tasks and identifies their critical actions and evaluation settings.

G Task Description

  • In WebShop, agents browse an e-commerce site to buy requested products; click[Buy Now] is critical, and evaluation uses Actor success rate.The evaluation covers 300 tasks; the dataset contains 1.18 million products and 12k user instructions.Source: G Task Description · p. 22
  • HotPotQA agents use Wikipedia search, lookup, and finish actions; finish[answer] is critical, and evaluation uses exact match.The evaluation comprises 300 tasks.Source: G Task Description · p. 22
  • ALFWorld agents complete household tasks; Clean, Heat, Cool, and task completion are critical actions.The task includes six task types and is evaluated on 134 unseen validation tasks.Source: G Task Description · p. 22

H User Study for collaboration between InferAct, Actor, Human

A WebShop user study tested InferAct with feedback from three human users on 100 randomly sampled tasks. The Actor guided by InferAct achieved the highest success rate with human feedback, while GPT-4-Turbo feedback achieved comparable performance.

H User Study for collaboration between InferAct, Actor, Human

Loading 2407.11843v4…