TrustAgent: Towards Safe and Trustworthy LLM-based Agents

1 Introduction

• Tool-using agents’ interactions with the physical world and people create safety concerns beyond those of text-focused LLMs, including risks such as passcode exposure and failure to activate fume hoods.These examples motivate robust safety knowledge for agents.Source: §1 · p. 1
• The paper introduces an Agent Constitution focused on safe actions and tool use, addressing limited research on formulating comprehensible rules and guiding agents to follow them during planning.The paper contrasts this action-focused emphasis with attention to verbal harm.Source: §1 · p. 1
• TrustAgent combines pre-planning safety knowledge injection, in-planning moderation, and post-planning inspection, and is evaluated across five domains with safety, helpfulness, and plan metrics.The evaluation covers housekeeping, finance, medicine, chemistry experiments, and food, using four closed-source models and Mixtral-8x7B-Instruct.Source: §1 · pp. 1–2
• TrustAgent significantly enhances both agent safety and helpfulness, while the findings identify inherent LLM reasoning as critical for safe adherence in complex scenarios.The paper argues that safety protocols alone are not enough; reasoning capabilities also matter.Source: §1 · p. 2

2 Related Work

• LLM trustworthiness encompasses truthfulness, safety, fairness, robustness, privacy, and machine ethics, and prior work has proposed principles and methods governing textual outputs.Other cited work assesses LLM understanding of basic moral concepts.Source: §2 · p. 2
• Agent alignment must account for physical actions and interactions with tools and environments, extending beyond conversational output requirements.The passage describes this as an added layer of complexity for real-world problem-solving agents.Source: §2 · p. 2
• Prior agent-safety work often observes and assesses risks, while AgentMonitor is described as a rudimentary monitoring tool for identifying unsafe scenarios.The paper proposes a framework intended to improve agent safety through an Agent Constitution and three strategies.Source: §2 · p. 2

3 Design of Agent Constitution

• The constitution’s scope includes agent-human conduct, multi-agent relations, and tool or environment interactions; this paper focuses on safety regulations for single-agent tool use.The paper recommends collaborative drafting authority involving ethicists, legal experts, technologists, and public- and private-sector representatives, while basing its own constitution on existing tool-use regulations.Source: §3 · p. 3
• The paper adopts explicit rule-based regulations because well-formatted precedents pairing agent actions with safety suggestions or critiques are scarce.It notes that larger collections of precedents may become available as agents develop and are used.Source: §3 · p. 3

3.1 Agent Constitution Implementation: The TrustAgent Framework

• TrustAgent is an LLM-based emulation framework whose safety strategies augment agent decision-making under an Agent Constitution, with safety measures integrated across planning stages.The paper presents the pipeline as an alternative to relying only on post-execution safety verification.Source: §3.1 · pp. 3–4

3.2 Pre-planning Safety

• Pre-planning safety injects safety knowledge before action planning through regulation learning and hindsight learning.Regulation learning reformulates each rule into five question-and-answer variants; hindsight learning uses instructions, tentative plans, and inspector critiques as examples.Source: §3.2 · p. 4

3.3 In-planninng Safety

• TrustAgent implements in-planning safety through prompting, retrieving the five most relevant regulations iteratively using the user instruction and current plan trajectory.The paper discusses prompting and decoding as possible controls, but implements prompting alone in this stage.Source: §3.3 · p. 4

3.4 Post-planning Safety

• Post-planning safety uses an inspector to check regulation relevance and plan adherence, then gives feedback for revisions when it finds a violation.The process halts for safety concerns if the planning agent repeats the same mistake despite suggestions.Source: §3.4 · p. 5
• Inspector-agent interactions can form hindsight-learning data containing instructions, trajectories, proposed steps, relevant regulations, and safe-or-unsafe feedback with explanations.The described training uses positive feedback to train safe next steps and negative feedback to train unsafe steps aligned with criticism.Source: §3.4 · p. 5

4 Experiment

• The 70-point evaluation covers five domains and assesses safety, helpfulness, trajectory overlap, correct prefixes, and total steps across five backbone models.Everyday and finance data are adapted from ToolEmu; the other domains are manually constructed, and models are run twice per datapoint and averaged.Source: §4 · p. 6

4.1 Experiment Result

• Across domains, TrustAgent’s three safety strategies improve safety, improve helpfulness in medicine, food, and chemistry, and GPT-4 remains the safest and most helpful model.Table 2 reports safety, helpfulness, correct steps, correct-prefix length, and total steps with and without the strategies.Source: §4.1 · p. 6, Table 2 · p. 7

TrustAgent improves action order alignment

• TrustAgent narrows the gaps between prefix steps and total steps and between prefix steps and correct steps, indicating closer alignment with the ground-truth action order.Table 3 reports prefix/correct and prefix/total ratios for action trajectories.Source: Table 3 · p. 7, TrustAgent improves action order… · p. 8

4.2 Ablation Study

• In medicine, inspection raises safety scores above 2 for all models, while prompting alone leaves GPT-3.5 and Mixtral-Instruct below 2.Prompting alone exceeds 2 for GPT-4, Claude-2, and Claude-instant; Table 4 compares prompting-only and inspection-only results.Source: Table 4 · p. 7, §4.2 · p. 8
• Prompting increases trajectory length, inspection reduces it by interrupting repeated mistakes, and combining them keeps total steps stable while increasing the share of correct actions.The combined approach also increases correct prefixes relative to total steps.Source: §4.2 · p. 8
• Pre-planning fine-tuning, tested only on GPT-3.5 with the available small dataset, produced no significant improvement or decline across five domains or metrics.Table 5 reports the GPT-3.5 pre-planning-only results.Source: §4.2 · p. 8, Table 5 · p. 8

• TrustAgent implements the Agent Constitution as the paper’s principal mechanism for addressing agent safety, a foundation of trustworthiness.Source: §5 · p. 8
• The experimental findings report that TrustAgent improves agents’ safety and helpfulness.Source: §5 · p. 8
• Future work calls for more Agent Constitution design and highlights regulation-specific decoding and pre-planning learning as promising strategies.The authors suggest large-scale preference data with RLHF or Direct Policy Optimization as possible approaches.Source: §5 · p. 8

• The study focuses on safety, leaving explainability, fairness, controllability, robustness, and other trustworthiness attributes for broader future work.Source: Limitations · pp. 8–9
• The study uses a limited number of data points because scenarios involving unsafe actions and negative consequences are difficult to collect and generate.Source: Limitations · p. 9
• The framework does not yet incorporate highly complex technical methods for its three safety strategies, because the study prioritizes proposing the Agent Constitution and framework.Source: Limitations · p. 9

• The Agent Constitution has two parts: general-domain safety regulations and domain-specific safety regulations.Source: A Agent Constitution: Regulations · p. 10
• General-domain regulations provide universal safety protocols that are not tailored to particular tools, technologies, or environments.Source: A Agent Constitution: Regulations · p. 10
• Domain-specific regulations address a domain’s particular tools and circumstances with more granular, actionable guidance.Source: A Agent Constitution: Regulations · p. 10

A.1 General-Domain Agent Constitution

• Asimov’s Four Laws anchor general-domain regulations, with the humanity-level directive taking precedence over the first three.Source: §A.1 · p. 10
• General-domain rules can be too abstract and ambiguous to guide specific decisions consistently, especially when concepts such as “harm” lack operational definitions.Source: §A.1 · p. 10

A.2 Domain-Specific Agent Constitution

• Domain-specific regulations cover everyday, finance, medicine, food, and chemistry, combining formal guidance with practical common-sense norms.Source: §A.2 · pp. 10–11

B Example of Chain of Hindsight

• The example asks the model to generate a regulation-violating action trajectory and explain why it violates the rule.The prompt supplies both a user instruction and a safety regulation.Source: B Example of Chain of Hindsight · p. 12
• The example regulation requires a reasonable, acceptable sale price for assets including BTC.Source: B Example of Chain of Hindsight · p. 12
• The trajectory checks BTC’s price and balance, then places a five-BTC limit sell order at price 0.Source: B Example of Chain of Hindsight · p. 12
• The explanation says the below-market sale could leave the user with less money than expected and recommends a price closer to market or a market order.Source: B Example of Chain of Hindsight · p. 12

C Dataset

• The dataset spans housekeeping, finance, medicine, food, and chemistry, with everyday and finance examples sourced from ToolEmu and the other domains constructed manually.Source: C Dataset · p. 12
• Housekeeping and finance scenarios include unauthorized home access, personal-information leaks, overdrafts, and transactions with unsuitable prices, parties, or amounts.Source: C Dataset · pp. 12–13
• Medicine, food, and chemistry scenarios cover medication risks, allergies and cross-contamination, and unsafe laboratory procedures or measurements.Source: C Dataset · p. 13
• The authors observe that LLMs may know relevant safety information when directly asked but fail to apply it during planning.Source: C Dataset · p. 13

D.1 Case Study

• The instruction calls for checking interactions, allergies, and expiration before administering Naproxen, if safe.Source: §D.1 · p. 13
• Before TrustAgent, GPT-3.5 and Mixtral give Naproxen without checks, whereas GPT-4 checks interactions and allergies, finds the medicine expired, and disposes of it.Source: §D.1 · p. 13
• After TrustAgent, GPT-3.5 and Claude-2 add medication-history checks, while Mixtral checks Andy’s age and condition and rejects the request after finding a possible interaction.Source: §D.1 · p. 13
• Safe action trajectories can be longer and more complex, and TrustAgent does not intrinsically improve an LLM’s logical reasoning ability.Source: §D.1 · pp. 13–14
• The authors say TrustAgent is especially useful with agents that already have sufficient reasoning skills; limited-reasoning models may struggle to combine safety with practical task execution.Source: §D.1 · p. 14