Source-linked AI summary
AgentSpec: Customizable Runtime Enforcement for Safe and Reliable LLM Agents
Haoyu Wang, Christopher M. Poskitt, Jun Sun
TL;DR
LLM agents can take harmful or unlawful actions autonomously, while existing safeguards have limitations in robustness, interpretability, adaptability, and runtime enforcement. AgentSpec introduces a lightweight DSL for defining and enforcing runtime constraints across agent domains. Its evaluations report strong safety outcomes across code, embodied-agent, and autonomous-driving tasks, with millisecond-level overhead, while identifying trajectory-based safety analysis as an important limitation.
Problem
LLM-agent autonomy creates risks including security vulnerabilities, legal violations, and unintended harmful actions, while existing mitigations have limited robustness, interpretability, adaptability, or runtime enforcement.
Method
AgentSpec uses a lightweight domain-specific language in which users define triggers, predicates, and enforcement mechanisms for runtime intervention across LLM-agent domains.
Results
AgentSpec prevents unsafe code executions, eliminates hazardous embodied-agent actions, ensures lawful autonomous-driving decisions, and operates with millisecond-level overhead.
Takeaways & Limitations
AgentSpec provides an interpretable, modular, and efficient approach to enforcing customizable safety constraints across diverse LLM-agent applications.
Takeaways & Limitations
AgentSpec enforces deterministically at discrete checkpoints and does not analyze whether action sequences may lead to unsafe states several steps into the future.
Abstract
from arXiv · showhide
Agents built on LLMs are increasingly deployed across diverse domains, automating complex decision-making and task execution. However, their autonomy introduces safety risks, including security vulnerabilities, legal violations, and unintended harmful actions. Existing mitigation methods, such as model-based safeguards and early enforcement strategies, fall short in robustness, interpretability, and adaptability. To address these challenges, we propose AgentSpec, a lightweight domain-specific language for specifying and enforcing runtime constraints on LLM agents. With AgentSpec, users define structured rules that incorporate triggers, predicates, and enforcement mechanisms, ensuring agents operate within predefined safety boundaries. We implement AgentSpec across multiple domains, including code execution, embodied agents, and autonomous driving, demonstrating its adaptability and effectiveness. Our evaluation shows that AgentSpec successfully prevents unsafe executions in over 90% of code agent cases, eliminates all hazardous actions in embodied agent tasks, and enforces 100% compliance by autonomous vehicles (AVs). Despite its strong safety guarantees, AgentSpec remains computationally lightweight, with overheads in milliseconds. By combining interpretability, modularity, and efficiency, AgentSpec provides a practical and scalable solution for enforcing LLM agent safety across diverse applications. We also automate the generation of rules using LLMs and assess their effectiveness. Our evaluation shows that the rules generated by OpenAI o1 achieve a precision of 95.56% and recall of 70.96% for embodied agents, successfully identify 87.26% of the risky code, and prevent AVs from breaking laws in 5 out of 8 scenarios.
1 Introduction
LLM agents automate tasks across domains but can diverge from user expectations, while existing safeguards often lack interpretability, generalizability, or active runtime enforcement. AgentSpec addresses these gaps with customizable runtime rules and demonstrates safety improvements across code, embodied-agent, and autonomous-driving tasks.
- LLM agents autonomously perceive, plan, and act across domains including software development, healthcare, and autonomous systems.
- Autonomous execution can produce context-dependent safety risks, such as unreviewed financial transfers or medication changes that may be acceptable in one setting but unsafe in another.
- Existing mitigations trade off risk detection, interpretability, generalizability, and enforcement, leaving agents vulnerable to runtime deviations.
- AgentSpec specifies triggers, predicates, and enforcement mechanisms in a domain-specific language, with rules defined manually or generated automatically for review.
- Over 90% of unsafe code executions were prevented, all hazardous embodied-agent actions were eliminated, and autonomous vehicles achieved full compliance in tested law-violation scenarios.
2 Background and Problem Definition
LLM agents iteratively transform observations into states and actions, enabling autonomous interaction but creating risks from unconstrained execution. AgentSpec formulates runtime safety as continuous trajectory monitoring and intervention through human-readable, domain-agnostic rules.
- 2.1 LLM Agents: An LLM agent is modeled as a tuple (S, A, Ω, Π, ∆) representing states, actions, observations, perception, and policy.
- 2.1 LLM Agents: Agents repeatedly receive instructions, update internal states from observations, and plan actions, producing a trajectory over time.
- 2.1 LLM Agents: Unconstrained autonomous execution can cause data loss, privacy violations, or unsafe system modifications.
- 2.2 Motivating Example: A transaction rule requests explicit confirmation when the recipient is not a verified family member, allowing approval to proceed or otherwise aborting the action.
- 2.3 Problem Definition and Goal: AgentSpec enforces human-readable constraints in real time rather than relying on static policies or post-hoc evaluations.
- 2.3 Problem Definition and Goal: Runtime enforcement evaluates the current trajectory and planned action, intervening when their combination could violate a rule and aiming to keep the resulting trajectory safe.
3 The AgentSpec Language
AgentSpec is a DSL for expressing customizable safety rules that regulate LLM-agent behavior in real time. Its semantics combine monitored triggers, predicates, and enforcement functions to intervene when planned actions violate specified constraints.
- Language overview: AgentSpec rules regulate agent behavior through conditions and enforcement actions tied to specific inputs or situations.The DSL is intended to balance strict behavioral constraints with flexibility across domains.
- Rule structure: Each rule comprises a trigger, predicate checks, and enforcement functions that determine when and how intervention occurs.Triggers may precede actions, respond to state changes, or detect task completion.
- Triggers: AgentSpec monitors general and domain-specific events, including actions, state changes, environmental signals, obstacles, traffic signals, and road events.Its event system is designed to be extensible when relevant events can be monitored and abstracted into meaningful triggers.
- Checks: Predicates are evaluated when triggers occur, and enforcement is applied only when all rule predicates evaluate to true.Predicate inputs depend on the trigger type, including state for state-change events and state plus action for action events.
- Enforcements: Available enforcements include stopping, user inspection, predefined actions, LLM self-examination, and corrective interventions that transform the agent trajectory.These mechanisms can terminate execution, request approval, or update behavior after a violation.
- Semantics: AgentSpec continuously re-evaluates the environment and trajectory after interventions or actions until the agent remains aligned with the intended outcomes.Formally, violated rules apply enforcement functions to update the current trajectory before execution proceeds.
4 AgentSpec Implementation
AgentSpec integrates runtime rule enforcement into LangChain’s iterative agent loop by intercepting execution points and evaluating constraints before actions occur. The framework also supports user- or LLM-generated predicates and adaptation to other agent frameworks.
- LangChain integration: AgentSpec intercepts LangChain’s iter_next_step loop to evaluate predefined constraints before an action executes.The loop repeatedly plans, executes, observes results, and continues until task completion.
- Rule processing: Rules are defined in the AgentSpec DSL and parsed with ANTLR4 before enforcement is integrated into the agent workflow.This separates rule specification from the execution mechanism.
- Predicate generation: Predicates may be supplied by users or generated automatically by an LLM for domain-specific and open-ended safety requirements.LLM generation produces predicate functions that inspect user input and the agent’s current trajectory for potential risks.
- Execution interface: A flexible execution interface requires predicate implementations and, when needed, enforcement actions from users adopting the framework.The interface standardizes rule enforcement for LangChain-based agents, including domains beyond those evaluated.
- Framework portability: AgentSpec’s enforcement principles can be adapted to other multi-step agent frameworks by instrumenting analogous action-handling components.AutoGen is given as an example of a framework that can support this adaptation.
5 Evaluation
AgentSpec enforces runtime safety constraints across code, embodied-agent, and autonomous-driving tasks. Its evaluations show strong hazard prevention, while LLM-generated rules generalize with measurable gaps and runtime costs remain lightweight.
- Code agents: AgentSpec correctly identified most unsafe code-agent actions, but some flagged cases required human interpretation of whether the generated code was genuinely harmful.Unsafe deserialization was judged harmless and allowed, while ‘Miss default case’ and ‘Duplicate key in the associative list’ produced no violations.
- Embodied agents: 100% of unsafe embodied-agent categories avoided hazardous instructions with AgentSpec enabled, while successful safe-task execution declined from 58.62% to 54.26%.The enforcement rule blocks actions predicted to cause damage or harm while preserving functionality on benign instructions.
- LLM-generated rules: 87.26% of risky code was enforced by LLM-generated rules, but failures resulted from overfitting to explicitly demonstrated files, directories, or unsafe patterns.The generated rules detected listed examples more reliably than broader classes of similar risks.
- LLM-generated rules: LLM-generated embodied-agent rules achieved 95.56% precision and 70.96% recall, with false negatives involving omitted action checks and complex object properties.The rules could also become overly rigid when requirements were vague.
- Runtime overhead: AgentSpec adds millisecond-scale overhead: parsing averages 1.42 milliseconds, predicate evaluation 2.83 milliseconds for code and 1.11 milliseconds for embodied agents.Enforcement latency varies by mechanism, user response, invoked action, or LLM response time.
6 Discussion
AgentSpec provides declarative, semantically expressive runtime enforcement that can monitor diverse agent actions and apply predefined interventions. Its current checkpoint-based design is reliable and interpretable but does not assess long-term consequences of action sequences.
- Comparison: AgentSpec differs from dialogue-level and syntactic safeguards by intervening at execution-critical junctures and targeting semantic properties.This enables finer-grained intervention immediately before high-impact or potentially unsafe actions.
- Expressiveness: AgentSpec enforces semantic constraints on tool invocations and logical predicates, supporting privacy, safety, and reliability policies.Examples include monitoring email access, file manipulation, and API usage, with interventions such as halting execution.
- Reliability: Declarative rules externalized from the LLM promote consistent behavior across runs, environments, and model versions while enabling inspection and auditing.The llm_self_examine strategy can also support recovery from violations through agent reflection.
- Limitations and Future Work: AgentSpec currently performs deterministic enforcement at discrete checkpoints rather than reasoning about whether multi-step trajectories will reach unsafe future states.The paper proposes probabilistic reachability analysis as future work for proactive intervention on risky paths.
7 Related Work
AgentSpec is positioned within work on red-teaming, blue-teaming, risk assessment, and runtime verification for LLM agents. It contributes a flexible rule-based framework intended to configure enforcement across these threat and verification settings.
- Red- and Blue-Teaming: Red-teaming research identifies and exploits agent vulnerabilities, including memory poisoning and privacy risks in adversarial environments.Examples include AgentPoison and environmental injection attacks.
- Red- and Blue-Teaming: Blue-teaming approaches defend agents through toxic-prompt detection, autogenerated guardrails, and model editing techniques.The cited defenses include fast toxic prompt detection, GuardAgent, SafeEdit, and layer-specific editing.
- AgentSpec: AgentSpec differs by offering flexible and generalizable rule-based enforcement that can be configured for the identified LLM-agent threats.Its distinction is framed as enforcement rather than solely vulnerability discovery or model modification.
- Risk Assessment: Related risk-assessment benchmarks evaluate agent safety through sandbox simulation, unsafe code execution, and hazard avoidance in embodied agents.The cited resources include ToolEmu, RedCode, and SafeAgentBench.
- Runtime Verification: Runtime-verification research studies enforceable properties and applies runtime verification to security-sensitive systems such as Trusted Execution Environments.AgentSpec extends this general enforcement context to LLM-agent behavior.
8 Conclusion
The paper introduces AgentSpec, a DSL for enforcing customizable runtime constraints on LLM agents. Evaluations across diverse domains report safety improvements with minimal runtime overhead, while LLMs can generate rules with high accuracy.
- Contribution: AgentSpec combines structured rule definitions with flexible enforcement mechanisms to provide customizable runtime constraints for LLM agents.The framework is designed to support safety and reliability across diverse domains.
- Findings: Empirical evaluations report prevention of unsafe code executions, avoidance of hazardous embodied-agent actions, and lawful autonomous-driving decisions.The conclusion also reports that rules can be generated manually or automatically.
- Findings: LLMs achieve high accuracy in specifying and enforcing safety conditions through automatically generated rules.This result is presented alongside the manual-rule evaluation across the paper’s application domains.