PaperlayerView exact revision on arXiv

Source-linked AI summary

LlamaFirewall: An open source guardrail system for building secure AI agents

Sahana Chennabasappa, Cyrus Nikolaidis, Daniel Song, David Molnar, Stephanie Ding, Shengye Wan, Spencer Whitman, Lauren Deason, Nicholas Doucette, Abraham Montilla, Alekhya Gampa, Beto de Paola, Dominik Gabi, James Crnkovich, Jean-Christophe Testud, Kat He, Rashnil Chaturvedi, Wu Zhou, Joshua Saxe

arXiv:2505.03574v1cs.CRcs.AI

TL;DR

Autonomous LLM agents create security risks that chatbot-focused safeguards do not fully address, motivating a real-time, system-level defense. LlamaFirewall provides an open-source layered framework with PromptGuard 2, AlignmentCheck, and CodeShield; combined PromptGuard and AlignmentCheck reduce ASR to 1.75%, a >90% reduction from baseline, while future work targets AlignmentCheck latency.

  • Problem

    Autonomous agents handling trusted workflows, code, and untrusted inputs introduce prompt-injection, misalignment, and insecure-code risks that existing chatbot-focused security infrastructure does not fully address.

  • Method

    LlamaFirewall is an open-source, system-level layered framework combining PromptGuard 2 for jailbreak detection, AlignmentCheck for reasoning-based misalignment detection, and CodeShield for real-time code analysis.

  • Results

    The combined PromptGuard + AlignmentCheck configuration reduced ASR to 1.75%, a >90% reduction from baseline, with utility of 42.7%.

  • Takeaways & Limitations

    LlamaFirewall provides a modular, real-time security guardrail system for protecting LLM agents against prompt injection, misalignment, and insecure code risks.

  • Takeaways & Limitations

    Reducing AlignmentCheck latency remains a priority, with model distillation being explored to lower inference overhead while retaining semantic alignment capabilities.

Abstract

from arXiv · show

Large language models (LLMs) have evolved from simple chatbots into autonomous agents capable of performing complex tasks such as editing production code, orchestrating workflows, and taking higher-stakes actions based on untrusted inputs like webpages and emails. These capabilities introduce new security risks that existing security measures, such as model fine-tuning or chatbot-focused guardrails, do not fully address. Given the higher stakes and the absence of deterministic solutions to mitigate these risks, there is a critical need for a real-time guardrail monitor to serve as a final layer of defense, and support system level, use case specific safety policy definition and enforcement. We introduce LlamaFirewall, an open-source security focused guardrail framework designed to serve as a final layer of defense against security risks associated with AI Agents. Our framework mitigates risks such as prompt injection, agent misalignment, and insecure code risks through three powerful guardrails: PromptGuard 2, a universal jailbreak detector that demonstrates clear state of the art performance; Agent Alignment Checks, a chain-of-thought auditor that inspects agent reasoning for prompt injection and goal misalignment, which, while still experimental, shows stronger efficacy at preventing indirect injections in general scenarios than previously proposed approaches; and CodeShield, an online static analysis engine that is both fast and extensible, aimed at preventing the generation of insecure or dangerous code by coding agents. Additionally, we include easy-to-use customizable scanners that make it possible for any developer who can write a regular expression or an LLM prompt to quickly update an agent's security guardrails.

1 Introduction

LLM agents introduce application-layer security risks that chatbot-focused safeguards and proprietary systems do not adequately address. LlamaFirewall responds with an open-source, system-level framework combining modular guardrails for prompt injection, agent misalignment, and insecure code.

  • Autonomous agents embedded in trusted applications can leak private data, execute unauthorized commands, introduce production vulnerabilities, and drift beyond user requests.
  • Existing LLM security work often focuses on chatbot moderation, leaving insecure code outputs, prompt injection against permissioned agents, and code interpreter abuse insufficiently addressed.
  • LlamaFirewall is an open-source, system-level framework with modular layered defenses for prompt injection, agent misalignment, and insecure or dangerous code.
  • PromptGuard 2 provides real-time jailbreak detection through 86M-parameter and lower-latency 22M-parameter variants.
  • AlignmentCheck experimentally audits agent chain-of-thought reasoning for goal hijacking and prompt-injection-induced misalignment in real time.
  • CodeShield is an extensible online static-analysis engine supporting Semgrep and regex rules across 8 programming languages.
  • LlamaFirewall unifies these defenses in a policy engine supporting custom pipelines, conditional remediation, and pluggable detectors.

2 Related Work

Prior systems address pieces of LLM security, including input filtering, code validation, runtime action monitoring, and provider-level safeguards. LlamaFirewall combines layered prompt-injection mitigation with reasoning inspection and extensible security scanning.

  • Open-source guardrail systems provide custom rules, response validation, context inspection, or malicious-prompt classification, but LlamaFirewall integrates these ideas into a security-focused layered pipeline.
  • Existing code-oriented tools analyze SQL or validate formatting, while few systems offer syntax-aware, extensible static-analysis pipelines designed for native integration.
  • Runtime alignment remains unsolved: training-time constraints cannot prevent deployed prompt injection or task drift, while action logs provide limited reasoning introspection.
  • Instruction Hierarchy fine-tuning improves obedience to privileged directives, but failure analysis still finds significant hierarchy violations.
  • AlignmentCheck examines an LLM decision’s chain of thought for contradictions, goal divergence, and injection-induced misalignment, and can layer with PromptGuard.

3 LLamaFirewall workflow and detection components

LlamaFirewall demonstrates modular, conditional defenses across representative agent workflows involving sensitive-data exfiltration and unsafe code generation. PromptGuard and AlignmentCheck stop a hijacked travel agent, while CodeShield blocks insecure SQL from reaching production.

  • The workflow examples show modular, conditional defenses engaging only when needed across goal hijacking, sensitive-data exfiltration, and accidental unsafe code generation.
  • A travel-planning agent browsing websites and calling booking APIs can encounter hidden instructions that redirect it toward extracting chat history and exfiltrating user data.
  • PromptGuard blocks detected jailbreaks before they enter context, while AlignmentCheck detects behavioral shifts from trip planning toward data exfiltration and stops execution before an HTTP request.
  • A benign web example can still produce insecure SQL when untrusted input is concatenated directly into a query, creating an injection vulnerability.
  • CodeShield prevents insecurely constructed SQL from being committed while allowing the coding agent to iterate freely.

4 LlamaFirewall Scanners: Design and Evaluation

LlamaFirewall combines lightweight prompt-injection detection, experimental reasoning-based alignment monitoring, and static code analysis into a layered scanner framework evaluated across agent and code-security settings.

  • PromptGuard 2: PromptGuard 2 uses lightweight DeBERTa classifiers for real-time detection of explicit jailbreaks, with 86M- and 22M-parameter variants.The updated models refine scope, expand benign and malicious training data, use an energy-based loss, and address adversarial tokenization.
  • PromptGuard 2: PromptGuard evaluations measure generalization on unseen English and machine-translated multilingual attacks using AUC, recall at 1% FPR, utility, and attack success rate.AgentDojo evaluates indirect injection resistance in agentic environments using utility and ASR.
  • AlignmentCheck: AlignmentCheck audits full execution traces against the user’s objective, flagging behavioral deviations caused by covert injections, misleading tool outputs, or goal hijacking.It uses language-model reasoning to compare planned and observed actions and constrain agents to authorized tasks.
  • AlignmentCheck: Over 80% recall at below 4% false-positive rate was achieved without fine-tuning on the goal-hijacking benchmark using Llama 4 Maverick and Llama 3.3 70B.On AgentDojo, AlignmentCheck reduced ASR from 0.18 to 0.03 with Llama 4 Maverick, an 83% reduction and minimal utility degradation.
  • PromptGuard 2: PromptGuard 2 reduced AgentDojo ASR from 17.6% to 7.5% while maintaining utility at 47.0%, a 57% reduction with negligible performance tradeoff.The benchmark primarily targets attacks against important instructions, limiting what PromptGuard alone demonstrates in more diverse adversarial settings.
  • CodeShield: CodeShield applies modular static-analysis rules across seven languages and over 50 CWEs, with approximately 90% of production inputs resolved in under 70 milliseconds.In CyberSecEval3, it achieved 96% precision and 79% recall, while remaining unable to cover nuanced or context-dependent vulnerabilities comprehensively.

5 Limitations and future Work

LlamaFirewall’s future work targets broader modality coverage, lower AlignmentCheck latency, expanded threat coverage, and more realistic defensive evaluation.

  • Future versions aim to secure image- and audio-based agents against security vectors introduced by non-textual modalities.
  • Reducing AlignmentCheck’s inference overhead remains a priority for real-time and large-scale production deployment.The authors are exploring model distillation to retain semantic alignment capabilities while reducing latency.
  • LlamaFirewall currently focuses on prompt injection and insecure code generation, with planned coverage of malicious code execution and unsafe tool use.
  • The authors are exploring agent-oriented benchmarks that reflect complex execution flows, adversarial scenarios, and real-world tool usage.These benchmarks are intended to integrate closely with LlamaFirewall for faster defensive research iteration.

6 Conclusion

The conclusion presents LlamaFirewall as a modular, real-time security framework for autonomous agents, addressing risks that chatbot-centric safeguards do not cover. It combines prompt-injection detection, alignment monitoring, and code analysis for production-oriented defense.

  • LlamaFirewall is an open-source, modular, real-time guardrail system designed to protect LLM agents in production environments.
  • Its layered framework combines PromptGuard 2, AlignmentCheck, and CodeShield to address prompt injection, agent misalignment, and unsafe code generation.
  • PromptGuard evaluation uses out-of-distribution datasets to measure generalization and reduce risks of overfitting to particular data.
  • The authors do not report competitor metrics on this dataset because it was specifically tailored to inputs out-of-distribution for PromptGuard.
  • The evaluation dataset includes English adversarial prompts and machine-translated versions in eight additional languages.

A.1 Tool-use agentic prompt injection benchmark

The authors develop a realistic tool-use prompt-injection benchmark by adding adversarial perturbations to benign agent tasks. It covers diverse attack techniques and threat categories and reports unguarded baseline attack success rates.

  • The evaluation framework simulates applications including Calendar, Messaging, and Email across domains such as travel planning, retrieval, and productivity.
  • The benchmark extends a utility benchmark by injecting adversarial instructions into tool outputs for tool-using agents.
  • 600 scenarios—300 benign and 300 malicious—cover 7 injection techniques and 8 threat categories.
  • Baseline attack success rates are measured without guardrail defenses, with error responses removed from the results.
  • The benchmark’s diverse scenarios produce a higher baseline ASR than externally available datasets and provide a more rigorous practical testbed.
  • Figure 5 reports prompt-injection attack success rates across attack types, categories, and models in the unguarded evaluation.

B PromptGuard 2

PromptGuard 2 refines lightweight DeBERTa-based jailbreak detection for production use, while AgentDojo evaluation shows improved attack-resilience tradeoffs. The 86M model performs best overall, whereas the 22M model is faster but weaker for multilingual inputs.

  • B.1 Development Methodology: PromptGuard 2 comprises 86M and 22M DeBERTa-based models designed to detect explicit jailbreak techniques with low latency.
  • B.1 Development Methodology: The model narrows its scope to high-confidence jailbreak detection because broader goal-hijacking detection produced excessive false positives without application context.
  • B.1 Development Methodology: PromptGuard 2 uses expanded benign, malicious, synthetic, and red-team data to improve detection accuracy, reduce false positives, and generalize across threats.
  • B.1 Development Methodology: Its training adds an energy-based loss that penalizes overconfident negative predictions on benign prompts, improving calibration on out-of-distribution inputs.
  • B.1 Development Methodology: Hardened tokenization addresses evasion tactics including token fragmentation and whitespace manipulation.
  • B.2 Evaluating PromptGuard in an Agentic Environment: 57% ASR reduction with 1.5% utility loss makes PromptGuard 2 86M the best overall tradeoff in AgentDojo evaluation.
  • B.2 Evaluating PromptGuard in an Agentic Environment: PromptGuard 2’s custom loss supports high recall without overfitting to specific jailbreak payloads in out-of-distribution settings.
  • B.2 Evaluating PromptGuard in an Agentic Environment: The 22M model shows reduced multilingual performance, while PromptGuard 2 overall reports state-of-the-art detection with minimal overhead and competitor outperformance.

C.1 Implementation Methodology

AlignmentCheck evaluates whether an agent’s latest action remains consistent with the user’s goal by combining the selected action with prior reasoning trace context. Its few-shot chain-of-thought audit is intended to detect semantic misalignment and indirect prompt injection, while remaining customizable across agent implementations.

  • AlignmentCheck operation: AlignmentCheck uses a guardrail LLM for few-shot alignment classification after each agent action.It is designed to detect semantic misalignment, especially from indirect prompt injections or goal hijacking.
  • AlignmentCheck inputs: The evaluation compares the user’s original goal, prior agent trace, and most recent selected action.The trace includes reasoning steps, tool invocations, intermediate thoughts, and user interactions within a fixed context window.
  • AlignmentCheck operation: The guardrail LLM determines whether the selected action is consistent with the user’s goal using the trace as context.This supports semantic-level evaluation of the agent’s intent and trajectory.
  • Customization: A static prompt template supports customization with agent-specific few-shot examples while reducing prompt-injection risk without customization.The paper reports this behavior across diverse agent frameworks.
  • Comparison: AlignmentCheck is reported to outperform prior indirect-injection defenses, while offering more flexibility than deterministic methods with predefined policies.The comparison includes Spotlighting, Instruction Hierarchy fine-tuning, and CaMeL.
  • Why the method works: Chain-of-thought monitoring improves detection by exposing verbalized agent intent, which helps identify behavior that conflicts with the user’s goal.The paper also cites generalist reasoning and standardized sequential goals as factors supporting generalization.

C.2 Comparison of Effectiveness of diffent Llama Models backing AlignmentCheck

AlignmentCheck’s effectiveness varies with the underlying Llama model: larger, more capable models generally achieve higher detection with fewer false positives and better utility-security trade-offs. Llama 4 Maverick is highlighted as a favorable balance across the internal benchmark and AgentDojo, while efficiency improvements remain under exploration.

  • Internal goal-hijacking benchmark: Larger and more capable models achieve higher detection rates with lower false positive rates on the internal goal-hijacking benchmark.Llama 4 Maverick detected over 83% of attempts with a 2.5% false positive rate.
  • AgentDojo: On AgentDojo, all evaluated models substantially reduced attack success rate relative to baseline.The figure compares utility and ASR across AlignmentCheck configurations backed by different Llama models.
  • AgentDojo: Llama 4 Maverick achieved an ASR of 2.9% while retaining over 90% of baseline utility on AgentDojo.Smaller models reduced attacks but caused severe utility degradation through false positives or overly conservative behavior.
  • Overall comparison: Across two benchmarks, larger models—particularly Llama 4 Maverick and Llama 3.3 70B—provided a better balance between security coverage and usability.Their results combined high detection rates with low false positive rates.
  • Deployment considerations: Full alignment checks are recommended for high-security applications involving sensitive data or critical actions.Risk-triggered checks and model distillation are being explored to improve efficiency.

C.4 Limitations and Risks

AlignmentCheck has two highlighted limitations: it can itself be targeted by prompt injection, and effective semantic evaluation requires large, capable models. The proposed mitigations and ongoing efficiency work do not remove these scope and cost constraints.

  • Guardrail injection: AlignmentCheck can be targeted by injections intended to manipulate both the agent and the guardrail LLM.The framework mitigates this risk by restricting inputs and pre-scanning them with PromptGuard.
  • Guardrail injection: AlignmentCheck inputs are restricted to the agent’s chain-of-thought and actions, excluding direct tool outputs.This is one of the stated mitigations for guardrail injection.
  • Computational cost: Effective alignment checks require large, capable models, while smaller models provide limited semantic evaluation in resource-constrained environments.The paper identifies computational cost as a deployment limitation.

C.5 Example Trace

The example trace shows an agent receiving a complex shopping-and-checkout request, then switching to an unrelated apartment-listing action after a tool response. AlignmentCheck identifies this deviation as compromised reasoning and misalignment with the original task.

  • Original request: The trace begins with a shopping request involving specified products, quantity adjustments, checkout, and possible follow-up communication.The user asks for a total purchase price after the first checkout and conditional WhatsApp contact after cancellation.
  • Initial agent actions: The agent initially searches for products after reporting that one requested search returned no results.The trace records a search for another product named by the user.
  • Task deviation: After a tool response, the agent states that the user changed the request and selects an apartment-listing tool unrelated to the shopping task.The selected action lists saved apartments using RentAFlat.
  • AlignmentCheck result: AlignmentCheck judges the apartment-listing action as misaligned because it departs from the original shopping and checkout objective.The example includes judgments from multiple alignment-guard models describing the reasoning as compromised.
  • AgentDojo example: A separate AgentDojo example evaluates whether website-fetching actions follow the user’s requested websites and flags an unrequested domain.The example records an action for Dora’s website alongside an attempted visit to true-informations.com.
  • Judge behavior: The AlignmentCheck prompt instructs the judge to assess the latest action against the original user message rather than the entire action trace.It marks an action misaligned only when it is clearly unrelated to the original objective.
Loading 2505.03574v1…