Source-linked AI summary
NeMo Guardrails: A Toolkit for Controllable and Safe LLM Applications with Programmable Rails
Traian Rebedea, Razvan Dinu, Makesh Sreedhar, Christopher Parisien, Jonathan Cohen
TL;DR
LLM applications can drift off-topic, hallucinate, or succumb to jailbreaks, motivating runtime controls. NeMo Guardrails adds model-independent programmable rails through a Colang-interpreting dialogue-manager proxy. Initial evaluations report strong moderation and improved handling of fact-checking and hallucination cases, while the approach has important safety and efficiency limits.
Problem
LLMs can veer off-topic, generate fabricated facts, and remain vulnerable to prompt-injection attacks, creating challenges for controllable and trustworthy conversational systems.
Method
NeMo Guardrails uses a runtime dialogue manager and Colang to define interpretable, developer-programmed rails that operate across LLMs.
Results
Using both input and output moderation rails blocks close to 99% of harmful prompts while blocking 2% of helpful requests on gpt-3.5-turbo; the hallucination rail boosts performance up to 95%.
Takeaways & Limitations
Programmable rails supplement embedded model rails and support controllable, safe LLM applications across dialogue, fact-checking, hallucination, and moderation tasks.
Takeaways & Limitations
The authors recommend against using programmable rails as a stand-alone safety solution and report that the current sequential prompting design costs about 3 times a normal call.
Abstract
from arXiv · showhide
NeMo Guardrails is an open-source toolkit for easily adding programmable guardrails to LLM-based conversational systems. Guardrails (or rails for short) are a specific way of controlling the output of an LLM, such as not talking about topics considered harmful, following a predefined dialogue path, using a particular language style, and more. There are several mechanisms that allow LLM providers and developers to add guardrails that are embedded into a specific model at training, e.g. using model alignment. Differently, using a runtime inspired from dialogue management, NeMo Guardrails allows developers to add programmable rails to LLM applications - these are user-defined, independent of the underlying LLM, and interpretable. Our initial results show that the proposed approach can be used with several LLM providers to develop controllable and safe LLM applications using programmable rails.
1 Introduction
LLMs need runtime controls for steerability, factuality, topic adherence, and jailbreak resistance. NeMo Guardrails provides user-defined, interpretable rails that operate independently of the underlying model.
- LLMs can veer off-topic, hallucinate facts, and be manipulated through prompt-injection or jailbreak attacks.
- NeMo Guardrails is an open-source toolkit for adding programmable rails to LLM-based applications.
- Rails constrain outputs through harmful-topic restrictions, predefined dialogue paths, specific responses, language styles, or structured-data extraction.
- Unlike training-time alignment or prompt-tuned embedded rails, NeMo Guardrails defines custom rails at runtime independently of alignment strategies and across different LLMs.
- The toolkit uses a dialogue-manager-like runtime and Colang, an interpretable custom modeling language for defining rails.
- It supports several LLM providers, starter applications, evaluation tools, and Apache 2.0 licensing.
2 Related Work
Prior approaches embed general or application-specific behavior through alignment, prompt tuning, or prompting. NeMo Guardrails instead combines an LLM-based end-to-end dialogue approach with a dialogue-management runtime that interprets Colang flows.
- Model alignment embeds rails during training but requires large collections of manually labeled prompts and responses.
- Embedded alignment rails cannot easily be changed at runtime, while prompt tuning can add application-specific embedded rails.
- Prompt engineering and in-context learning add user-defined runtime behavior by specifying desired model behavior in surrounding text.
- Chain-of-thought prompting extends runtime control through examples, multi-step prompts, or dialogue-like role interactions.
- Traditional task-oriented agents use NLU and dialogue-management engines with finite intents, states, and response sets, but require substantial human design and updating.
- NeMo Guardrails combines end-to-end LLM dialogue generation with a dialogue-management-like runtime that maintains Colang flow state.
3 NeMo Guardrails
NeMo Guardrails places a Colang-interpreting runtime between users and an LLM to enforce programmable dialogue and safety rules. Its topical and execution rails guide flows, invoke actions, and check factuality, hallucination, and harmful content.
- The Guardrails runtime acts as a proxy, interpreting Colang rules that guide LLM behavior and enforce developer-defined or automatically generated rails.
- Colang scripts define user canonical forms, dialogue flows, and bot canonical forms, with indexed definitions supporting nearest-neighbor few-shot retrieval.
- Canonical forms encode message meaning like intents but are LLM-generated and guided rather than restricted to a fixed closed set.
- Topical rails use event-driven dialogue management to keep developer-specified flows active in the current conversation context.
- The topical-rail runtime generates a user canonical form, decides and executes the next step, then generates bot messages conditioned on that step.
- Execution rails are Python-defined custom actions that monitor LLM inputs or outputs and support fact-checking, hallucination, and moderation.
- The fact-checking rail frames retrieval-grounded response checking as binary entailment prediction using evidence and a generated response.
- The hallucination rail samples multiple answers and checks their agreement, while moderation screens inputs before the dialogue system and outputs before delivery.
4 Sample Guardrails Applications
NeMo Guardrails supports simple Colang-defined conversation controls and more complex execution rails implemented as custom actions. Examples connect topical flows to external tools and allow jailbreak decisions to control returned responses.
- Adding rails to conversation applications requires only simple Colang scripts.
- Topical rails can combine with execution rails to call specific actions or define complex dialogue flows for task-oriented agents.
- The example uses two topical rails and a Python custom action to invoke WolframAlpha for math and distance queries.
- Execution rails are added by defining an action, invoking it from Colang with execute, and specifying how its output affects the dialogue flow.
- The toolkit includes action examples for input and output moderation, with additional execution-flow examples in the appendices.
- A jailbreak flow can call check_jailbreak and replace the generated answer with default text when the user message is flagged.
5 Evaluation
The evaluation examines topical, moderation, fact-checking, and hallucination rails across balanced task-specific datasets. Results indicate that programmable rails can guide conversations with smaller open-source models, improve moderation robustness, and intercept unanswerable prompts.
- Topical Rails: The Banking topical-rails evaluation uses 231 test samples spanning 77 intents, with at most three randomly sampled examples per intent.Intents are mapped to canonical forms and simple dialogue flows in a Colang application.
- Topical Rails: Topical rails successfully guide conversations even with smaller open-source models such as falcon-7b-instruct and llama2-13b-chat.The reported results are shown for the top three performing models.
- Topical Rails: Each step in the three-step approach improves performance, while at least k = 3 vector-database samples per canonical form support good performance.The three steps are user canonical form, next step, and bot message.
- Moderation Rails: Using both input and output moderation rails is more robust than using either rail individually.With both rails, gpt-3.5-turbo blocks close to 99% of harmful prompts and 2% of helpful requests.
- Fact-Checking Rail: Both text-davinci-003 and gpt-3.5-turbo obtain 80% overall accuracy on the fact-checking rail.The evaluation combines equally sampled positive and hard-negative context-question-answer triples.
- Hallucination Rail: The hallucination rail intercepts 70% of unanswerable prompts for text-davinci-003 and boosts gpt-3.5-turbo performance up to 95%.For gpt-3.5-turbo, deflection or marking uncertainty already occurs in 65% of cases.
6 Conclusions
NeMo Guardrails enables developers to build controllable and safe LLM applications with programmable rails expressed in Colang. A dialogue-manager runtime acts as a proxy between the application and the LLM to enforce user-defined rules.
- Conclusions: NeMo Guardrails is a toolkit for building controllable and safe LLM-based applications with programmable rails.The toolkit is presented as an open-source contribution.
- Conclusions: Rails are expressed using Colang and can also be implemented as custom actions when they require complex logic.The runtime interprets Colang code and enforces the user-defined rails between the application and the LLM.
7 Limitations
NeMo Guardrails has practical scope and resource limitations: safety rails should generally complement embedded protections, and runtime prompting adds cost and latency. The authors also identify task coverage and model customization as ongoing constraints.
- 7.1 Programmable Rails and Embedded Rails: Safety-specific programmable rails should not generally be used as a stand-alone solution; the authors advocate combining them with embedded rails.The toolkit’s moderation rails provide better jailbreak protection when added to existing safety rails in powerful LLMs such as ChatGPT.
- 7.1 Programmable Rails and Embedded Rails: Customized models for every possible task and topical rail are difficult to develop, motivating NeMo Guardrails for task-oriented agents without extra mechanisms.The authors still plan p-tuned models for better performance on some tasks, including canonical form generation.
- 7.2 Extra Costs and Latency: The three-step chain-of-thought runtime incurs sequential, unbatchable calls, making current latency and cost about 3 times those of normal bot-message generation.The authors are investigating whether one call can generate the user canonical form, next flow steps, and bot message.
- 7.2 Extra Costs and Latency: More complex prompts and few-shot in-context learning add slightly more latency and cost than vanilla bot-message generation.Developers can choose a simpler prompt when those overheads are unacceptable.
8 Broader Impact
NeMo Guardrails provides an open-source runtime and Colang language for defining programmable controls over LLM applications. The toolkit includes examples, evaluation tools, and reference rails, but these examples require customization and testing before production use.
- Broader impact: The toolkit supplements embedded model rails with user-defined programmable rails for controlling LLM applications.Fact-checking rails can support retrieval-based applications and model assessment.
- Availability: Developers can install the toolkit from GitHub or pip, run applications through a CLI or web server, and combine rails with open-source models and LangChain chains.Documentation, installation guidance, and evaluation tooling are also provided.
- Reference applications: The toolkit provides five reference applications covering topical, moderation, fact-checking and hallucination, secure execution, and jailbreak rails.These applications demonstrate how to build different rail types rather than serving as out-of-the-box safety features.
- Deployment boundary: The reference rails should be customized, strengthened, and thoroughly tested before production deployment, especially for safety-related applications.The toolkit includes evaluation tools for topical and execution rails.
- Architecture: NeMo Guardrails acts as a dialogue-manager-like runtime proxy between users and LLMs, interpreting Colang flows that encode events, messages, actions, and context.The runtime uses event-driven processing, while custom actions can access conversation history.
- Colang: Colang models conversational sequences and guardrails through flows that may include messages, events, and branching logic.Its syntax combines natural language and Python, and its concepts include utterances, messages, events, actions, and context.
F Sample Guardrails Flows using Actions
The sample flows show how custom safety actions are inserted into Colang applications. Jailbreak, output-moderation, hallucination, and fact-checking rails modify or supplement bot responses according to their checks.
- Jailbreak rail: The jailbreak flow checks each user message and can replace a flagged response with a default moderation-policy message.The check_jailbreak action verifies the latest user message and removes the generated answer when flagged.
- Output moderation: Output moderation is triggered after a bot message event to assess the generated response.The flow applies output moderation after generation rather than before the dialogue system processes the user message.
- Hallucination rail: The hallucination flow warns the user about a possible incorrect answer without removing the bot message.It targets questions about persons, where the paper notes GPT models are prone to hallucinate.
- Fact-checking rail: The fact-checking flow applies a similar response-checking mechanism to questions about an employment report.The cited passage introduces the employment-report example but does not state the complete resulting action.
- Evaluation: The toolkit provides evaluation tooling and a CLI methodology for assessing topical and execution rails, with experiments designed to be replicated.The evaluation page is regularly updated with new results, including new LLMs.
G.1 Topical Rails
The topical-rails evaluation tests whether canonical forms and dialogue flows can guide conversations across chit-chat and banking datasets. Results indicate successful guidance with several LLMs, while prompting and retrieval-sample choices remain important.
- Evaluation setup: Topical rails evaluate Guardrails’ canonical-form and dialogue-flow mechanism on chit-chat and banking conversational-NLU datasets.The datasets differ in granularity: chit-chat is generic and coarse-grained, while banking is domain-specific and fine-grained.
- Evaluation setup: The experiments use balanced test sets with at most 3 samples per intent and compare multiple LLMs and vector-database sample counts.The reported settings include k = all, 3, and 1 samples per intent, with random seed 42.
- Results: Topical rails can guide conversations successfully even with smaller open-source models such as falcon-7b-instruct and llama2-13b-chat.The paper presents results for the top three performing models in Fig. 5.
- Caveat: Because LLM performance depends heavily on prompting, better prompting might improve the reported results.The paper attributes this possibility to NeMo Guardrails’ complex prompt.
- Results: Each step in the three-step Guardrails approach improves performance, and at least k = 3 indexed samples per user canonical form are important for good performance.The three steps are user canonical form, next step, and bot message.
- Results: Models such as gpt-3.5-turbo may produce varied canonical forms, making similarity matching and its threshold useful instead of exact matching.The similarity threshold becomes an important inference parameter in these cases.
G.2.1 Moderation Rail
The moderation evaluation measures whether input and output rails block harmful prompts while allowing helpful ones. Combining both rails is more robust than using either alone, with strong reported blocking rates for the tested models.
- Evaluation setup: The evaluation uses balanced harmful and helpful sets derived from Anthropic Red-Teaming and Helpful datasets.Red-Teaming prompts are rated by their ability to elicit inappropriate responses, while Helpful prompts are genuine queries.
- Metrics: Moderation performance is measured by the proportions of harmful prompts blocked and helpful prompts allowed.The ideal outcome is 100% blocking of harmful prompts and 100% allowance of helpful prompts.
- Pipeline: The pipeline applies input moderation before dialogue generation and output moderation before returning the response.Only inputs not flagged by the jailbreak rail reach the conversational agent.
- Results: Combining input and output moderation is more robust than using either rail individually.The paper also notes that output-moderation evaluation is subjective and prompts can be modified to reflect the deployer’s beliefs.
- Results: 97% of harmful and 5% of helpful requests were blocked by text-davinci-003 using both rails, while gpt-3.5-turbo blocked close to 99% and 2%, respectively.The evaluation set contained 200 samples split equally between harmful and helpful requests.
G.2.2 Fact-checking Rail
The fact-checking rail is evaluated on balanced grounded and hard-negative answer triples, with both tested models reaching 80% overall accuracy. Their strengths differ by example type: gpt-3.5-turbo detects negatives better, while text-davinci-003 performs better on positives.
- The evaluation uses MSMARCO context-question-answer triples to assess whether answers are grounded in their evidence.
- Hard negatives are created by rewriting positive answers so they remain similar but are not grounded in the evidence.
- 80% overall accuracy was achieved by both text-davinci-003 and gpt-3.5-turbo on the fact-checking rail.
- gpt-3.5-turbo is better at discovering negatives, whereas text-davinci-003 performs better on positive samples.