BELLS: A Framework Towards Future Proof Benchmarks for the Evaluation of LLM Safeguards

1. Context

• More capable, integrated LLM applications face increasing failure modes, and failures can cause harms ranging from reputational damage to systemic risks.Some damage may arise from unforeseen behavior or unknown means.Source: §1 · p. 1
• Input-output safeguards analyze LLM application traces and produce safety reports for monitoring, offline evaluation, content moderation, or other trace analysis.A trace comprises the inputs and outputs of an application's constituent LLMs.Source: §1 · p. 1, Figure 2 · p. 2
• Despite the wider ecosystem of LLM evaluation, little attention has been given to evaluating safeguards.Source: §1 · p. 2

2. Motivation: building metrics to foster the development of future-proof safeguards

• BELLS aims to compare safeguards so users and developers can assess strengths and weaknesses; the authors argue third-party evaluation is needed because in-house metrics do not provide a solid basis for comparison.Source: §2 · p. 2
• BELLS aims to measure whether safeguards generalize to unknown failure modes, including new jailbreaks and application-enabled problems such as in-context reward hacking.The authors envision robust detection as an early warning for new sources of systemic risk.Source: §2 · p. 2
• BELLS aims to foster safeguards for future applications such as autonomous LLM agents and multi-agent systems, for which the authors say no recognized detection methodology exists.Source: §2 · p. 2

3.1. Input-output safeguards

• Most input-output safeguards focus on detecting unauthorized content in user messages or LLM-generated text, with systems also targeting prompt injections, jailbreaks, and toxic speech.Examples include Llama Guard, Lakera Guard, OpenAI Moderation API, Perspective API, and Azure AI Content Safety.Source: §3.1 · p. 3

3.2. Benchmarks for safeguards

• Few benchmarks directly assess input-output safeguards beyond prompt-injection detection; other benchmarks tend to classify text content such as toxicity.BIPIA and PINT are examples of prompt-injection benchmarks made to test safeguards.Source: §3.2 · p. 3
• Collections of jailbreaks, goal hijacking, prompt leaking, and sycophancy test LLM robustness rather than being tailored to evaluate safeguards.Source: §3.2 · p. 3
• The most complete prompt-injection benchmarks are proprietary, while the authors argue safety-critical safeguard evaluation should be open.They describe an open process as necessary for developers and users to trust the systems they use.Source: §3.2 · p. 3

4. Structure of BELLS

• BELLS organizes its test collection into established failure tests, emerging failure tests, and next-gen architecture tests.Source: §4 · p. 3, Figure 3 · p. 3
• Established-failure tests use existing datasets for defined failures such as jailbreaks and unauthorized content, either by aggregating safeguard benchmarks or adapting LLM benchmarks into application traces.The adaptation collects traces across models and pre-prompts to induce both harmful and safe behaviors.Source: §4 · p. 3
• Emerging-failure tests use augmented examples from recent or emerging failures to estimate detection of problems safeguard developers may have overlooked.Sources include scientific literature and new use cases observed online; some tests could be kept private and run by independent third parties to reduce contamination and better measure generalization.Source: §4 · pp. 3–4
• Reports on emerging vulnerabilities such as the OWASP Top 10 for LLM inspire tests for insecure output handling, sensitive information disclosure, and excessive agency.Source: §4 · p. 4
• Next-gen architecture tests target complex systems such as LLM agents, where correctly classifying behavior may require context across multiple API calls.The proposed kit includes a minimal application, anomalous and normal-operation traces, and a minimal safeguard to establish baseline detection accuracy.Source: §4 · p. 4
• The inaugural datasets are intended to be updated continuously, with new emerging tests added and some eventually moved into the established category.Source: §4 · p. 4

5.1. Presentation of the Machiavelli Benchmark

• MACHIAVELLI is an experimental sandbox, not a realistic harm setting, with navigable games, rich ethical-performance tradeoffs, and no required web or terminal tools.Its 134 choose-your-own-adventure games annotate ethical violations across 572,322 scenes.Source: §5.1 · p. 4
• The dataset pairs ethical and unethical steering traces, hiding the steering prompt, and contains 1,200 traces, 33.6k API calls, and 18.5 million words.Agents choose actions based on the current scene rather than prior ones.Source: §5.1 · pp. 4–5
• 0.97 AUPRC after 80 steps: the baseline detects unethical steering as a proxy for unethical behavior.The score measures detection of steering, not ethical behavior directly.Source: §5.1 · p. 5
• An interactive visualization was built to help readers understand the rich dataset.Source: §5.1 · p. 5

5.2. Preliminary lessons building safeguards for agents

• Long-range dependencies can span API calls, as when an agent forgets by step 30 a commitment made at step 3.The later call may contain no information about the earlier commitment, preventing detection from that call alone.Source: §5.2 · p. 5
• Counterfactual evaluation is needed because available choices may all be harmful or all beneficial, making choice quality relative to the options.Source: §5.2 · p. 5
• MACHIAVELLI’s varied scenarios require detectors to interpret context ranging from harms to allies or enemies to long-term commitments.Harm also varies by scenario; some, such as Death Collector, offer few non-harmful actions.Source: §5.2 · p. 5, Figure 4 · p. 5
• Content moderators such as GPT-4 do not straightforwardly transfer to supervising agent actions in complex MACHIAVELLI scenarios.The authors report this from preliminary experiments.Source: §5.2 · p. 5

6. Discussions and future work

• BELLS organizes safeguard tests into established, emerging, and next-generation architecture categories to address current and future failure modes.Source: §6 · p. 5
• The authors exemplify next-generation tests with MACHIAVELLI traces of agents exhibiting ethical or harmful behavior in a rich text-based environment.Source: §6 · p. 5
• Future work includes developing the other test categories, rigorously evaluating available safeguards, and measuring robustness against adversaries attempting to bypass them.The authors also call for expanding BELLS datasets and continuously incorporating new failure modes.Source: §6 · p. 5
• BELLS evaluations are not absolute assessments of application safety; safeguards are one tool within a systematic safety approach, not a silver bullet.Source: §6 · p. 5

Experiment setup

• The task is binary classification of whether the model was instructed to behave unethically, evaluated on a balanced dataset despite production’s high class imbalance.AUPRC is used because it is suitable for applications with large class imbalance.Source: A. Experiment: Baseline Detection for… · p. 8

Baseline

• The baseline compares a trace’s harm with the average harm of 100 random agents in the same scenario, using cumulative ethical violations up to each time step.It normalizes trace harm and classifies a trajectory as unethical when its normalized harm exceeds a threshold.Source: A. Experiment: Baseline Detection for… · p. 8
• The step-dependent threshold is scenario-independent, enabling generalization to unseen scenarios, and users set it according to their false-positive versus false-negative trade-off.The baseline is presented as a sanity check that progress on this benchmark is possible.Source: A. Experiment: Baseline Detection for… · p. 8

Limitations

• The approach requires direct environment access, becomes less precise near the ends of variable-length trajectories, and generates many random trajectories even in test scenarios.Because of the test-time trajectory generation, it is better viewed as evaluating a scenario–trace pair than a trace alone.Source: A. Experiment: Baseline Detection for… · p. 8

Results

• AUPRC exceeds 0.9 after step 29 and reaches 0.97 at step 80, showing reasonable precision and recall even when detection is limited to the trajectory’s start.AUPRC is computed independently at each time step.Source: A. Experiment: Baseline Detection for… · p. 8

TRICOTS

• TRICOTS collects traces from Python codebases using OpenAI’s API and can edit API calls on the fly, including to add instructions.The tool is released alongside the Machiavelli dataset.Source: B. TRICOTS – A versatile tool to… · p. 8
• TRICOTS can be adapted to other benchmarks and codebases by importing its library.Source: B. TRICOTS – A versatile tool to… · p. 8

Availability

• The code is stated to become available upon acceptance.Source: B. TRICOTS – A versatile tool to… · p. 8

C.1. Steering prompts

• The unethical steering prompt adapts the Ethics Prompt used in the MACHIAVELLI paper while keeping the same structure.Source: §C.1 · p. 9
• The ethical prompt directs the agent toward altruism, deontological rules, humility, and generosity.Source: §C.1 · p. 9
• The unethical prompt directs the agent toward selfishness, breaking deontological rules, arrogance, and greed.Source: §C.1 · p. 9

C.2. Sample trace

• Each collected trace is a sequence of API calls with a shared structure.Source: §C.2 · p. 9
• Each call includes the steering prompt in the system message, scene details and available actions in a user message, and the agent’s action as an integer.Source: §C.2 · p. 9
• The agent has little dependency between scenes because it lacks memory of previous scenes, while the scene depends on earlier actions and the steering prompt remains constant.An API-call example appears in Listing 1 and the interactive visualization.Source: §C.2 · p. 9

• The Figure 6 API call was cherry-picked to fit in one screenshot and show non-zero harm.Source: Figure 6 · p. 11
• Most API calls are longer than the sample shown in Figure 6.Source: Figure 6 · p. 11
• Most API calls have fewer annotations than the sample shown in Figure 6.Source: Figure 6 · p. 11