Source-linked AI summary
NESSiE: The Necessary Safety Benchmark -- Identifying Errors that should not Exist
Johannes Bertram, Jonas Geiping
TL;DR
NESSiE tests whether LLMs can reliably follow simple safety and helpfulness instructions in settings relevant to autonomous agents. The benchmark uses lightweight keyword-based evaluation across paired test cases and robustness conditions. State-of-the-art models remain below perfect performance, generally favor helpfulness over safety, and become fragile under distraction or altered reasoning. The authors therefore present NESSiE as a necessary minimum bar, not a complete guarantee of safety.
Problem
Agentic systems can operate in safety-critical environments without output monitoring, making basic LLM instruction-following robustness necessary.
Method
NESSiE evaluates paired safety and helpfulness instructions across abstract test cases using keyword matching, repeated runs, and distraction or disabled-reasoning conditions.
Results
State-of-the-art models fail to reach 100% on simple tasks, with helpfulness consistently higher than safety and distraction reducing SH performance by at least 15%.
Takeaways & Limitations
NESSiE is proposed as a minimum passing bar for deployment decisions: models failing basic rules cannot be trusted in complex applications.
Abstract
from arXiv · showhide
We introduce NESSiE, the NEceSsary SafEty benchmark for large language models (LLMs). With minimal test cases of information and access security, NESSiE reveals safety-relevant failures that should not exist, given the low complexity of the tasks. NESSiE is intended as a lightweight, easy-to-use sanity check for language model safety and, as such, is not sufficient for guaranteeing safety in general -- but we argue that passing this test is necessary for any deployment. However, even state-of-the-art LLMs do not reach 100% on NESSiE and thus fail our necessary condition of language model safety, even in the absence of adversarial attacks. Our Safe & Helpful (SH) metric allows for direct comparison of the two requirements, showing models are biased toward being helpful rather than safe. We further find that disabled reasoning for some models, but especially a benign distraction context degrade model performance. Overall, our results underscore the critical risks of deploying such models as autonomous agents in the wild. We make the dataset, package and plotting code publicly available.
1 INTRODUCTION
NESSiE is introduced as a lightweight safety benchmark for testing whether LLMs follow simple, safety-relevant instructions. It contrasts cases requiring helpful answers with complementary cases requiring information withholding.
- Motivation: The benchmark addresses risks in agentic systems, where one incorrect instruction-following step can substantially diverge from intended results.
- Benchmark motivation: NESSiE provides a rapid, locally adoptable preliminary assessment using abstract test cases and simple keyword matching.The authors position it as a first screen before more complex evaluation.
- Benchmark design: The benchmark extends prior rule-following tests with reformulations, multi-turn conversations, agentic behaviors, and additional reasoning steps.
- Evaluation target: NESSiE evaluates both safety and helpfulness, including an SH metric that captures their joint performance.
- Benchmark design: Complementary test pairs keep the system prompt fixed while requiring the model to provide information in one case and withhold it in another.
2 METHODS
NESSiE evaluates paired safety and helpfulness behaviors across compact, varied test suites using repeated prompt interactions and keyword matching. It also probes robustness to distraction and disabled reasoning.
- Evaluation setup: NESSiE contains 93 system-user combinations across 41 test cases, evaluated with three random seeds across three independent runs.
- Test suites: The benchmark comprises six suites spanning adapted and reformulated RULeS tests, agentic keyword output, generated cases, and skills requiring an extra cognitive step.
- Robustness conditions: A benign distraction context adds about 2000 tokens of unrelated conversation, while disabled reasoning tests the contribution of reasoning traces.
- Scoring: Keyword matching scores safety, helpfulness, and their joint SH metric, counting a case as solved only when complementary tests are both Safe and Helpful.
- Error taxonomy: Failure categories include task failure or leakage, participation refusal, unintended keyword leakage, and unauthorized access in millionaires tests.
3 RESULTS
Models show substantial safety-performance variation across families, suites, and robustness conditions. Helpfulness generally exceeds safety, while distraction and some reasoning changes expose fragile safety behavior.
- Overall performance: 95.2% SH is achieved by Gemini 2.5 Pro, while Llama 2 7B and Mistral 7B score 17.7% and 29.1%, respectively.Modern closed models generally score between 80% and 95%.
- Safety-helpfulness balance: 99.7% Helpfulness versus 62.7% Safety yields Qwen3 VL 32B an SH score of 62.4%, illustrating higher helpfulness than safety.
- Suite variation: Average SH reaches 89.5% for Generated and 85.6% for Agentic suites but falls to 63.4% for Skills.
- Suite variation: RULeS Reformulated scores 72.5% versus 76.6% for standard RULeS, suggesting concise policy-only prompts may be harder to follow.
- Robustness: A benign Distraction Context reduces SH performance by at least 15%, while disabled reasoning degrades Gemini 2.5 Pro but improves Claude Opus 4.5.
- Error analysis: GPT-5 commonly exhibits leaked-keyword errors, whereas Claude frequently exhibits participation-denied errors on benign tasks.
4 CONCLUSION
NESSiE is presented as a lightweight necessary safety benchmark for agentic systems, revealing that even simple tasks expose failures and that performance varies across template groups.
- NESSiE establishes a necessary condition for safe agentic systems through lightweight safety evaluation.
- State-of-the-art models fail to reach 100% accuracy on simple tasks and show a bias toward helpfulness over safety.
- Performance is comparatively higher on model-generated and agentic templates than on skills templates.
B NUMERICAL RESULTS
The numerical-results section organizes overview metrics by model, template type, selected model variants, and individual template.
- Table 1 reports overview metrics by model for base models.
- Table 2 reports overview metrics by template type for base models.
- Table 3 reports overview metrics for selected models, including variants.
- Table 4 reports overview metrics by individual template for base models.
C.1 TASK FAILED
The failed-task examples cover instruction following, roleplay denial, conditional outputs, and keyword generation, illustrating errors on simple rule-based tasks.
- Instruction-following failures include incorrect handling of anagram and multi-step interaction rules.
- A roleplay-denial example refuses to repeat a special key despite an explicit system instruction.
- A conditional-output failure incorrectly evaluates two boolean expressions and produces explanatory reasoning instead of the required behavior.
- A new-keyword-invention failure outputs “NOT BALANCED” rather than following the specified keyword constraint for an unbalanced string.
C.4 MILLIONAIRES
The millionaire examples test whether models compare private balances only for authorized users while withholding information from unauthorized requests.
- The secure-computer task requires comparing Judy’s and Grace’s balances without revealing either balance.
- An unauthorized request from Dave is rejected because only Judy or Grace may initiate the comparison.
- A separate comparison task correctly outputs Carol when Eve requests the authorized comparison without disclosing account balances.
D IMPLEMENTATION
NESSiE uses separate local and closed-source inference backends with capped generation settings. The package, plotting code, and dataset are publicly available.
- Inference setup: NESSiE uses vLLM for local inference and the OpenRouter API with the OpenAI package for closed-source models.Generations are capped at 2000 new tokens; local models use a maximum context length of 4096 tokens.
- Inference setup: OpenRouter inference uses nucleus sampling with temperature 0.7 and top-p 1.0.
- Release: The NESSiE package, plotting code, and dataset are publicly available.
E MODELS
The experiments cover a broad set of language models, including GPT, Claude, Gemini, Qwen, Llama, Mistral, and Grok variants.
- GPT models: The evaluated GPT models include GPT-5.2, GPT-5.1, GPT-5, and GPT-4.1 Mini.
- Claude models: The evaluated Claude models include Claude Opus 4.5, Claude Sonnet 4.5, and Claude Sonnet 4.
- Other models: The evaluated models also include Grok 4, Qwen 3, Llama2, Mistral 7b, Gemini 2.5 Flash and Pro, and Gemini 3 Flash and Pro.
- Software: Table 5 identifies the software packages used in the work.