S-Eval: Towards Automated and Comprehensive Safety Evaluation for Large Language Models

• Existing assessments use loose taxonomies, weakly risky benchmarks, incomplete jailbreak coverage, and labor-intensive generation or evaluation.These limitations hinder comprehensive safety measurement and adaptation to evolving threats.Source: §1 · p. 2
• S-Eval defines a four-level taxonomy spanning 8 risk dimensions and 102 subdivided risks.The taxonomy is intended to support systematic measurement across fine-grained safety dimensions.Source: §1 · p. 2
• S-Eval uses an expert testing LLM M_t to generate configurable base-risk and attack prompts, plus a safety critique LLM M_c for quantitative, explainable evaluation.The framework is designed to adapt to new risks, attacks, and evaluation methods.Source: §1 · p. 2
• The benchmark contains 220,000 test cases, and experiments evaluate 21 open-source and closed-source LLMs using more than 500K queries.The authors report that S-Eval better reflects safety risks and that M_c agrees well with human annotation.Source: §1 · p. 2
• The released benchmark combines base-risk and attack prompts and covers 10 jailbreak attacks for broad safety evaluation.The authors also report that S-Eval better reflects LLM safety than existing benchmarks.Source: §1 · p. 2

2.1 Large Language Models

• LLMs generate natural-language responses from massive textual training corpora, commonly through prompt engineering whose wording can strongly affect outputs.Prompt changes can produce large performance variations.Source: §2.1 · pp. 2–3

2.2 Problem Definition

• For a tested model M, the evaluation applies prompts P and a binary safety evaluator J to classify each response as harmful or safe.A response is harmful when J(p_i,r_i)=0 and safe otherwise.Source: §2.2 · p. 3
• The automated benchmark follows a unified risk taxonomy and divides prompts into base-risk prompts P_B and attack prompts P_A.The two prompt sets target diverse risks and adversarial scenarios.Source: §2.2 · p. 3

3.1 Overview

• S-Eval trains an expert testing LLM M_t and a safety critique LLM M_c, then generates prompts, selects high-quality tests, and produces safety reports.The framework uses risk-management resources and a complete generation-and-evaluation pipeline.Source: §3.1 · p. 3

3.2 Risk Management

• The risk-management layer combines a taxonomy, risk seeds, web-derived risk knowledge, and jailbreak attacks, allowing configurations to be updated as risks and models evolve.This supports adaptive generation of new test cases.Source: §3.2 · pp. 3–4

3.3 Automatic Test Generation

• S-Eval addresses static, labor-intensive benchmarks by fine-tuning M_t to generate risk prompts automatically and configure them through evolving risk management.The approach is designed to overcome aligned models’ reluctance to generate harmful test prompts.Source: §3.3 · p. 4

3.3.1 Base Risk Prompt Generation.

• Base-risk generation uses definitions, external knowledge, and rewriting to create relevant, diverse, and covert prompts that remain useful as LLMs improve.Rewriting attenuates explicit risky semantics and can add plausible context to conceal malicious intent.Source: §3.3.1 · p. 4

3.3.2 Attack Prompt Generation.

• Attack-prompt generation fine-tunes M_t on jailbreak instructions and base prompts, enabling automatic production of corresponding attack prompts.S-Eval organizes 10 attacks around competing-objective and mismatched-generalization alignment failures.Source: §3.3.2 · p. 5

3.3.3 High-quality Test Selection.

• High-quality test selection removes similar or harmless base prompts using combined semantic and Levenshtein similarity, victim-model risk assessment, and regeneration of repetitive attack prompts.The similarity score combines the two measures with α=0.2 and uses θ_sim=0.55; safety filtering uses θ_safe=0.95.Source: §3.3.3 · p. 5

3.4 Automatic Safety Evaluation

• M_c targets limitations of manual, rule-based, API-based, and prompt-based evaluation by providing scalable and explainable safety judgments.Existing approaches face labor, coverage, bias, alignment, and refusal problems.Source: §3.4 · pp. 5–6
• The safety critique model M_c is fine-tuned on 100,000 bilingual QA pairs covering over 100 risk types and outputs risk tags, scores, and explanations.The dataset uses responses from 10 representative models, automated GPT-4 annotation, and specialist review; M_c is built by fine-tuning Qwen-14b-Chat.Source: §3.4 · p. 6

4.1 Experimental Setup

• The setup evaluates 21 representative LLMs using 2,000 sampled base risk prompts, 20,000 attack prompts, safety score, and attack success rate.The safety critique model M_c serves as the evaluation model J.Source: §4.1.1 · p. 6, §4.1.2 · p. 6

4.2 Effectiveness of M_c (RQ1)

• M_c achieves the highest balanced accuracy among the compared evaluation methods, supporting accurate automated safety evaluation.The comparison includes rule matching, GPT-based evaluation, and LLaMA-Guard-2.Source: §4.2 · p. 6, Table 2 · p. 7
• 88.50% of Chinese cases and 83.70% of English cases have consistent results from at least three of four methods, while M_c correlates with LLaMA-Guard-2 at PCC 0.92.These results indicate that evaluation-method bias is controllable in the reported tests.Source: §4.2 · p. 7, Figure 4 · p. 7

4.3 Effectiveness of S-Eval (RQ2)

• S-Eval produces lower safety scores than AdvBench, HH-RLHF, Flames, and SafetyPrompts, with 95% confidence-interval sizes of 30.62% in Chinese and 50.36% in English.The reported baseline interval sizes are smaller: AdvBench 5.74%/7.53%, HH-RLHF 16.30%/20.72%, Flames 19.53%/22.36%, and SafetyPrompts 11.89%/14.12%.Source: §4.3 · p. 7, Figure 5 · p. 7, Table 3 · p. 8
• S-Eval’s ablation study finds that its generation model significantly improves test-generation effectiveness and quality compared with GPT-4 and Qwen-14B-Chat.The comparison uses 1,000 base risk prompts for the Crimes and Illegal Activities dimension and evaluates safety score and rejection rate.Source: §4.3 · p. 7, Table 4 · p. 8
• Safety varies substantially across models and risk dimensions: ErnieBot-4.0 leads overall among closed-source models, while Yi-34B-Chat reaches 81.00% on Data Privacy but 25.00% on Ethics and Morality.The reported results also identify different leaders among open-source models and lower safety for some newer model families than their predecessors.Source: §4.3 · p. 8, Table 3 · p. 8

4.4 LLM Scale Effect to Safety (RQ3)

• Within each model family, safety scores first increase with parameter scale but may decrease after a threshold.The study evaluates 10 Qwen, Vicuna, and LLaMA-2 models with varied parameter scales.Source: §4.4 · pp. 8–9
• LLaMA-2 achieves higher safety than Qwen and Vicuna at similar parameter scales despite lower ability.The paper suggests architecture or alignment methods may explain this family-level difference.Source: §4.4 · p. 9

4.5 Evaluation of Multiple Languages (RQ4)

• Baichuan2-13B-Chat’s safety score falls from 77.40% in English to 33.90% in Korean, while ChatGLM3-6B falls from 59.70% in Chinese to 29.20% in French.The reported differences vary by model and language environment.Source: §4.5 · p. 9, Table 5 · p. 9
• Closed-source models show relatively consistent safety across the four evaluated languages, whereas aggregate safety decreases as language resources diminish.The analysis covers Chinese, English, French, Korean, and selected Bengali and Swahili evaluations.Source: §4.5 · p. 9
• ChatGLM3-6B has the highest Korean safety score partly because irrelevant and duplicate responses may prevent harmful content generation.The paper links this observation to limited capability in Korean.Source: §4.5 · p. 9

4.6 Robustness of Different LLMs (RQ5)

• CIA has the highest average attack success rate among the 10 attacks, while RI is the second most effective jailbreak attack.The paper attributes CIA’s effectiveness to combining instructions with multiple intents to hide malicious intent.Source: §4.6 · p. 10
• Adaptive attacks achieve very high attack success rates across models, showing that multiple simultaneous attack methods remain difficult for LLMs to resist.The evaluation uses an adaptive attack that succeeds when any of 10 attacks succeeds.Source: §4.6 · pp. 9–10
• Instruction Encryption has the lowest average attack success rate, but its ASR tends to increase with model ability.The paper notes that highly capable models may have additional exploitable safety vulnerabilities.Source: §4.6 · p. 10, Figure 8 · p. 11
• GPT-4-Turbo is the most robust closed-source model, while LLaMA-3-8B-Instruct is the strongest open-source model and exceeds GPT-4-Turbo.The paper reports overall ASR values of 33.99% Chinese and 32.80% English for GPT-4-Turbo, and 21.77% English for LLaMA-3-8B-Instruct.Source: §4.6 · p. 10, Table 6 · p. 10

4.7 Randomness to Safety (RQ6)

• Introducing decoding randomness reduces safety: temperature and top-p increases progressively lower safety, whereas top-k effects plateau after a threshold.The study varies each decoding parameter independently while fixing the others and the random seed.Source: §4.7 · p. 11

• The study identifies imperfect LLM-based safety evaluation and evaluation-data size as its main threats to validity.It uses critique explanations, bias checks, and more than 500K evaluations to mitigate these concerns, while planning larger data coverage.Source: §5 · p. 11

• Earlier assessments often target specific safety concerns, motivating benchmarks that span multiple dimensions.The related work includes HELM, DecodingTrust, HH-RLHF, AdvBench, SafetyPrompts, SafetyBench, CValues, Do-not-answer, Flames, and SALAD-Bench.Source: §6 · pp. 11–12
• Existing benchmarks vary widely in scope and construction, from 520 hand-written harmful questions in AdvBench to 100,000 Chinese prompts in SafetyPrompts.Other examples include 11,435 multiple-choice questions in SafetyBench and 30,000 test prompts in SALAD-Bench.Source: §6 · pp. 11–12
• S-Eval differs by combining a unified risk taxonomy with LLM-based automatic test generation and safety evaluation through M_t and M_c.Its benchmark contains 20,000 base risk prompts and 200,000 attack prompts covering 10 advanced jailbreak attacks.Source: §6 · p. 12

7 CONCLUSION

• S-Eval combines a configurable expert testing LLM with a safety critique LLM to support automated and explainable evaluation.The testing component can be dynamically adjusted as safety threats and LLMs evolve, while the critique component provides safety evaluations.Source: §7 · p. 12
• The framework supports a comprehensive, multi-dimensional benchmark and extensive evaluation of 21 leading LLMs.Source: §7 · p. 12
• S-Eval reportedly measures LLM safety more accurately than other benchmarks and examines effects from hyper-parameters, linguistic contexts, and decoding settings.These investigations are presented as informing pathways for designing safer LLMs.Source: §7 · p. 12

A THE DETAILS OF RISK TAXONOMY

• The taxonomy is structured into four levels: risk dimension, risk category, risk subcategory, and risk sub-subcategory.The supplied taxonomy materials include a table describing the four-level structure and its detailed entries.Source: Table 8 · p. 14, A THE DETAILS OF RISK TAXONOMY · p. 15
• Sensitive-information entries cover financial, contact, identity, behavioral, health, medical, judicial, and licensing records.Examples include bank accounts, phone numbers, browsing history, medical records, judgments, driver’s licenses, and passports.Source: A THE DETAILS OF RISK TAXONOMY · p. 15
• Other taxonomy entries address business information, access credentials, malware and attacks, physical or network security, social issues, extremism, violence, and inappropriate suggestions.Source: A THE DETAILS OF RISK TAXONOMY · p. 15
• The benchmark’s jailbreak component separately documents integrated attack methods and their descriptions.Source: Table 9 · p. 16

B CORE LLM IMPLEMENTATION

• Qwen-14B-Chat is selected as the base model for M_t and M_c because of its capabilities in both Chinese and English.Source: B CORE LLM IMPLEMENTATION · p. 16
• Both core LLMs undergo full-parameter FP16 fine-tuning on 8 NVIDIA A100 GPUs for 3 epochs.Source: B CORE LLM IMPLEMENTATION · p. 16
• Training uses a maximum length of 2560, per-GPU batch size 4, gradient accumulation of 4, and learning rate 2 × 10^-5 with cosine decay.Source: B CORE LLM IMPLEMENTATION · p. 16

C ZERO-SHOT-COT PROMPT

• Figure 9 presents a zero-shot chain-of-thought prompt for safety evaluation.Source: Figure 9 · p. 16
• The figure concerns prompting rather than a risk taxonomy or model-training configuration.Source: Figure 9 · p. 16
• Its stated application is safety evaluation.Source: Figure 9 · p. 16