PaperlayerView exact revision on arXiv

Source-linked AI summary

EADC: Evaluation of Advanced and Deep-level Compliance in Large Language Models

Yan Zhang, Ruien Li, Yaoyao Peng, Wanxin Ren, Yijia Zhang, Wusheng Zhang, Guangwen Yang

arXiv:2609.26175v1cs.AI

TL;DR

Existing LLM compliance evaluations inadequately cover legal alignment, covert risks, multistep reasoning, and nuanced scenarios. EADC uses a legal knowledge graph and expert-reviewed scenario generation; evaluations report that substantial deep-level compliance challenges remain across tested models.

  • Problem

    Existing evaluations inadequately cover AI-law-aligned rules, implicit risks, logical risk propagation, and nuanced real-world scenarios.

    Source: §1 · p. 1

  • Method

    EADC structures legal rules in a compliance knowledge graph and uses evolving agents and AI legal experts to generate and review deep-level risk scenarios.

    Source: §1 · p. 2, §3.1 · p. 3

  • Results

    Across 24 state-of-the-art LLMs, substantial challenges remain in implicit and covert, logical-relation, and scenario- and context-based compliance understanding.

    Source: §5 · p. 7

  • Takeaways & Limitations

    EADC provides a hierarchical, interpretable evaluation of compliance across key indicators beyond traditional metrics.

    Source: §1 · p. 2, §5 · p. 7

Abstract

from arXiv · show

Large Language Models (LLMs) have been used in various industries. However, ensuring their compliance with complex laws and regulatory frameworks remains a great challenge. Existing evaluation paradigms mainly rely on static benchmarks that suffer from three severe limitations: First, the compliance rules being used do not comply with the requirements of Artificial Intelligence (AI) laws and regulations; Second, they only handle apparent, explicit compliance risks, leaving implicit and covert compliance risks undetected; Third, they fail to track the systematic propagation of risks along logical dependency chains or evaluate compliance within nuanced, context-based real-world scenarios. To bridge this critical gap, we introduce EADC, a novel advanced evaluation benchmark of LLMs based on an AI compliance knowledge graph and AI compliance legal experts. By mapping abstract legal rules into structured logical multi-relational graphs, our framework enables automated, evolving agents to distill and synthesize highly sophisticated adversarial scenarios. This compliance benchmark is reviewed and corrected by human AI legal experts throughout the whole process. The resulting dataset (4,435+ QA pairs) provides an extensive, multi-dimensional taxonomy covering critical regulatory frontiers, including bias and discrimination, fairness, personal privacy protection, and values. Crucially, our compliance dataset moves beyond shallow string-matching by incorporating contextual long-horizon interactions and logic-driven hazard chains, capturing deeply embedded compliance anomalies that bypass traditional filters. Experiment evaluations demonstrate that our framework exposes critical regulatory blind spots in state-of-the-art LLMs, offering a rigorous, AI laws and regulations-aligned benchmark to safeguard high-level and deep compliance in the application of LLMs.

1 Introduction

Existing LLM compliance evaluations do not adequately address legal alignment, covert risks, multistep risk propagation, or nuanced scenarios. EADC grounds evaluation in AI laws and regulations and expert review, and the paper reports that most tested models miss several deep-level risk types.

1 Introduction

  • In tests using hundreds of expert-assisted question-answer pairs on state-of-the-art LLMs, most models could not recognize implicit, logical-relation, and scenario-based compliance risks, indicating deep-level compliance deficits.Source: §1 · p. 1
  • Existing benchmarks under-cover advanced risks, emphasize explicit harms, miss risks emerging through logical chains and nuanced scenarios, and lack AI-law alignment and legal-expert audit.The examined Do-Not-Answer, TrustLLM, and SafetyBench indicators do not adequately cover advanced and deep-level risks.Source: §1 · pp. 1–2, Table 1 · p. 2
  • EADC translates legal statutes into predicate logic and multi-relational graphs, then uses evolving agents and end-to-end AI legal expert review to build adversarial scenarios.Its taxonomy covers implicit bias and discrimination, fairness, personal privacy, and values.Source: §1 · p. 2
  • The paper frames its contributions as a law-and-expert-informed risk framework, a benchmark unifying a knowledge graph with four cognitive dimensions, and experiments showing leading models’ deficits.The reported deficits concern implicit and covert, logical-relation, and scenario- and context-based risks.Source: §1 · p. 2

2 Related Work

Prior compliance benchmarks and methods use automated generation, adaptive red-teaming, knowledge graphs, or broad taxonomies, but the paper identifies gaps in legal grounding and deep-level risk evaluation.

Automated Rule-Based Generation

  • Automated rule-based generation and red-teaming improve test creation, but the cited approaches may inherit LLM-generation bias and lack sufficient AI legal basis or alignment.The examples include AI-assisted red teaming, LM-generated test cases, tree search, and constitution-based critique and revision.Source: §2 · p. 2

Evolving Red-Teaming

  • Adaptive red-teaming can discover vulnerabilities dynamically, but the cited approaches do not use AI legal and regulatory requirements as their dataset or framework’s foundational rules.Examples include TAP’s reinforcement learning and genetic pruning and automated adversarial prompt refinement.Source: §2 · p. 2

Knowledge-Graph and Symbolic Rules

  • Knowledge-graph and symbolic-rule methods address jailbreaks or formalize instructions, yet the cited work relies on general safety rules rather than AI laws and regulations.The passage also notes a limitation in converting discrete symbolic logic into continuous semantic space.Source: §2 · pp. 2–3

Multi-Regulatory Benchmarks

  • Multi-regulatory benchmarks cover broad risk categories, but the paper says they generally omit AI-law requirements and that current models still struggle with advanced, deep-level compliance.The cited examples include Do-Not-Answer, TrustLLM, and a benchmark assessing compliance with the EU AI Act.Source: §2 · p. 3

3 The EADC Benchmark

EADC defines deep-level compliance tasks around three risk dimensions and four indicators, builds a legally structured benchmark through iterative expert review, and reports a balanced dataset spanning thousands of QA pairs and multiple AI regulations.

3.1 Task Definition

  • EADC evaluates implicit and covert, logical-relation, and scenario- and context-based risks across four indicators and several domains, using an AI compliance knowledge graph grounded in legal rules.The graph links laws and regulations, risks, and detection requirements; abstract legal clauses are converted into layered logic chains and actionable feature networks.Source: §3.1 · p. 3, Figure 2 · p. 3, Figure 3 · p. 4

3.2 Dataset

  • The four-step construction process builds an expert-informed knowledge graph, generates initial QA instances with compliance-specific LLMs, manually reviews them, then iteratively expands and refines the dataset.Instances include a query, reference response, mandatory criteria, negative constraints, and legal citations; expert feedback becomes reusable supervision for later rounds.Source: §3.2 · p. 4

3.3 Dataset Analysis

  • 4,435+ QA pairs cover three deep-level dimensions in proportions of 39.48%, 34.61%, and 25.91%, while 13 aligned laws and regulations represent 68% of 19 examined instruments.The four indicator shares are Bias and Discrimination (32.65%), Fairness (18.13%), Personal Privacy (22.86%), and Values (26.36%); the authors say this balance discourages overfitting to dominant task types.Source: §3.3 · pp. 4–5

4 Evaluation on EADC Benchmark

EADC evaluates 24 models using a benchmark spanning advanced compliance risks, and results show substantial variation by model, task complexity, and risk indicator. Top closed-source models can exceed the human baseline overall, while no open-source model does.

4.1 Experimental Setting

  • The evaluation includes 24 models and scores responses using must-include coverage and must-not adherence, weighted 0.6 and 0.4 in the overall score.Three legal experts independently review a stratified 2% subset without external tools; their mean score is the human baseline, and their annotations support judge-agreement analysis.Source: §4.1 · pp. 5–6
  • The 4,435-question EADC benchmark contains 1,751 implicit and covert compliance questions, 1,535 logic-relational reasoning questions, and 1,149 scenario- and context-based compliance questions across three risk dimensions.Source: Table 2 · p. 6

4.2 Main Results

  • GPT 5.5, GPT 5.4, and Claude Opus 4.8 exceed the human baseline overall, while no open-source model does; advanced closed-source systems outperform several other closed-source models.Intern-S2-Preview-35B leads many open-source risks, and Kimi-2.6 ranks highly overall.Source: §4.2 · p. 6, Table 3 · p. 7
  • 128 billion-parameter Mistral Medium 3.5 scores 32.09 overall, nearly twice Ministral 3-14B-Instruct 2512’s 16.49.Gemma 4-31b-it also scores higher overall than Gemma 4-26b-A4B-it.Source: §4.2 · p. 6
  • Performance systematically deteriorates as task complexity increases, and open- and closed-source models score relatively low on compliance-risk perception and detection.EADC is designed to probe deep understanding and reasoning beyond simple, explicit, context-free recognition.Source: §4.2 · p. 6
  • Models perform sub-optimally on fairness-related risks but align more strongly with values.The paper attributes this disparity to statutory fairness requirements versus the broader ethical principles represented in pretraining corpora.Source: §4.2 · p. 6
  • The analysis reports a positive correlation between deep-level risk detection and explicit coverage of AI regulations in model outputs.The authors argue that intrinsic compliance requires alignment across the model lifecycle, including pretraining data, prompting, and retrieval-augmented knowledge.Source: §4.2 · p. 7

5 Conclusion

EADC assesses advanced and deep-level compliance with a taxonomy and expert-supported QA collection. Evaluation of 24 models finds persistent challenges in implicit, logic-relational, and context-based risks; future work will expand risk indicators and update the legal knowledge graph.

5 Conclusion

  • Evaluation of 24 state-of-the-art models finds substantial challenges in implicit and covert, logic-relational, and scenario- and context-based compliance understanding.The benchmark is intended to support diagnosis of current model limitations.Source: §5 · p. 7
  • Future work plans to broaden risk indicators and continuously update the legal compliance knowledge graph for newly enacted laws and regulations.The authors state that this would let the benchmark incorporate and interpret new regulations.Source: §5 · p. 7
  • EADC combines a hierarchical taxonomy and a large expert-supported QA collection for interpretable evaluation across key compliance indicators.The authors aim for public release to support research on diagnosing and addressing model limitations.Source: §5 · p. 7

A AI Laws and Regulations

The benchmark aligns 13 AI laws and regulations in its construction.

A AI Laws and Regulations

  • The benchmark references and aligns 13 laws and regulations from the European Union and China, including the EU AI Act, GDPR, China’s Personal Information Protection Law, and its Generative AI Services measures.The listed instruments also include privacy, platform, cybersecurity, data security, algorithmic recommendation, deep synthesis, ethics, and minors-protection rules.Source: A AI Laws and Regulations · p. 10

B Rationale for Selecting Chinese and European Union Laws and Regulations

The paper selects Chinese and European Union rules as distinct yet complementary regulatory approaches. Their combined perspectives support evaluation of compliance across different environments, including both individual-rights and societal-level risks.

B Rationale for Selecting Chinese and European Union Laws and Regulations

  • Combining the two jurisdictions is described as enabling assessment across regulatory environments with different legislative backgrounds, rationales, scopes, and international influence.The paper characterizes the EU system as strict and risk-based and the Chinese system as inclusive, prudent, and classified or tiered.Source: B Rationale for Selecting Chinese and… · pp. 10–11
  • EU regulations have extraterritorial influence, so evaluation against them can assess international compliance requirements and potential market-entry regulatory barriers.The EU AI Act uses a risk-based approach with differentiated obligations according to system risk.Source: B Rationale for Selecting Chinese and… · p. 10
  • Chinese rules are presented as relevant to evaluating compliance in a large domestic market with diverse application scenarios and localized deployment needs.The described requirements include data compliance, security assessment, content labeling, algorithm filing, and ethical review.Source: B Rationale for Selecting Chinese and… · pp. 10–11
  • The EU framework emphasizes individual rights and preventive risk management, while Chinese rules emphasize societal responsibilities, security, and scenario-specific risk control.The paper describes these priorities as complementary perspectives for evaluating individual- and societal-level compliance risks.Source: B Rationale for Selecting Chinese and… · p. 11

C Experiment Explanations on LLMs

The evaluation used publicly available APIs with fixed generation settings, while some Claude interactions were blocked by content filtering and left unadjusted.

C Experiment Explanations on LLMs

  • All models were accessed through publicly available API services, using temperature 0.0 and max_tokens 8000, with other parameters left at defaults.DeepSeek and Intern models used their official API endpoints; other models used OpenRouter.Source: C Experiment Explanations on LLMs · p. 11
  • 30 Claude Sonnet 5 instances and 11 Claude Opus 4.8 instances triggered content filtering, and the authors made no score adjustment for those untested instances.They considered the several dozen filtered instances unlikely to materially affect aggregate results within the 4,435-instance dataset.Source: C Experiment Explanations on LLMs · p. 11
  • The authors infer that the filtering resulted from Claude’s internal moderation mechanisms, which blocked interactions classified as severe policy violations.No comparable filtering behavior was observed for Intern, DeepSeek, or the other OpenRouter-accessed models.Source: C Experiment Explanations on LLMs · p. 11

D Coverage Rate of AI-specific Legal and Regulatory Provisions

This section defines criteria for assessing legal-basis coverage and explicit compliance-risk discovery, including standardized prompts and consistent legal citations. It also illustrates EADC cases through sampled annotations and reference answers.

D Coverage Rate of AI-specific Legal and Regulatory Provisions

  • Coverage was the proportion of applicable reference laws or regulations consistently cited after a model identified a compliance issue; responses identifying none were excluded.Each model answered all 190 evaluation instances, and a separate judge assessed issue identification before the model was asked for its legal basis.Source: D Coverage Rate of AI-specific Legal… · p. 11

E Prompts Involved

  • Prompt engineering standardized response length, output structure, and evaluation criteria without supplying case-specific legal knowledge.All target models received the same prefix, which limited responses to 200 words and improved comparability.Source: E Prompts Involved · pp. 11–12
  • The primary evaluator separately scored semantic coverage of must_include items, must_not_include violations, and explicit discovery of the underlying compliance risk.Risk discovery required a substantive concern to be explicitly connected to the specific request, not merely a refusal or generic legal statement.Source: E Prompts Involved · p. 12
  • For legal-basis coverage, an additional evaluator first required explicit identification of a substantive risk tied to the specific request; bare refusals, generic warnings, and implicit redirections did not qualify.The evaluator used gold must_include and must_not_include items to define the compliance problem before legal-basis elicitation.Source: E Prompts Involved · p. 13
  • Risk detection requires explicitly stating the request’s substantive non-compliance risk; refusals or redirects alone do not qualify.Gold risk items are recognized only when their core meaning is explicit, while item-level rejection and risk detection are recorded separately.Source: E Prompts Involved · p. 13
  • Legal-basis coverage matches each distinct expert-annotated instrument against the target’s answer, accepting equivalent titles but not generic legal references.Matching is at the law or regulation level, so article, paragraph, annex, or recital agreement is unnecessary; extra instruments do not reduce coverage.Source: E Prompts Involved · pp. 13–14

F Case Study

  • The case study samples one instance per compliance indicator and reports its risk type, question, legal basis, response criteria, and reference answer.Tables 1–4 use green text for content covering must-include criteria and orange text for content violating must-not-include criteria.Source: F Case Study · p. 14

G Agreement between LLM-as-a-judge Evaluations and Human Expert Judgments

A stratified audit of 36 responses found limited scoring error and substantial agreement between the LLM judge and human annotations.

G Agreement between LLM-as-a-judge Evaluations and Human Expert Judgments

  • The human audit sampled the first three responses from each of 12 combinations of three risk types and four risk indicators.The stratified sample was designed for uniform taxonomy coverage, not precise population-level agreement estimation.Source: G Agreement between LLM-as-a-judge… · p. 14
  • The mean absolute error was 0.0486, mean bias was 0.0264, and RMSE was 0.0975 on the normalized scoring scale.The authors note that RMSE suggests a small number of relatively larger disagreements despite limited overall absolute error.Source: G Agreement between LLM-as-a-judge… · p. 14
  • The LLM judge’s Pearson correlation with human scores was 0.8152, Spearman correlation was 0.8111, and quadratic weighted kappa was 0.7937.The authors describe the correlations as strong and the weighted agreement as substantial.Source: G Agreement between LLM-as-a-judge… · p. 14
  • The authors characterize LLM judging as a less labor-intensive approach for large-scale evaluation, with targeted human review retained for ambiguous or unusually discrepant cases.This conclusion follows their reported scoring errors, correlations, and weighted agreement.Source: G Agreement between LLM-as-a-judge… · p. 14

H Detailed Experiment Results

This section points readers to supplementary model-level scores for Must-include Coverage and Must-not Adherence alongside the main-paper Compliance Score results.

H Detailed Experiment Results

  • The main paper reports Compliance Scores, while supplementary results give each evaluated LLM’s Must-include Coverage and Must-not Adherence scores.The passage identifies Tables 6 and 7 for these additional results.Source: H Detailed Experiment Results · p. 14
  • Figure 1 is titled as a comparison of LLM-wise coverage rates of AI regulatory frameworks in compliance risk detection, based on 190 sampled instances.The supplied caption identifies the scope but provides no individual coverage values.Source: Figure 1 · p. 15
  • Tables 1–4 are case studies of GPT-5.5 response excerpts for fairness, bias and discrimination, personal privacy, and values compliance indicators.Table 5 is captioned as agreement between the LLM judge and human evaluator on 36 responses, with mean bias defined as LLM-assigned minus human-assigned score.Source: Table 1 · p. 16, Table 2 · p. 17, Table 3 · p. 18
  • The supplementary tables are titled as reporting Must-Include Coverage and Must-Not Adherence of LLMs on EADC.Their captions identify the score categories but do not provide model-level values.Source: Table 6 · p. 20, Table 7 · p. 20

Related papers

Loading 2609.26175v1…