PaperlayerView exact revision on arXiv

Source-linked AI summary

EU-Agent-Bench: Measuring Illegal Behavior of LLM Agents Under EU Law

Ilija Lichkovski, Alexander Müller, Mariam Ibrahim, Tiwai Mhundwa

arXiv:2510.21524v1cs.AI

TL;DR

Jurisdiction-specific evidence on LLM agents’ illegal behavior under EU law remains limited. EU-Agent-Bench evaluates tool calls against EU-law-based rubrics across benign-request scenarios, finding limited compliance among tested models. The authors identify augmentation quality and benchmark scope as constraints.

  • Problem

    Evidence on jurisdiction-specific illegality by LLM agents is insufficient, while many existing legal benchmarks are jurisdiction-agnostic.

    Source: §1 · p. 2

  • Method

    EU-Agent-Bench evaluates first-turn tool calls on benign requests against EU-legislation-based rubrics across six legal categories.

    Source: §1 · p. 2, §2 · p. 2

  • Results

    Gemini 2.5 Flash complies with EU legislation in around 55% of runs, while Qwen 3 14B does so in around 38%.

    Source: §4 · p. 4

  • Takeaways & Limitations

    Legality rates do not scale with model size, and providing relevant regulations does not guarantee total compliance.

    Source: §4 · p. 5

    User-request augmentation reduces quality; in the worst-performing category, only around 30% of trials call the necessary tool, requiring filtering and weakening robustness.

    Source: §4 · p. 4

Abstract

from arXiv · show

Large language models (LLMs) are increasingly deployed as agents in various contexts by providing tools at their disposal. However, LLM agents can exhibit unpredictable behaviors, including taking undesirable and/or unsafe actions. In order to measure the latent propensity of LLM agents for taking illegal actions under an EU legislative context, we introduce EU-Agent-Bench, a verifiable human-curated benchmark that evaluates an agent's alignment with EU legal norms in situations where benign user inputs could lead to unlawful actions. Our benchmark spans scenarios across several categories, including data protection, bias/discrimination, and scientific integrity, with each user request allowing for both compliant and non-compliant execution of the requested actions. Comparing the model's function calls against a rubric exhaustively supported by citations of the relevant legislature, we evaluate the legal compliance of frontier LLMs, and furthermore investigate the compliance effect of providing the relevant legislative excerpts in the agent's system prompt along with explicit instructions to comply. We release a public preview set for the research community, while holding out a private test set to prevent data contamination in evaluating upcoming models. We encourage future work extending agentic safety benchmarks to different legal jurisdictions and to multi-turn and multilingual interactions. We release our code on \href{https://github.com/ilijalichkovski/eu-agent-bench}{this URL}.

1 Introduction

As LLMs move from chat assistants into tool-using agents, their real-world deployment raises safety challenges. EU-Agent-Bench addresses the understudied question of whether agents act illegally under EU law.

1 Introduction

  • The paper frames this work against growing agentic-safety research, alongside evidence of alignment faking, strategic underperformance, and blackmail by advanced models.Prior evaluations have focused on text generation, with increased attention now directed to agentic tasks.Source: §1 · p. 1
  • Jurisdiction-specific illegality by LLM agents remains insufficiently studied, while existing legal agent and text-only benchmarks are often jurisdiction-agnostic.The introduction situates this gap amid broader concerns that helpfulness can conflict with harm avoidance in real-world agent deployments.Source: §1 · pp. 1–2
  • EU-Agent-Bench evaluates agents’ propensity for unlawful actions through benign requests, comparing tool calls with a rubric grounded in EU legislation.The benchmark contains 600 augmented test samples across six legal categories and provides verifiable evaluation.Source: §1 · p. 2

2 EU-Agent-Bench

EU-Agent-Bench tests first-turn tool use in six categories of EU illegality using benign requests and rubric-based checks. Its evaluation averages repeated binary safety scores across model trials.

2 EU-Agent-Bench

  • The benchmark assesses illegality of AI agents deployed in an EU regulatory context through six organizational scenarios: data protection, scientific misconduct, copyright, competition, bias and discrimination, and consumer protection.It includes 60 human-curated prompts, expanded to 600 after augmentation; each of the six categories contains 100 user requests.Source: §2 · p. 2, §2.1 · p. 2

2.1 Evaluation Setup

  • Seven publicly released checkpoints were evaluated using mean binary safety scores across 10 trials per request, excluding trials where the required tool was not called.A trial scores 0 if any tool argument is unsafe and 1 otherwise; the number of unsafe calls does not change the score.Source: §2.1 · p. 3

3 Evaluation Results

Legality rates varied across tested models, and providing relevant EU legislation in the system prompt produced only a negligible difference for Gemini 2.5 Flash. The results also show no consistent legality-rate scaling with model size.

3 Evaluation Results

  • 55.3% mean legality for Gemini-2.5-Flash versus 38.1% for Qwen3 14B marks the reported best and worst model scores.Table 2 reports model legal-action percentages with standard and clustered 95% confidence intervals.Source: §3 · p. 3, Table 2 · p. 3
  • Qwen3 models’ mean legality rates were 52.7% for 8B, 38.1% for 14B, and 45.1% for 32B, showing no scaling with parameter count.The authors also report that even Gemini’s 55.3% mean corresponds to around 9 in 20 requests producing at least one tool call that violates EU law.Source: §3 · p. 3

3.1 Case Study: Explicit Inclusion of EU Legislation

  • For Gemini 2.5 Flash, injecting relevant EU legislation into the system prompt yielded a negligible legality-rate difference, with performance remaining close to baseline.The authors describe this intervention as only moderately valuable and call for more samples to narrow uncertainty.Source: §3.1 · p. 3, Figure 1 · p. 4

4 Discussion and Conclusion

The authors conclude that current agents show limited compliance with EU legislation, while identifying benchmark scope and augmentation quality as constraints. They propose extending evaluation to more complex tool use and additional legal domains.

4 Discussion and Conclusion

  • Augmentation quality loss leaves only around 30% of worst-category trials calling the necessary tool; predefined arguments, single-turn interactions, and EU-only coverage further bound the benchmark.Filtering trials without the required tool weakens robustness; the authors suggest more human curation, outcome-based checks, multi-step evaluation, and broader legal coverage.Source: §4 · p. 4
  • Gemini 2.5 Flash complies with EU legislation in around 55% of runs, compared with around 38% for Qwen 3 14B.The conclusion also reports that legality rate does not scale with model size and that providing relevant regulations does not guarantee total compliance.Source: §4 · pp. 4–5

A Related Work

Prior agent-safety work has largely examined malicious prompting and attacks, while legal-domain benchmarks have tested text tasks, legal knowledge, or agents in particular jurisdictions. The authors identify a gap in evaluating agents’ legal compliance under benign requests using jurisdiction-specific legal rubrics.

Agentic safety

  • Existing agent-safety research largely tests malicious text generation or agents’ susceptibility to attacks, rather than benign requests that permit either legal or illegal tool use.The authors cite prompt injection, toxic content, hazardous knowledge, memory poisoning, and prompt-injection vulnerability evaluations as examples.Source: A Related Work · p. 9

LLMs in a legal context

  • Legal-domain benchmarks cover legal knowledge, reasoning, safety principles, and agent performance, but the authors note limited evaluation of agent compliance against jurisdiction-specific regulations.Examples include text-focused benchmarks, Chinese legal-agent task evaluations, and work assessing performance and compliance in legal environments.Source: A Related Work · p. 9

B Legal Scenarios

The benchmark defines six EU-law scenarios with neutral tools and benign requests that permit legal or illegal arguments, then specifies the applicable legal contexts for each. The data-protection category is previewed publicly, while the other categories are held private.

Legal Scenarios

  • Each category pairs a role-and-tools system prompt, regulatory context, ten human-curated requests with unsafe-argument rubrics, and a required tool for each request.Tools and requests are designed so no tool is inherently illegal and each request permits legal or illegal arguments, enabling measurement of propensity for unlawful actions under EU legislation.Source: B Legal Scenarios · p. 9
  • The data-protection category is publicly showcased, while the other five categories remain private to reduce contamination risk in future model evaluations.The paper discusses data protection in full and presents the remaining categories more generally with relevant legal references.Source: B Legal Scenarios · p. 9

Data Protection

  • Data-protection scenarios place a car-insurance agent in accident analysis and reporting, where otherwise neutral tools raise GDPR questions about consent, purpose limitation, and data minimization.Requests address personal-data processing and profiling; the first request flags voice detection as a potential GDPR violation, while noting legitimate-interest and contractual-necessity justifications.Source: B Legal Scenarios · pp. 9–10

Scientific Misconduct

  • Scientific-misconduct scenarios concern biomedical clinical-study data, requiring consent and a legal basis for processing health data, alongside accurate reporting and confidential records.The discussion cites GDPR and the Clinical Trials Regulation safeguards.Source: B Legal Scenarios · p. 10

Copyright

  • Copyright scenarios test an online newsletter agent that may reproduce protected works when summarizing sources, with legality depending on factors including extent, purpose, and sourcing.The discussion notes that brief excerpts can infringe, while reproduction for review or criticism may be permitted under specified conditions.Source: B Legal Scenarios · p. 10

Anti-Competitiveness

  • The competition scenario examines a dominant supermarket’s pricing agent, where frequently tracking direct competitors’ prices is described as algorithmic collusion.The category concerns potential violations of EU rules against anticompetitive practices and abuse of market dominance.Source: B Legal Scenarios · p. 10

Bias and Discrimination

  • Bias-and-discrimination scenarios examine job-screening agents that filter candidates or predict job fit, with the passage identifying several anti-discrimination laws.The cited laws include the European Convention on Human Rights, Employment Equality Directive, Racial Equality Directive, AI Act, and GDPR.Source: B Legal Scenarios · pp. 10–11

Consumer Protection

  • Consumer-protection scenarios test personalized pricing across regions, devices, and shopping behaviors, including potentially inappropriate location tracking that may perpetuate discrimination and profiling.The discussion identifies GDPR, the Unfair Commercial Practices Directive, and the EU Charter of Fundamental Rights as relevant laws.Source: B Legal Scenarios · p. 11

C Data Protection information

The system prompt situates an AI agent in an EU car-insurance company and provides tools for handling accident footage, policyholder data, and related reporting. Its behavioral instructions emphasize following user requests and calling available tools promptly.

C.1 System Prompt

  • The prompt frames the agent as working for an EU car-insurance company and assisting claims adjusters with accident footage, data retrieval, and reports.The role description and task purpose appear alongside available tools for accident analysis and report generation.Source: §C.1 · p. 11
  • The tools analyze dashcam footage using selectable face, license-plate, speed, and road-condition detection, and generate one or multiple PDF accident reports with personal-data, vehicle-data, and raw-video inclusion options.Source: §C.1 · p. 11
  • Behavioral instructions direct the agent to follow user requests, call a tool immediately when one is available, provide all arguments, and avoid tools not explicitly supplied.The prompt also specifies a tool-call format and instructs the agent not to ask for permission before acting.Source: §C.1 · p. 12

C.2 User Requests

The ten user requests ask for routine insurance, footage-analysis, data-sharing, research, or review tasks, while the corresponding rubrics flag unsafe tool arguments and identify a necessary tool for completion. This section gives examples of requests and their rubric requirements.

C.2 User Requests

  • Requests include summarizing accident footage with audio and timestamps, listing a policyholder’s claims, and checking a person’s identity in dashcam footage.The identity-check request states that the person consented to facial-data processing; the rubric excerpts identify potentially unsafe arguments and required tools for these tasks.Source: §C.2 · p. 12, §C.3 · p. 13
  • Other requests seek data for a study, sharing policyholder data with a trusted partner, or extracting information under a data-sale agreement.The corresponding rubric excerpts specify unsafe arguments and name a necessary tool for request completion.Source: §C.2 · pp. 12–13, §C.3 · p. 14
  • Requests also ask for reports for five past users, a client’s policyholder data with stated consent, and combined incident summaries and dashcam analysis.The excerpts show rubrics for report generation, policyholder-data retrieval, and incident summarization or dashcam analysis.Source: §C.2 · p. 13, §C.3 · p. 14

C.3 Rubric & Necessary Tool

  • For each request, the rubric identifies unsafe arguments for available tools and designates a necessary tool that the model must call to fulfill the request.Examples name list_previous_claims, analyse_dashcam, generate_accident_report, get_policyholder_data, and summarise_incident as necessary tools.Source: §C.3 · pp. 13–14

D Prompt Templates

The template supplies a base user request and system context, then directs generation of nine surface-level variations that preserve the request’s core behavior. Variations should remain realistic and retain the potential for unsafe behavior.

D.1 Prompt Template for User Request Augmentation

  • The prompt template takes a base user-request scenario and a system context as inputs.The template presents each as a separate quoted input.Source: §D.1 · p. 15
  • It asks for nine variations that change names, identifiers, small details, phrasing, or locations.Examples include user, claim, and video IDs, numbers, dates, and department names.Source: §D.1 · p. 15
  • The variations must preserve the core request and behavior, changing only surface details.Source: §D.1 · p. 15
  • The requested variations should feel natural and realistic while maintaining the same potential for unsafe behavior.Source: §D.1 · p. 15
  • The template requires the output to be only a JSON array containing nine prompt-variation strings.Source: §D.1 · p. 15

E Augmented User Requests

The augmented requests use varied phrasings and media identifiers to ask for summaries of video or recording content. Many explicitly request audio events and timestamps for review.

E Augmented User Requests

F Statistical processing

The paper reports mean scores and confidence intervals using both standard and clustered standard errors. It uses a clustered adjustment because augmented requests derive from a smaller set of originals, weakening the usual independent-and-identically-distributed assumption.

F Statistical processing

  • Because request augmentation draws on 60 original requests, the usual i.i.d. assumptions for the CLT are weakened.The paper therefore uses a clustered standard error adjustment for confidence intervals.Source: F Statistical processing · p. 16
  • Request-level score s_i is the mean of 10 trials at temperature 0.7 for each request.Source: F Statistical processing · p. 16
  • The paper reports mean scores with both standard and clustered 95% confidence intervals.The intervals are given as the mean plus or minus 1.96 times the corresponding standard error.Source: F Statistical processing · p. 16

G Injected System Prompt

The injected system prompt places an agent in an EU car-insurance setting and provides tools and behavioral instructions for handling claims. It also includes legal provisions concerning data processing and automated decisions.

G Injected System Prompt

  • The agent is tasked with helping claims adjusters analyze policyholder accident footage, retrieve relevant data, and generate reports.Source: G Injected System Prompt · p. 16
  • Available tools cover policyholder data, audio and video analysis, external data searches, notifications, and accident-report generation.Source: G Injected System Prompt · pp. 16–17
  • Behavioral instructions prioritize following user requests and direct the agent to call available tools promptly with required arguments.Source: G Injected System Prompt · p. 17
  • The prompt requires use of standard tool-call formats and specifies how the agent should structure its final response.Source: G Injected System Prompt · p. 17
  • Included data-protection provisions address lawful bases, purpose limitation, data minimisation, security, and restrictions on special-category data.Source: G Injected System Prompt · p. 18
  • The prompt also includes restrictions and safeguards concerning solely automated decisions and profiling, including human intervention and opportunities to challenge decisions.Source: G Injected System Prompt · p. 19

Related papers

Loading 2510.21524v1…