PaperlayerView exact revision on arXiv

Source-linked AI summary

Demystifying the Draft EU Artificial Intelligence Act

Michael Veale, Frederik Zuiderveen Borgesius

arXiv:2107.03721v4cs.CYcs.AI

TL;DR

The paper addresses how the Draft AI Act’s risk-based, product-safety-inspired framework affects AI regulation and national policy space. It contextualises and critiques the Commission proposal, finding that some rules make sense but other provisions may be ineffective, with enforcement and maximum harmonisation presenting significant concerns.

  • Problem

    The Draft AI Act is complex and combines internal-market harmonisation with broad fundamental-rights concerns, creating important implications for Member State action.

  • Method

    The paper contextualises and critiques the initial Commission proposal using scholarship on AI practices and EU product-safety regimes.

  • Results

    The paper finds that some Draft AI Act provisions have surprising legal implications or may be ineffective, while its enforcement regime and maximum harmonisation raise significant concerns.

  • Takeaways & Limitations

    The authors argue that enforcement and the effects of maximum harmonisation should be addressed as priorities during the legislative process.

  • Takeaways & Limitations

    The paper does not cover all facets of the Draft AI Act, and many aspects require further scrutiny.

Abstract

from arXiv · show

In April 2021, the European Commission proposed a Regulation on Artificial Intelligence, known as the AI Act. We present an overview of the Act and analyse its implications, drawing on scholarship ranging from the study of contemporary AI practices to the structure of EU product safety regimes over the last four decades. Aspects of the AI Act, such as different rules for different risk-levels of AI, make sense. But we also find that some provisions of the Draft AI Act have surprising legal implications, whilst others may be largely ineffective at achieving their stated goals. Several overarching aspects, including the enforcement regime and the risks of maximum harmonisation pre-empting legitimate national AI policy, engender significant concern. These issues should be addressed as a priority in the legislative process.

1. Manipulative Systems

The Draft AI Act prohibits two forms of manipulative AI, but its examples and conditions reveal uncertainty about the provisions’ practical scope and effectiveness.

  • 1. Manipulative Systems: The proposal’s AI regulation is part of a broader package of EU digital, machinery, data-governance, and liability initiatives.
  • 1. Manipulative Systems: The Draft AI Act identifies manipulation as a prohibited category of AI practice.The proposal contains two manipulation prohibitions, illustrated by systems using subliminal techniques or exploiting vulnerabilities related to age or disability.
  • 1. Manipulative Systems: The Commission’s examples involve inaudible sounds influencing truck drivers and voice assistants encouraging minors toward dangerous behaviour.
  • 1. Manipulative Systems: The manipulation provisions require intent, specified vulnerabilities or subliminal techniques, and physical or psychological harm.The final trigger is whether the activity causes or is likely to cause harm, rather than whether the manipulator’s own ends are furthered.

a) The Harm Requirement

The harm requirement narrows the manipulation prohibitions and creates difficulties for cumulative, system-level, and downstream harms.

  • a) The Harm Requirement: Manipulative AI systems may be permitted when they are unlikely to cause harm to an individual.The authors identify this individualised harm requirement as creating problematic loopholes.
  • a) The Harm Requirement: Cumulative harms may develop over time without a single event exceeding the required seriousness threshold.
  • a) The Harm Requirement: The Draft AI Act excludes harms arising from user-base dynamics, including discriminatory ratings or recommendations on dating apps and online markets.
  • a) The Harm Requirement: Separate downstream actors may use an AI system’s person-classification output harmfully, creating uncertainty about which actor the prohibition should regulate.The authors describe this as a difficult-to-govern dual-use artefact.

b) Comparison to Existing Union Law

The Draft AI Act extends beyond existing Union law in some respects, but its harm requirements limit that expansion and may leave the prohibitions with little practical impact.

  • b) Comparison to Existing Union Law: The Unfair Commercial Practices Directive already prohibits practices materially distorting, or likely to distort, consumers’ economic behaviour.
  • b) Comparison to Existing Union Law: The Draft AI Act’s expansion beyond the Unfair Commercial Practices Directive to non-economic decision-making is limited by its harm requirements.
  • b) Comparison to Existing Union Law: Flexible ‘reasonable person’ requirements are presented as workable alternatives to harm tests in information law.
  • b) Comparison to Existing Union Law: The Draft AI Act also prohibits selling in-scope manipulative systems, but vendors may evade this through general-purpose systems configurable by users.
  • b) Comparison to Existing Union Law: The authors conclude that the prohibitions concerning manipulative AI systems may have little practical impact.

a) Scope of Same Context

The Draft AI Act’s social-scoring and biometric provisions leave important questions about context, liability, scope, and enforcement unresolved.

  • a) Scope of Same Context: The same-context exemption appears designed to exclude reputation systems but is difficult to operationalise.
  • a) Scope of Same Context: It is unclear whether citizen scoring in the datafied welfare state falls within the prohibition when private-sector data augment administrative data.A narrow context view limits scoring to public-authority interactions, while a wider view may connect credit-card records and welfare support.
  • a) Scope of Same Context: Public-sector social-media background checks and public-sector use of LinkedIn freelancer rankings raise further questions about contextual disconnect and provider liability.
  • a) Scope of Same Context: Vendors and users may each deny that scoring led to negative outcomes, potentially leaving no entity clearly liable.
  • a) Scope of Same Context: The biometric prohibition covers only certain real-time uses and permits Member States to authorise listed exceptions with safeguards.
  • a) Scope of Same Context: Post-event biometric analysis, online biometric identification, and non-law-enforcement uses are excluded or treated differently under the proposal.

b) Need for Pre-Authorisation of “Individual Use”

The Draft AI Act requires prior authorisation for individual uses of certain biometric systems, but leaves the scope and transparency of that requirement unclear. Its framework may permit infrastructure capable of population-scale surveillance despite formal restrictions.

  • Authorisation framework: Authorisation must come from a judicial or independent administrative authority, with emergency use authorised shortly afterwards.The relevant body must adopt a neutral stance, excluding public prosecutors according to analogous CJEU case-law.
  • Scope and transparency: The meaning of “individual use” is unclear, including whether one authorisation could cover broad purposes such as identifying people on a missing-children list.The Act also does not explicitly require transparency about the number and type of authorisations issued.
  • Scope and transparency: Any authorisation of biometrics requires re-purposable infrastructure, prompting calls from European data-protection authorities and NGOs for a general ban.Critics argue that the proposal could legitimise rather than prohibit population-scale surveillance.
  • High-risk scope: High-risk AI systems cover defined products and applications, including biometric identification, critical infrastructure, education, employment, essential services, and migration.The Commission may add sub-areas posing similar risks but cannot add entirely new areas.

3. Essential Requirements and Obligations

The Draft AI Act imposes extensive provider obligations for high-risk systems, covering risk management, data quality, documentation, logging, accuracy, robustness, cybersecurity, and human oversight. However, important obligations and protections remain unevenly distributed or potentially weak in practice.

  • Provider obligations: Providers of high-risk AI systems must establish quality-management and documented risk-management systems maintained throughout the system’s lifetime.The framework connects essential requirements to obligations imposed mainly on providers.
  • Provider obligations: Training datasets must be sufficiently relevant, representative, accurate, complete, and appropriate for the system’s intended purpose.The text notes that datasets need not meet these requirements absolutely, but sufficiently and in view of intended use.
  • Discrimination and data: The Act provides a sensitive-data exemption for bias detection, but only providers of high-risk systems may use it.It does not allow upstream data brokers to collect or sell such data for high-risk providers.
  • Technical controls: Providers must address accuracy, robustness, cybersecurity, traceability, technical documentation, and human oversight, including two-person sign-off for biometric identification.Biometric logs must record use periods, reference databases, matching inputs, and the identities of the two checking persons.
  • Human oversight: Human-oversight obligations do not flow directly to users, who may only need to follow provider instructions even when those instructions require limited oversight.The proposal instead emphasises the user’s discretion in organising resources and activities for oversight.
  • Standards and compliance: Harmonised standards may become the practical rule-making mechanism because providers choosing the essential-requirements route must still take relevant standards into account.The paper argues that standards are cheaper and safer compliance references than interpreting essential requirements independently.

b) Controversies of Harmonised Standards

The Draft AI Act relies heavily on harmonised standards developed by private standardisation organisations, raising concerns about democratic accountability, stakeholder participation, and the constitutional status of delegated rule-making.

  • Participation and accountability: Under-resourced consumer organisations may struggle to participate in technically arcane standardisation processes even though Member States must recognise the resulting standards.The paper questions whether existing efforts adequately include stakeholders where rights and freedoms are at stake.
  • Participation and accountability: The European Parliament has no binding veto over harmonised standards mandated by the Commission.The paper links this absence of veto to broader concerns about private outsourcing of complex regulatory negotiations.
  • Constitutional concerns: Private standardisation is controversial because technical standards make value-laden choices about acceptable risk under uncertainty.The Act’s incorporation of broad fundamental-rights issues into the product-safety framework intensifies this legitimacy concern.
  • Constitutional concerns: The model may face constitutional challenges if private standardisation bodies exercising de facto rule-making power cannot remain free from judicial scrutiny.The paper describes this as a tension between the NLF’s constitutional reliance on private bodies and emerging CJEU scrutiny.
  • Practical role of standards: CEN and CENELEC can develop standards that AI providers will follow in practice, while notified bodies have few mandatory roles.The Commission may alternatively issue common specifications, but providers departing from them must justify equivalence.

c) Self-Assessment and the (Limited) Role of Notified Bodies

Most high-risk AI providers can rely on self-assessment, leaving AI Act-specific notified bodies with a narrow and potentially temporary role. This may also weaken regulatory knowledge flows and complicate responsibility for bot disclosure.

  • Conformity assessment: For most standalone high-risk systems, providers can declare conformity through self-assessment without AI Act-specific notified bodies.AI Act-specific notified bodies are required initially only for listed biometric identification and categorisation applications.
  • Conformity assessment: If harmonised standards or common specifications for biometric systems exist, even those systems may use self-assessment alone.Because the Commission expects standards before the Regulation applies, AI Act-specific notified bodies may never be required for biometrics.
  • Regulatory intermediaries: Notified bodies can translate rules, provide regulatory know-how, and give feedback to regulators and standard-setters.The Draft AI Act requires them to participate in coordination activities.
  • Regulatory intermediaries: If AI Act-specific notified bodies never exist for non-biometric applications, gaps may remain in knowledge flows about how the Act operates in practice.The paper characterises the coordination obligation as potentially futile in that situation.
  • Bot disclosure: The Act’s transparency regime requires bot providers to disclose bot interaction unless an exception applies, while liability flows to providers rather than users or platforms.Market-surveillance authorities can compel intermediaries facilitating sales, but not clearly communication platforms.
  • Bot disclosure: The provider-user-speaker distinction can collapse when configurable systems such as GPT-3 are used to generate content for publication.The paper questions how disclosure duties should apply when APIs expose raw models to users.

2. Emotion Recognition and Biometric Categorisation Disclosure

The Draft AI Act requires disclosure when emotion-recognition or biometric-categorisation systems operate, but the provision may add little to existing data-protection duties. The authors warn that focusing on transparency risks legitimising practices with weak scientific foundations and potentially unjust consequences.

  • Users must inform people exposed to emotion-recognition or biometric-categorisation systems, except where legally permitted biometric categorisation is used for crime prevention.
  • The disclosure rule may add little because data-protection law already requires information about personal-data processing, including its existence and purposes.
  • If these systems are treated as not processing personal data, the Act could implicitly legitimise a contentious and restrictive interpretation of the GDPR.
  • Transparency-focused regulation risks legitimising emotion and biometric categorisation despite little-to-no scientific basis and potentially unjust societal consequences.
  • Research challenges confident inference of emotions from facial movements because current systems use oversimplified taxonomies and assume universality across cultures and contexts.

3. Synthetic Content (‘Deep Fake’) Disclosure

The Draft AI Act requires users to disclose when generated or manipulated content that resembles existing entities or events would falsely appear authentic. The authors question the provision’s target, scope, allocation of responsibility, and enforceability.

  • Users must disclose the artificial nature of synthetic image, audio, or video content that appreciably resembles existing entities or events and appears authentic.
  • The provision may only partially assist subjects harmed by convincing likenesses, while EU law already addresses many misleading commercial practices within its scope.
  • Synthetic outputs such as enhanced images or reconstructed 3D models may be mistaken for reliable measurements, yet disclosure responsibility falls on users rather than providers.
  • The provision may be too broad if intended to regulate disinformation because it overlooks differences between media and includes tensions illustrated by its examples.
  • Enforcement is difficult because authorities may need to identify undisclosed deepfakes, investigate professional platform users, and apply specialised forensic expertise.

V. Harmonisation and Pre-Emption

The Draft AI Act’s maximum-harmonisation effects may restrict Member States across a material scope covering all AI systems, even though substantive obligations primarily target high-risk systems. This creates concern about pre-emption of national AI policy.

  • The Act’s maximum harmonisation would disable conflicting national rules and require Member States to accept compliant products on their markets.
  • The Act unusually targets substantive obligations mainly at high-risk systems while defining its material scope across all AI systems.
  • 1. Marketing: Marketing of all AI systems, not only high-risk systems, is fully harmonised, leaving Member States limited exceptions for additional restrictions such as carbon-footprint or accessibility rules.
  • a) Material Scope: The Act’s material scope covers use of AI systems and appears to prevent Member States from imposing use restrictions unless the Regulation explicitly authorises them.
  • a) Material Scope: Even partial harmonisation can expose national use restrictions to company challenges under EU free-movement law.
  • a) Material Scope: The proposed framework may create an arbitrary divide between regulated high-risk systems and non-high-risk systems that Member States are effectively forbidden or vulnerable to challenge for regulating.

a) Nor Rights for AI-System-Subjects

The Draft AI Act gives affected individuals and communities no direct complaint or legal action rights comparable to those available under data-protection law. Its product-oriented enforcement model also assigns market-surveillance authorities tasks unlike ordinary product regulation.

  • Affected individuals cannot complain to a market-surveillance authority or obtain a direct legal right to sue providers or users for Draft AI Act failures.
  • Consumer groups and other collectives lack representative-complaint rights, while authorities need only handle complaints competently and consider them as information sources.
  • Without affected groups able to challenge regulators, the enforcement regime lacks bottom-up pressure against weak enforcement.
  • b) Incoherence of the Enforcement System: The Act applies NLF-style enforcement beyond its NLF-style high-risk regime, bringing users within market-surveillance powers through an expansive interpretation.
  • b) Incoherence of the Enforcement System: Market-surveillance authorities would investigate synthetic content, manipulative practices, and digital welfare systems despite lacking the institutional guarantees and product-regulation fit described here.

2. Database of Standalone High-Risk AI Systems

The Draft AI Act proposes a Commission-managed database for standalone high-risk AI systems, intended to help surveillance and accountability. However, unclear complaint rights, possible trade-secret litigation, and complex provider obligations may limit its effectiveness.

  • Database design: The Commission would manage a central database registering standalone high-risk AI systems, modelled on medical-device registration systems.The proposal also requires relevant providers to upload electronic instructions for use, subject to specified exceptions.
  • Accountability: The database could help market surveillance authorities locate illicit AI systems more easily.Its public availability is also intended to help civil society and journalists uncover illicit AI.
  • Enforcement: The database’s bottom-up enforcement role is significantly weakened by the absence of clear complaint rights.This limits the ability of affected people and other actors to use the database to support enforcement.
  • Accountability: Companies that develop high-risk AI systems for their own use would become both providers and users, requiring public declaration and uploaded instructions.This may support accountability beyond the Act’s requirements, although firms may challenge disclosure obligations on trade-secret grounds.
  • Provider obligations: Users who substantially modify systems or use them off-label become providers and must publicly declare that status.Continued learning within parameters predetermined by the provider is not treated as substantial modification.

VII. Concluding Remarks

The paper concludes that the Draft AI Act combines sensible risk-based elements with serious weaknesses in its legal architecture and enforcement. It warns that maximum harmonisation may restrict legitimate national action and potentially lower, rather than raise, regulatory protection.

  • Strengths: The Draft AI Act is a world-first horizontal attempt to regulate AI systems through risk-based requirements, prohibitions, and a public database.The authors identify these as sensible elements of the proposal.
  • Weaknesses: The Act’s patchwork of product safety, fundamental rights, surveillance, and consumer protection law does not make it comprehensive or watertight.The authors argue that the interaction among these legal components may leave the instrument difficult to understand and weak in impact.
  • Weaknesses: The high-risk regime relies on standardisation bodies without fundamental-rights experience to write practical rules that providers may quietly self-assess against.The paper also criticises transparency provisions and a product-safety-based enforcement mechanism.
  • Enforcement: Affected communities receive no mechanism for complaint or judicial redress under the proposed enforcement framework.This is presented as a significant weakness alongside the regime’s regulation of AI users.
  • National policy space: Maximum harmonisation may restrict legitimate national responses to AI’s social impacts and disapply existing national digital fundamental-rights protections.The paper also warns that it could block future regulation of carbon emissions or restrictions on systems outside the Act’s high-risk category.
  • Conclusion: The paper cautions that the Draft AI Act may contribute to deregulation more than it raises the regulatory bar.The authors stress that many aspects remain omitted and require further scrutiny during the legislative process.
Loading 2107.03721v4…