Source-linked AI summary
Preventing Repeated Real World AI Failures by Cataloging Incidents: The AI Incident Database
Sean McGregor
TL;DR
AI practitioners lack a shared way to learn from real-world harms caused by intelligent systems, creating a need for an incident collection. The paper presents the AI Incident Database as extensible infrastructure for discovering, documenting, and studying such incidents, with early collaboration requests indicating strong adoption interest. Its multi-stakeholder governance makes prescriptive norms difficult to promulgate.
Problem
AI systems cause real-world harms, but the AI community lacks formal systems for practitioners to discover and learn from past mistakes.
Method
The paper introduces the AI Incident Database, combining searchable incident reports, user-oriented discovery, and extensible infrastructure for research and development uses.
Results
The AIID archives more than 1,000 public incident reports and supports product, risk, engineering, and research use cases.
Takeaways & Limitations
Shareable, searchable incident records help practitioners discover past harms and develop requirements, mitigations, tests, and research around them.
Takeaways & Limitations
Because the AIID is governed by corporate and nonprofit civil-society partners, prescriptive norms are difficult to promulgate.
Abstract
from arXiv · showhide
Mature industrial sectors (e.g., aviation) collect their real world failures in incident databases to inform safety improvements. Intelligent systems currently cause real world harms without a collective memory of their failings. As a result, companies repeatedly make the same mistakes in the design, development, and deployment of intelligent systems. A collection of intelligent system failures experienced in the real world (i.e., incidents) is needed to ensure intelligent systems benefit people and society. The AI Incident Database is an incident collection initiated by an industrial/non-profit cooperative to enable AI incident avoidance and mitigation. The database supports a variety of research and development use cases with faceted and full text search on more than 1,000 incident reports archived to date.
1 Introduction
Intelligent systems are increasingly deployed in safety-critical and socially challenging domains, yet practitioners lack formal systems for learning from real-world failures. The AI Incident Database addresses this gap by systematizing incidents involving safety, fairness, and other real-world problems.
- Intelligent systems are increasingly deployed in transportation, law enforcement, recruiting, and other consequential domains.
- Failures in these systems can pose serious risks to life and wellbeing.
- The AI Incident Database systematizes incidents in which intelligent systems caused safety, fairness, or other real-world problems.
- The work contributes industry infrastructure, a dataset of more than 1 thousand incident reports, and an extensible architecture for research products.
2 Other Incident Databases
Aviation and cybersecurity incident databases show how shared records can support safety and security across established operational contexts. The AIID draws on these models while emphasizing extensibility for AI’s broader and less predictable applications.
- Aviation improves safety by systematically analyzing and archiving past accidents and incidents within a shared database.Decades of iterative improvements decreased fatalities 81 fold since 1970 when normalized for passenger miles.
- Unlike aviation’s well-defined operational context, AI systems span unforeseen contexts, technologies, and failures.
- The AIID architecture is inspired by aviation incident databases but places greater emphasis on extensibility.
- The CVE system provides cross-industry security infrastructure through publicly disclosed vulnerabilities linked by consistent identifiers.Other systems use these identifiers for taxonomies, research, and more secure software development.
3 The AI Incident Database
The AIID links multiple public reports to citeable AI incidents and supports discovery by product managers, risk officers, engineers, and researchers. These uses connect incident knowledge to requirements, mitigation practices, testing, and population-level study.
- The AIID indexes more than 1,000 public incident reports, joining multiple reports about one incident under a single identifier.Incident number 3 contains 18 reports on the Boeing 737 MAX 8 crashes.
- Sampling multiple reports provides differing viewpoints and broader word coverage for discovering incidents relevant to users’ needs.
- Incident numbers make AI incidents citeable, crediting the first submitter while leaving incidents intentionally untitled as descriptions evolve.
- Product managers can use discovered incidents to introduce technological, marketing, and content moderation requirements for new products.The example concerns recommender systems for children and inappropriate content recommendations.
- Risk officers can search reports to identify mitigation practices when mistakes cannot currently be prevented technologically.The example recommends clearly indicating that text is machine translated.
- Researchers can search and cite incidents to study changes over time and connect publications with potentially mitigable harms.A search for “policing” returned 14 distinct incidents.
- Shareable database entries help users communicate that mitigation is necessary by making bad outcomes difficult to ignore.
System Architecture
The AIID combines web applications with a MongoDB document database and extensible taxonomy infrastructure. Its architecture supports multiple viewpoints, searchable discovery, submissions, faceted filtering, and periodically rendered data products.
- The AIID is a multi-stakeholder PAI project governed by corporate and nonprofit civil-society partners.Its infrastructure joins disparate viewpoints without providing top-down analysis or narrative.
- The system uses web applications connected to MongoDB for storing incident report text and metadata.The Discover application provides instant searches, while Submit accepts links to public incident reports.
- As the database grows, submission tooling must resolve new reports to existing incidents created earlier.
- Partners can build interoperable taxonomies, documentation, and summaries using namespaced tags that support faceted filtering.
- The AIID periodically pre-renders database views as static web applications to avoid repeatedly querying the complete database.
4 Conclusion
The AIID’s extensible architecture is expected to improve coverage of AI incidents over time and reduce negative real-world consequences. Its purpose reflects the importance of retaining knowledge of past failures rather than repeating them.
- The authors expect the extensible architecture to provide pragmatic coverage of AI incidents through time while reducing negative real-world consequences.
- The Discover application supports instantaneous facial-recognition searches, returning 89 reports with matching text snippets and metadata facets.Users can filter by publications, authors, submitters, and incident numbers.
- The paper frames retaining knowledge of past failures as important because those who cannot remember the past are condemned to repeat it.