Source-linked AI summary
Problematic Machine Behavior: A Systematic Literature Review of Algorithm Audits
Jack Bandy
TL;DR
Algorithm audits are increasingly important, but prior work had not been systematically synthesized to clarify its trajectory and future agenda. This paper conducts a PRISMA-guided review of over 500 papers, synthesizing 62 audits into a taxonomy of problematic behavior. The review finds real-world harms in public-facing systems and identifies underexamined priorities for future audits.
Problem
Prior algorithm-audit research had grown in commonality and public importance, but lacked a systematic synthesis of its past trajectory and future research agenda.
Method
The paper conducts a PRISMA-guided scoped review of over 500 papers and thematically analyzes 62 studies of public-facing algorithmic systems.
Results
The review identifies discrimination, distortion, exploitation, and misjudgement, with empirical evidence that public harms occur in real-world systems.
Takeaways & Limitations
Future audits should prioritize intersectional discrimination, advertising algorithms, and other underexamined behaviors, domains, methods, and organizations.
Takeaways & Limitations
The review may omit relevant work because it began with Scopus and included only academic papers, excluding books, journalism, and other mediums.
Abstract
from arXiv · showhide
While algorithm audits are growing rapidly in commonality and public importance, relatively little scholarly work has gone toward synthesizing prior work and strategizing future research in the area. This systematic literature review aims to do just that, following PRISMA guidelines in a review of over 500 English articles that yielded 62 algorithm audit studies. The studies are synthesized and organized primarily by behavior (discrimination, distortion, exploitation, and misjudgement), with codes also provided for domain (e.g. search, vision, advertising, etc.), organization (e.g. Google, Facebook, Amazon, etc.), and audit method (e.g. sock puppet, direct scrape, crowdsourcing, etc.). The review shows how previous audit studies have exposed public-facing algorithms exhibiting problematic behavior, such as search algorithms culpable of distortion and advertising algorithms culpable of discrimination. Based on the studies reviewed, it also suggests some behaviors (e.g. discrimination on the basis of intersectional identities), domains (e.g. advertising algorithms), methods (e.g. code auditing), and organizations (e.g. Twitter, TikTok, LinkedIn) that call for future audit attention. The paper concludes by offering the common ingredients of successful audits, and discussing algorithm auditing in the context of broader research working toward algorithmic justice.
1 INTRODUCTION
The paper addresses the limited synthesis of algorithm-audit research by reviewing prior studies of problematic behavior in public-facing systems. It organizes the evidence into a taxonomy and identifies priorities for future audits.
- Review purpose: The review screened over 500 papers and thematically analyzed 62 studies of public-facing algorithmic systems.It uses a scoped systematic literature review to clarify the field’s trajectory and future agenda.
- Review purpose: The taxonomy identifies four problematic behaviors: discrimination, distortion, exploitation, and misjudgement.These categories organize the review’s synthesis of prior algorithm audits.
- Review findings: 29 studies focused on distortion and 21 on discrimination, while search, advertising, and recommendation algorithms received the most attention.The review counts 25 search, 12 advertising, and 8 recommendation studies.
- Review findings: The reviewed audits provide empirical evidence that public harms of algorithmic systems occur in real-world systems rather than only hypothetical scenarios.The studies diagnose problematic behavior in public-facing systems affecting people in practice.
- Future agenda: Future audits should examine intersectional discrimination, advertising algorithms, and additional underexamined behaviors, domains, methods, and organizations.The review frames these priorities as areas needing further audit attention.
2 RELATED WORK
Related scholarship includes books, methodological reviews, ethics reviews, internal-audit frameworks, and domain-specific literature reviews. This paper extends that work through a broader review of public-facing algorithmic audits across domains.
- Related scholarship: Prior work has documented disparate impacts across algorithmic systems and reviewed methods for conducting algorithm audits.Related contributions include O’Neil’s examples of disparate impact and Sandvig et al.’s methodological review.
- Related scholarship: Ethics reviews identify challenges involving privacy, misuse, accuracy, validity, personal or group harm, subjective model design, and model misuse or misinterpretation.These challenges overlap with topics addressed by many algorithm audits.
- Scope distinction: Internal-auditing frameworks address organizational algorithm development, whereas this review focuses on public-facing algorithms that have already been deployed.The distinction concerns audit setting and stage of system development.
- Scope distinction: A sharing-economy review identified nine algorithm-auditing papers, while this review expands beyond sharing-economy platforms to systems such as search and recommendation.The broader scope supports cross-domain synthesis.
- Research questions: The paper asks what problematic machine behaviors previous audits diagnosed and what future audits should examine.These questions are formalized as RQ1 and RQ2.
3 METHODS
The study uses a PRISMA-guided scoped review to identify, screen, and synthesize empirical audits of public algorithmic systems. It searches broadly, supplements database results, and codes included studies by audit-relevant categories.
- Review design: The review follows PRISMA’s identification, screening, eligibility, and inclusion stages to survey a broad research field.The scoped-review design prioritizes an overview rather than detailed answers to narrow questions.
- Review design: An included audit must empirically investigate a public algorithmic system for potentially problematic behavior.The definition covers commercial and other public settings and requires evidence-based claims with defined outcome metrics.
- Search strategy: The search used iterative keywords and Scopus’s interdisciplinary database, including fields such as economics, journalism, and law.The search was designed to capture relevant work beyond computing alone.
- Search strategy: Additional sources added 36 potentially relevant studies and helped reduce keyword-search bias, including possible recency bias against studies before 2016.Sources came from full-text screening and peer-review recommendations.
- Screening and inclusion: The screening process examined 503 records, narrowed them to 123 full-text papers, and produced 62 included studies after exclusions and unavailable full texts.Exclusions included theory or methods papers, non-public systems, and development-history studies.
- Data extraction: The analysis coded year, domain, organization, and audit method, including code audits, direct scrapes, sock puppets, carrier puppets, and crowdsourcing.These categories structure the thematic analysis of included audits.
4 RESULTS FOR RQ1: PREVIOUS ALGORITHM AUDITS
The review organizes prior algorithm-audit findings around four types of problematic machine behavior: discrimination, distortion, exploitation, and misjudgement.
- Behavior taxonomy: The results for previous audits are organized by discrimination, distortion, exploitation, and misjudgement.These four categories define the review’s behavior-based structure.
4.1 Overview
The review organizes problematic machine behavior into four types and finds that distortion received more attention than discrimination overall, especially in search and recommender systems.
- The review identifies four types of problematic machine behavior: discrimination, distortion, exploitation, and misjudgement.
- Discrimination remained a central focus of the reviewed algorithm audits.
- 21 studies audited discrimination, while 29 focused on distortion.
- Distortion audits particularly examined search algorithms and recommender systems for political partisanship, false information, and source suppression.
4.2 Discrimination
The review finds discrimination across advertising, vision, search, and pricing algorithms, affecting allocation, representation, and performance across demographic and intersectional identities.
- Algorithmic discrimination can involve disparate treatment or impact across race, age, sex, gender, location, socioeconomic status, and intersectional identity.
- Advertising: Advertising audits found discrimination in targeting, housing advertisements, career ads, and representational associations between names and arrest-related ads.
- Vision: Vision audits found disparities by sex, race, geography, and intersectional identity, often associated with imbalanced training data.
- Vision: Intersectional evaluation revealed a Face++ error rate of 0.8% for lighter-skinned males versus 34.5% for darker-skinned females.
- Search: Search audits found sexualized results for “black girls,” exaggerated occupational gender stereotypes, and lower rankings for workers perceived to be Black.
- Search: Fairness-aware LinkedIn search produced gender-representative results for 95% of queries in A/B testing.
- Pricing: Pricing audits documented different prices for the same item and examined dynamic pricing as a source of inequality and consumer confusion.
4.3 Distortion
Distortion occurs when algorithms present media in ways that obscure underlying reality, especially through search and recommendation systems. Audits found small partisan leanings, source concentration, occasional personalization effects, and misinformation-related “rabbit holes,” while substantial echo-chamber effects were often absent.
- 4.3 Distortion: Distortion describes algorithms presenting media in ways that obscure an underlying reality, including partisanship, misinformation, hyper-personalization, and echo chambers.The review focuses especially on search and recommendation algorithms.
- 4.3.1 Distortion in Search: Search audits found small but statistically significant partisan leanings, although studies reported conflicting results across result components and query comparisons.Some audits found left-leaning news results, while others found slight rightward shifts or virtually no difference.
- 4.3.1 Distortion in Search: Search-result snippets amplified the partisanship of original webpages across query topics, political orientations, and webpage types.The effect appeared in snippets below links, including social-media, news, and sports pages.
- 4.3.1 Distortion in Search: Evidence for echo chambers was generally limited: personalization effects were often small or absent, though one study found up to 20% of relevant results removed.A YouTube audit also found a “rabbit hole” effect in which misinformation-related searches returned more such videos.
- 4.3.2 Distortion in Recommendation: Search and recommendation audits found source concentration, including 45% of Apple News recommendations from three sources and 35% of Google News stories from three sources.These findings indicate prominence was concentrated among a small number of media sources.
4.4 Exploitation
Exploitation concerns the inappropriate use of outside content or sensitive personal information. Audits found extensive reliance on user-generated content and examples of advertising systems using sensitive information without transparent consent.
- 4.4 Exploitation: Exploitation involves inappropriate use of outside content or sensitive personal information from people.Unlike discrimination and distortion, exploitation can target either people or media.
- 4.4.1 Exploitation in Search: Search engines relied heavily on user-generated and journalistic content, raising concerns involving copyright and monopoly power.Audits treated this dependence as a potential form of exploitation of outside content.
- 4.4.1 Exploitation in Search: Wikipedia content appeared on 81% of Google results pages for queries trending in Google Trends and on 90% of results pages for another reported query set.The audit linked this prevalence to Google’s reliance on voluntarily produced user-generated content.
- 4.4.2 Exploitation in Advertising: Advertising audits found systems using sensitive browsing and inferred-interest information for targeting without users’ clear awareness or consent.Examples included substance-abuse browsing histories and categories such as homosexuality, Judaism, and tobacco.
4.5 Misjudgement
Misjudgement refers to incorrect algorithmic predictions or classifications and may precede discrimination, distortion, or exploitation. Reviewed studies clustered in criminal justice and advertising, documenting accuracy, fairness, data-dropping, and user-inference problems.
- 4.5 Misjudgement: Misjudgement means an algorithm makes incorrect predictions or classifications, sometimes producing later discrimination, distortion, or exploitation.Seven reviewed studies focused on misjudgement, clustered in criminal justice and advertising.
- 4.5.1 Misjudgement in Criminal Justice: Criminal-justice audits found trade-offs between accuracy and fairness, and a small code change in a forensic DNA system caused substantial data-dropping and less accurate results.One case found machine learning more accurate but less fair than statistical models under demographic parity and error-rate balance.
- 4.5.1 Misjudgement in Criminal Justice: Criminal-justice algorithms were characterized as posing alarming potential for harm, with the review citing calls for holistic reform or abolition rather than incremental auditing alone.This boundary was stated specifically for criminal-justice applications.
- 4.5.2 Misjudgement in Advertising: Google correctly inferred age for 17% of logged-out females and 6% of logged-out males across the web.The audit concerned demographic inference in targeted advertising systems.
- 4.5.2 Misjudgement in Advertising: Only 27% of surveyed advertising interests were rated strongly relevant, while 40% of third-party attributes were reported as not at all accurate.These findings indicate substantial inaccuracies in demographic and interest-based user profiles.
5 RESULTS FOR RQ2: FUTURE ALGORITHM AUDITS
The review identifies under-explored behaviors, domains, and systems that warrant future algorithm audits, especially intersectional discrimination, language and recommendation systems, and exploitation of personal information.
- Discrimination: Future audits should examine discrimination involving intersectional identities rather than treating race, age, sex, or gender as isolated categories.The review specifically suggests auditing dynamic pricing for allocative harms involving intersecting identities.
- Discrimination: Language-processing and recommendation algorithms received negligible discrimination-audit attention despite evidence of racial discrimination and gendered recommendation patterns.One Spotify audit found approximately 8 out of 10 recommended artists were male.
- Discrimination: Criminal-justice risk-assessment audits remain sparse, although one reviewed study found models were twice as likely to misclassify people not from Spain.The review found only one discrimination audit of risk-assessment algorithms, while three other studies examined general misjudgement.
- Distortion: Future distortion audits should expand beyond established search and recommendation cases to advertising, language systems, emerging platforms, and personalized maps.Examples include Facebook political-ad exposure, partisan Google snippets, TikTok, Google Discover, and border representations in digital maps.
- Exploitation: Audits of exploitation should clarify how companies derive economic value from personal information and other content.One study found only 23% of participants gave a close estimate of their personal data’s actual value, about $1 per month.
5.4 Remaining Work: Misjudgement
The review recommends more specific harm analyses for misjudgement and highlights methodological, organizational, replication, and design priorities for future audits.
- Misjudgement: Future misjudgement audits should connect inaccurate algorithmic outputs to specific harms affecting users, advertisers, and other stakeholders.Examples include discomfort with platform profiling and advertisers paying for inaccurately targeted advertisements.
- Methods: Direct scraping was the most-used method, with 30 of 62 reviewed studies, but its lack of randomization or manipulation limits it mainly to descriptive tasks.Crowdsourcing did not share this limitation in the reviewed audits.
- Methods: Code auditing remains under-explored because many high-impact algorithms are proprietary and opaque, although open-source systems such as Wikipedia and DuckDuckGo may be suitable targets.Researchers will generally need other methods for systems from Facebook, Google, Amazon, and similar corporations.
- Organizations: Audit attention is concentrated on Google, with 30 studies, while influential social platforms including Twitter, Instagram, and LinkedIn remain comparatively underexamined.Legal challenges to scraping may partly explain this imbalance.
- Replication: Replication is important because frequent algorithm updates, alternative methods, and different organizations can change audit findings.Open datasets and scraping tools can help enable replication audits.
- Audit design: Successful audits commonly require a public-facing system, suspected problematic behavior, a compelling baseline, and metrics that quantify the behavior.The review also warns that flexible metric selection can invite p-hacking or fairness gerrymandering.
6 DISCUSSION
The review identifies scope and positionality limitations while situating algorithm audits within broader justice-oriented research. It also highlights complementary audit approaches, including user studies, development histories, non-public audits, and small-scale cases.
- Limitations: Scopus, keywords, English-only inclusion, and academic-publication restrictions may have excluded relevant audits, including older and non-English work.Many additional-source papers were published in or before 2015, suggesting recency bias in the initial search.
- Limitations: The author’s positionality as an English-speaking, United States-based cisgender white male may limit coverage of oppression and violence in algorithmic systems.The paper explicitly connects this standpoint to privilege within multiple social systems and structures.
- Algorithmic justice: Algorithm auditing is presented as one diagnostic tool within broader collective work toward algorithmic justice.The paper frames audits as clarifying and potentially improving the relationship between technology and society.
- Complementary approaches: User studies and algorithm-development histories can complement audits by revealing problematic behavior and providing socio-technical or historical context.The review describes surveys and interviews as indirect methods and development histories as resources for selecting behaviors or organizations to audit.
- Complementary approaches: Non-public audits can characterize potential problems even though this review focused on public-facing algorithmic systems.Examples include age-related discrimination in sentiment analysis and landmark detection, plus gender discrimination in semantic representations used for hiring.
- Complementary approaches: Small-scale audits can produce powerful evidence and prompt publicity, corporate response, justice-oriented changes, or larger studies.The review cites image-tagging, one-search-term, and single-screenshot cases as examples of small-scale audits exposing representational harms.
7 CONCLUSION
This review synthesizes 62 studies of public-facing algorithmic systems and identifies recurring forms of problematic machine behavior. It documents harms including advertising discrimination and search distortion, showing that these harms are empirically observable rather than merely theoretical.
- Conclusion: The review synthesizes 62 studies of public-facing algorithmic systems and identifies recurring types of problematic machine behavior.Its synthesis organizes findings around behaviors such as discrimination and distortion.
- Conclusion: Audits have diagnosed discrimination in advertising algorithms and distortion in search algorithms.These findings provide empirical evidence that public harms of algorithmic systems are not merely theoretical conjectures.