The Flaws of Policies Requiring Human Oversight of Government Algorithms

1. Introduction

• Human oversight became a centerpiece of global regulation because policymakers seek algorithmic benefits while using human review to guard against mistaken or biased decisions.Governments have also faced concerns that algorithms can contain errors and biases and fail to respond to individual circumstances.Source: §1 · pp. 1–2
• Here, human oversight means human judgment when an algorithm makes a specific prediction or decision, rather than broader democratic or institutional governance of algorithms.Policies generally allow algorithmic assistance but require a human to oversee or control the final decision, intending human quality control and protection of rights and dignity.Source: §1 · p. 2
• Reviewing 41 policy documents, the article finds human-oversight mandates lack empirical support for reliable oversight and legitimize flawed, unaccountable algorithms while shifting accountability to lower-level operators.The article reports that the vast majority of research suggests people cannot reliably perform the desired oversight functions.Source: §1 · p. 2
• The proposed alternative is not to abandon human oversight or prohibit government algorithms, but to develop governance that more rigorously addresses the concerns prompting oversight policies.Source: §1 · p. 2
• Institutional oversight would require agencies to justify high-stakes algorithm adoption and empirically support proposed human oversight, then publish the justification for public review and approval before use.Agencies should not use algorithms that violate fundamental rights, are poorly suited to the decision process, or are untrustworthy; insufficient evidence of oversight effectiveness or improved human decision-making also bars adoption.Source: §1 · p. 3

• Street-level bureaucrats make consequential decisions in courts, police departments, schools, welfare agencies, and social services, where policymakers strongly call for human oversight of algorithms.Examples include judges using risk assessments, child services workers using predictive models, and welfare agencies determining benefit eligibility.Source: §2 · p. 3
• Street-level bureaucracies face an “essential paradox”: rules promote consistency, while discretion enables flexibility, and no objective solution settles their balance (Lipsky, 2010; Wilson, 2000).Following prespecified rules to treat everyone equally conflicts with responding to the circumstances of individual cases.Source: §2 · p. 3
• Discretion worries publics and policymakers because it can invite biased or arbitrary treatment, yet it helps officials adapt ambiguous policies to complex cases and individual needs (Zacka, 2017; Lipsky, 2010).Restricting discretion can impose rigid logic on uncertain situations and prevent adaptation to complex or novel circumstances.Source: §2 · p. 3
• Algorithms attract governments with promises of greater accuracy, objectivity, and consistency, but government systems have shown high error rates and bias against women, minorities, and low-income individuals (Kleinberg et al., 2018; Angwin et al., 2016).Agencies also adopt algorithms to cut costs and enhance efficiency; evidence suggests they can make policy-relevant predictions more accurately, fairly, and consistently than public servants.Source: §2 · pp. 3–4
• Algorithmic rules cannot reflexively adapt to novel or marginal circumstances, raising concerns about reduced bureaucratic discretion, dehumanizing decisions, and threats to due process (Alkhatib & Bernstein, 2019; Citron, 2008).People may lack notice or a meaningful chance to inspect and challenge decisions; individual justice can require human judgment for ethical and contextual analysis.Source: §2 · p. 4
• Human oversight policies present a compromise: keeping a person in the decision-making loop aims to retain algorithmic benefits while guarding against errors, bias, and lost individualized judgment.The proposed compromise also aims to maintain discretion, expertise, and due process in street-level bureaucracies and administrative agencies.Source: §2 · p. 4

3. Survey of human oversight policies

• The author collected 41 official mandates or guidance documents across legislation, government guidance, and documents related to two controversial US risk-assessment tools, then inductively coded their oversight principles and mechanisms.The corpus comprises 23 proposed or passed laws, eight policy guidance documents, and ten documents related to the two risk-assessment tools.Source: §3 · p. 4

3.1. Restricting “solely” automated decisions

• Twenty of 41 documents, all legislation, restrict or prohibit solely automated decisions; 12 of 19 policies also provide a right to post hoc human intervention.The survey describes these policies as granting subjects rights and protections, including the ability to request human review of a decision.Source: §3.1 · p. 5

3.2. Emphasizing human discretion

• Fourteen documents emphasize human discretion, presenting it as a safety valve and, in two state supreme court cases, as grounds for allowing risk assessments to inform sentencing.The documents include one law, three policy guidance documents, and ten related to criminal-justice and child-welfare risk assessments.Source: §3.2 · pp. 5–6

3.3. Requiring “meaningful” human input

• Seven documents call for meaningful oversight: reviewers should be able to override outputs, understand how algorithms work, and weigh all relevant information rather than depend on recommendations.The policies do not precisely define meaningful involvement, but identify reviewer authority and competence, algorithmic understanding, and independent consideration of decision-relevant information as central components.Source: §3.3 · pp. 6–7

• Evidence from government and other settings suggests people generally cannot provide the protections against algorithmic errors, biases, and inflexibility envisioned by policymakers.The section evaluates whether people can perform the oversight functions that policies require.Source: §4 · p. 7
• Assumptions that human oversight is effective promote false security in adopting algorithms and reduce vendor and policymaker accountability for algorithmic harms.Source: §4 · p. 7

4.1. Flaw 1: Human oversight policies are not supported by empirical evidence

• Human oversight policies have little empirical basis, and research suggests people cannot reliably perform the oversight functions they prescribe.The section examines restrictions on solely automated decisions, human discretion, and mandates for meaningful oversight.Source: §4.1 · p. 7

4.1.1. Restrictions on “solely” automated decisions provide superficial protection

• Restrictions on “solely” automated decisions offer limited protection because existing human involvement can exempt scrutinized cases and nominal review can bypass safeguards.Post hoc review also puts the burden on harmed individuals and can be slow and onerous to obtain (Calo & Citron, 2021).Source: §4.1.1 · p. 7

4.1.2. Human discretion does not improve outcomes

• Human discretion often fails: people defer to incorrect algorithmic advice, misjudge outputs, and may introduce additional errors and racial disparities.London police overestimated facial-recognition match accuracy (Fussey & Murray, 2020), while judges’ deviations from risk recommendations can increase detention and racial disparities (Stevenson & Doleac, 2021).Source: §4.1.2 · pp. 7–8

4.1.3. Even “meaningful” human oversight does not improve outcomes

• “Meaningful” oversight is undefined in these policies, and its proposed components—override, explanations, transparency, and independent judgment—often fail to improve oversight.Explanations can increase trust in incorrect recommendations, transparency can reduce error detection, and automation bias can persist despite training and verification instructions.Source: §4.1.3 · p. 8

4.2. Flaw 2: Human oversight policies legitimize flawed and unaccountable algorithms in government

• Human oversight provisions create a loophole for adopting flawed algorithms while leaving agencies able to shirk accountability for resulting harms.The article argues that oversight does not protect against algorithmic harms, yet can reduce scrutiny of government algorithms.Source: §4.2 · p. 9

4.2.1. The assumption of effective human oversight provides a false sense of security in adopting algorithms

• Because oversight cannot mitigate underlying concerns, policies can provide cover for inappropriate algorithm use in government decisions.The assumption that human oversight makes algorithms safe lends a false sense of security despite concerns about error, bias, and inflexibility.Source: §4.2.1 · p. 9

4.2.1.1. Example 1: Human overrides of algorithmic deci-

• Human overrides appear to offer quality control, but people tend to override algorithms detrimentally and often over-rely on their advice.When algorithms omit relevant factors, people may give greater weight to the factors algorithms emphasize; the proposed remedies are to improve algorithms or stop using them if their flaws cannot be fixed.Source: §4.2.1.1 · p. 9

4.2.1.2. Example 2: Wisconsin v. Loomis.

• In Wisconsin v. Loomis, judicial discretion and limits on reliance did not resolve concerns about COMPAS errors, bias, or due process (Wisconsin Supreme Court, 2016).The Court’s reasoning was internally conflicted: COMPAS was presented as useful for improving outcomes but defended in Loomis because it had no discernible impact on the sentence. Evidence also suggests judges may be influenced by algorithmic advice without recognizing it; the article argues courts should scrutinize the tool’s quality and appropriateness instead.Source: §4.2.1.2 · p. 10

4.2.2. Relying on human oversight diminishes responsibility and accountability for institutional decision-makers

• Oversight shifts responsibility for algorithmic harms from agency leaders and vendors to frontline operators, who have little control over system design and implementation.Governments and vendors can praise an algorithm’s capabilities while defending it as a mere “second opinion” subject to human review, allowing them to claim credit for benefits and blame operators when problems occur.Source: §4.2.2 · pp. 10–11
• In Detroit, officials and vendors blamed operators after a facial-recognition match led to a wrongful arrest, although the system was acknowledged to be incorrect 96% of the time.The article argues that leadership and vendors, who chose to use a poorly tested system known to have low accuracy on Black faces, bear greater responsibility; it argues that police facial recognition should be banned.Source: §4.2.2 · p. 11

5. From human oversight to institutional oversight

• Policymakers should shift from relying on human oversight to a more rigorous and democratic approach to protecting against harms from government algorithms.The study finds that human oversight cannot provide the desired protections and can legitimize flawed, unaccountable algorithms without remedying their problems.Source: §5 · p. 11

5.1. The upper bound of human oversight

• Training and design may improve human–algorithm collaboration, but skill mismatch, automation bias, conflicting goals, and legitimization of unjust tools constrain human oversight.Research could test interventions such as decision support, structured collaboration, training, and override justifications; however, monitoring systems that outperform people and reconciling consistency with discretion remain difficult.Source: §5.1 · pp. 11–12

5.2. Institutional approach for overseeing government algorithms

• Institutional oversight raises the burden on agencies to justify whether and how to use algorithms, shifting accountability for algorithmic harms toward institutional leaders.The proposal does not call for either autonomous algorithms or exclusive reliance on human judgment.Source: §5.2 · p. 12

5.2.1. Stage 1: Agency justification and evaluation

• Before adoption, agencies must provide a written justification that algorithm use is appropriate and that proposed human oversight is empirically supported.The agency's report then proceeds to public review and approval.Source: §5.2 · p. 12, §5.2.1 · p. 12

5.2.1.1. Question 1: Is it appropriate to incorporate the algo-

• Agencies should rule out prohibited uses, then weigh decision discretion against algorithm trustworthiness when deciding whether an algorithm belongs in a decision.High discretion with an untrustworthy algorithm favors human decision-making; low discretion with a trustworthy algorithm favors algorithmic primacy, while hybrid cases default to human judgment unless empirical evaluation shows the algorithm improves outcomes.Source: §5.2.1.1 · pp. 12–14

5.2.1.2. Question 2: How should the algorithm be integrated

• Before adopting a human–algorithm collaboration, agencies should test whether it improves outcomes and whether people can perform the required oversight, then monitor its effects in practice.The proposed evaluations begin with laypeople in laboratory studies and proceed to practitioner studies; ongoing monitoring should track overrides and whether algorithms erode decision-makers’ moral agency.Source: §5.2.1.2 · p. 14

5.2.2. Stage 2: Democratic review and approval

• An agency’s adoption report should receive public or democratically accountable review and approval, using processes adapted to the jurisdiction.Examples include municipal council review of surveillance technologies and extending notice-and-comment procedures to agency algorithm use.Source: §5.2.2 · pp. 14–15

5.3. Benefits of institutional oversight approach

• Institutional oversight would require empirical evidence for proposed oversight, scrutinize algorithm appropriateness, strengthen institutional accountability, and enable democratic participation.The author argues that existing policies generally omit red lines, task-specific appropriateness, predeployment evaluations of human–algorithm collaboration, and public input and approval.Source: §5.3 · p. 15
• The Act bans some AI uses, such as public-authority social scoring, but classifies most applications highlighted by the coalition as “high-risk” rather than prohibiting them (European Digital Rights, 2021b).Source: §5.3 · p. 16
• Legislators have called for stronger facial-recognition bans, setting up future showdowns with member states and technology companies that support the technology (Kelly, 2021).Source: §5.3 · p. 16
• Recent US facial-recognition and predictive-policing bans required concerted local advocacy that overcame significant resistance from agencies (Hill, 2021; Miller, 2020).Source: §5.3 · p. 16

• Human oversight policies often fail because people cannot provide envisioned oversight, while legitimizing flawed, unaccountable government algorithms and hindering institutional accountability.Source: §6 · p. 16
• Institutional oversight would require agencies to justify algorithm adoption and evidence for proposed human oversight, then publish those justifications for democratic review and approval.These two stages precede an agency’s ability to adopt an algorithm.Source: §6 · p. 16
• Effective regulation must address algorithms’ social contexts and implementation evidence, since intuitive but ineffective rules can leave harms in place while creating a perception that the problem is solved.A sociotechnical, evidence-based approach can support more democratic and equitable decisions about algorithmic governance.Source: §6 · p. 16

Appendix: Summary of human oversight policies

• The appendix table summarizes the 41 policy documents reviewed in the study.These are the documents included in the author's review.Source: Appendix: Summary of human oversight… · p. 16
• Document classification distinguishes legislation, government or government-appointed guidance, and manuals, policies, and court cases concerning specified US tools and risk assessments.The third category covers the Allegheny County Family Screening Tool and risk assessments used in US criminal justice settings.Source: Appendix: Summary of human oversight… · p. 16
• The table categorizes oversight approaches as restricting solely automated decisions, emphasizing human discretion, or requiring meaningful human input.Documents are grouped by their approach to human oversight.Source: Appendix: Summary of human oversight… · p. 16