"People can change, and patterns can be broken": Contextualizing Tradeoffs in Automated Decision-Making Systems

• High-stakes ADM can affect health, safety, and fundamental rights, and its use has raised concerns about unfair outcomes, training-data privacy, and adversarial tampering.The paper notes deployments in sensitive domains including finance, recidivism, and healthcare.Source: §1 · p. 1
• Defenses for privacy, fairness, and security can impose costs on other objectives, while research had not examined how users perceive or prioritize these tensions.For example, differential privacy may harm model performance for minority groups, and some tampering defenses can increase privacy-attack vulnerability.Source: §1 · p. 1
• The study asks how people accept HDM and ADM across high-stakes scenarios and what considerations shape their views and tradeoff priorities.The authors argue that affected people’s perspectives are crucial to guiding ADM design and deployment.Source: §1 · p. 2
• Participants valued HDM for context-appropriate judgment and non-data considerations, but appreciated ADM’s efficiency and perceived objectivity in structured tasks.They prioritized tradeoffs contextually, sometimes accepting bias, privacy compromise, or adversarial manipulation in light of stakes, information sensitivity, and possible consequences.Source: §1 · p. 2

• ADM refers to systems that automate decisions through machine learning, artificial intelligence, or data-driven analysis, and is used in credit limits, health-risk coverage, hiring, and criminal-justice assessments.These systems are deployed with the intention of improving decision processes that materially affect people’s lives.Source: §2 · p. 2
• Security risks here are limited to model manipulation through evasion or poisoning, while privacy risks include membership, attribute, distribution, and data-reconstruction attacks.Examples include manipulating inputs to force a misclassification or adding tampered records to training data.Source: §2 · p. 2
• Fairness risks broadly cover discriminatory model outcomes, and defenses for privacy, fairness, or security may worsen another desirable property, such as accuracy.The background says real-world deployments must balance such tradeoffs and comply with law to avoid possible legal consequences.Source: §2 · pp. 2–3
• New laws and policies address ADM safety, fairness, resilience, public trust, and privacy, including high-risk applications identified in the EU Act.Examples include state-level U.S. regulations and the Illinois Artificial Intelligence Video Interview Act for hiring.Source: §2 · p. 3

• Privacy perceptions depend on formal protections, application context, and cultural and demographic contexts; protections include differential privacy, secure multiparty computation, and homomorphic encryption.Prior work finds that users’ trust and acceptability of such protections vary across domains such as healthcare, finance, and surveillance.Source: §3 · p. 3
• Fairness judgments vary by domain and reflect algorithmic factors, human values, and stakeholder characteristics, while users disagree on appropriate metrics and may prioritize accuracy in high-stakes settings.Prior work suggests fairness is socially constructed rather than a fixed system property.Source: §3 · p. 3
• Perceptions of resistance to adversarial manipulation have received less attention than privacy and fairness perceptions, with existing studies largely limited to recommendation systems and autonomous vehicles.The cited work focuses mainly on technical measures and pays limited attention to people negatively affected by manipulation failures.Source: §3 · p. 3
• Because deployed systems must balance objectives, the study asks whether people’s priorities remain stable or shift across ADM scenarios.The authors identify this understanding as relevant to selecting deployment objectives and designing techniques with different tradeoffs.Source: §3 · p. 3

4 Methods

• The quasi-experimental study examined how acceptance of HDM and ADM and perceptions of tradeoffs differed across four scenarios.Source: §4 · p. 3

4.1 Study Components

• Participants were randomly assigned to medical insurance, job screening, prison sentencing, or mortgage approval and rated HDM and ADM before explaining their choices and selecting tradeoff preferences.The study presented eight tradeoff combinations across two sets, with each participant receiving one set; a pilot led to removing the neutral option and adding explicit paired choices and context.Source: §4.1 · pp. 3–4, Table 2 · p. 4

4.2 Participants

• The final sample comprised 777 participants recruited in the United States, identified as experiential experts and spanning varied demographic and educational backgrounds.Ten responses were discarded after review; 86% reported not working in machine learning.Source: §4.2 · p. 4

4.3 Analysis Methods

• Analyses used paired and across-scenario statistical tests, summarized tradeoff choices by the share prioritizing each objective, and developed themes through iterative qualitative coding.Figure 1 presents acceptability distributions on a five-point scale, with sample sizes, means, and confidence intervals for each condition.Source: §4.3 · pp. 4–5, Figure 1 · p. 5

4.4 Limitations

• The scenarios do not cover every high-risk ADM type, and the WEIRD participant sample is not representative of the global population.Hypothetical scenarios may not capture real cases’ complexity or emotional weight, and the presented tradeoffs may simplify real-world technical and ethical considerations.Source: §4.4 · p. 5

5 Acceptability Results for HDM and ADM

• HDM was significantly preferred over ADM in Mortgage and Job, with large effects, and Prison, with small effects, but not Insurance.Positive acceptability ratings for HDM were 72.5% in Mortgage and 75.8% in Job; for ADM they were 46.2% and 44.3%, respectively.Source: §5 · p. 5
• HDM acceptability was significantly higher in Job and Mortgage than in Prison and Insurance.The overall scenario difference was significant (Kruskal–Wallis statistic 79.359, p < 0.001).Source: §5 · pp. 5–6
• ADM was significantly more acceptable in Job and Mortgage than in Prison; no other scenario pairs differed significantly.The overall difference across scenarios was significant (Kruskal–Wallis statistic 10.527, p = 0.015).Source: §5 · p. 6
• ADM acceptability differed by minority status in Prison and by ML-practitioner status in Insurance; no HDM demographic differences were significant.Non-minority participants rated Prison ADM more acceptably; ML practitioners rated Insurance ADM more acceptably.Source: §5 · p. 6

6.1 Perceived Advantages of HDM

• Participants valued HDM for nuanced judgments beyond data, including personal circumstances, soft skills, grey areas, and human experience.They described flexibility as supporting context-appropriate decisions.Source: §6.1 · p. 6

6.2 Perceived Disadvantages of HDM

• Participants worried that human biases, emotion, and inconsistent judgment could produce unfair or inaccurate outcomes across scenarios.Concerns included affinity bias, quota pressure, sympathy, anger, mood, and subjective interpretations.Source: §6.2 · pp. 6–7

6.3 Perceived Benefits of ADM

• Participants saw ADM as efficient, objective, and reliable for structured tasks, with perceived strengths in processing volume and detecting patterns.They considered fixed-rule screening suitable when ethical interpretation or individual context was not central.Source: §6.3 · p. 7

6.4 Perceived Disadvantages of ADM

• Participants feared ADM could perpetuate historical discrimination, reflect unrepresentative training data, and miss emotional or moral nuance, including the possibility of change.One participant cautioned that “People can change, and patterns can be broken.”Source: §6.4 · p. 7
• Participants also raised concerns about opaque decisions, weak accountability, privacy violations, and using ADM in sensitive high-stakes settings.They worried that errors could go unnoticed and that people might lack explanations or an avenue to appeal.Source: §6.4 · p. 7

7 Tradeoff Preference Within Scenarios

• Participants chose accuracy in fairness-versus-accuracy settings except Job, prioritized fairness in Prison and Job for fairness-versus-inference, and chose security in Prison and Mortgage for fairness-versus-security.For security versus accuracy, they chose accuracy in every scenario except Prison.Source: §7 · pp. 7–8

7.1 Fairness versus Accuracy

• 80% in Prison, 75% in Mortgage, 78% in Insurance, and 68% in Job chose accuracy-maximizing options, accepting bias as a consequence.The reported bias levels were split between strong and moderate options in each scenario.Source: §7.1 · p. 8

7.2 Fairness versus Membership Inference

• Fairness was preferred in Prison (63%) and Job (76%), while privacy was preferred in Mortgage (60%) and Insurance (54%) in the fairness-versus-membership-inference tradeoff.Participants accepting the fairness option also accepted membership-inference risk; participants choosing privacy accepted bias.Source: Figure 3 · p. 8, §7.2 · p. 8

7.3 Fairness versus Distribution Inference

• Fairness was preferred in Prison (73%) and Job (67%), while privacy was preferred in Mortgage (57%) and Insurance (56%) in the fairness-versus-distribution-inference tradeoff.The fairness choices accepted distribution-inference risk, while privacy choices accepted bias.Source: Figure 4 · p. 8, §7.3 · p. 8

7.4 Fairness versus Security

• Security was preferred in Prison (60%), Mortgage (67%), and Insurance (55%), whereas Job participants slightly favored fairness (51%).Those choosing fairness in Job accepted high or moderate manipulation risk.Source: Figure 5 · p. 8, §7.4 · p. 8

7.5 Security versus Accuracy

• In Prison, Mortgage, Job, and Insurance, respectively, 60%, 74%, 72%, and 68% chose accuracy-maximizing options, accepting risk of inaccurate predictions to prevent manipulation.Source: §7.5 · p. 9

7.6 Security versus Membership Inference

• For manipulation risk versus membership inference, privacy-maximizing options were chosen by 67% in Mortgage, 51% in Prison, and 52% in Insurance; 57% chose security-maximizing in Job.Source: Figure 7 · p. 9, §7.6 · p. 9

7.7 Security versus Distribution Inference

• For manipulation risk (MR) versus distribution inference (DI), privacy-preferred options were selected by 53% in Prison, 54% in Mortgage, 53% in Job, and 58.9% in Insurance.Source: Figure 8 · p. 9, §7.7 · p. 9

8 Tradeoffs Preference Across Scenarios

• Only fairness versus membership inference and fairness versus distribution inference showed significant preference differences across scenarios; all other examined tradeoffs did not.Both reported significant tradeoffs had p < .001.Source: §8 · pp. 9–10

9.1 Concerns Beyond Technical Fairness Metrics

• Participants treated fairness as broader than demographic parity, linking it to proportionality, privacy, accuracy, and protection from manipulation.Privacy disclosures could reinforce discrimination, manipulation could enable targeted discrimination, and inaccurate outcomes were viewed as unfair burdens.Source: §9.1 · p. 10

9.2 Contextual Influences on Prioritization

• Participants accepted revealing membership information when context made it useful or expected, but maintained limits on which attributes should affect decisions.Some considered criminal history relevant to hiring, while some rejected wealth as a factor in prison decisions despite its availability.Source: §9.2 · p. 10
• Participants saw aggregate demographic information as useful for fairness monitoring, yet often prioritized fair outcomes and resistance to manipulation over protecting such data.This priority was strongest when the information seemed non-sensitive and contextually appropriate to disclose.Source: §9.2 · pp. 10–11

9.3 Accepting Bias for the Greater Good

• Participants sometimes accepted favoritism or training-data bias over marginalization or deliberate manipulation, but resisted sacrificing accuracy for fairness or security.They often prioritized minimizing harm to the larger population and viewed deliberate human manipulation as more controllable than bias in training data.Source: §9.3 · p. 11

9.4 Consequences Beyond Biased Outcomes

• Participants connected biased outcomes with inaccuracy and viewed manipulation and aggregate-data disclosure as risks to system integrity beyond individual decisions.They also raised concerns that aggregate data could be misused to reinforce exclusionary practices or enable profiling.Source: §9.4 · p. 11

9.5 Negative Impact on Stakeholders Matters

• Participants weighed effects on end users, organizations, and society, citing lost trust, reinforced disadvantage, financial damage, legal exposure, and life-altering errors.In the prison scenario, inaccurate decisions could prolong an innocent person’s imprisonment or enable dangerous criminals’ release.Source: §9.5 · pp. 11–12

Folk theories of effectiveness and efficiency

• Participants sometimes describe ADM as error-free and fact-driven, while limited understanding can foster over-reliance and excessive confidence in its outcomes.The authors point to clearer reporting of system limitations and better ADM literacy as responses.Source: §10 · p. 12

Sensitivity of information is enmeshed with potential harm

• Participants relate information sensitivity to potential discrimination and personal relevance, accepting privacy risks more readily when information seems less sensitive or contextually appropriate.The passage connects this reasoning to privacy calculus, boundary regulation, and contextual integrity frameworks.Source: §10 · p. 12
• Structured dataset documentation should describe data use, proxy-discrimination risks, domain norms, and stakeholder expectations to ground high-stakes tradeoffs in perceived costs and benefits.The authors offer datasheets for datasets as one practical example.Source: §10 · p. 12

Deceptive linguistics is a concern for influencing participants’ preferences

• Framing an outcome as favoring one group rather than marginalizing another can influence preferences despite similar exclusionary outcomes.Participants viewed favoring a group more positively, although both outcomes were unfair.Source: §10 · p. 12

Presented bias in comparison to existing societal bias

• Indirect, hypothetical framing lets participants consider discrimination principles without taking an explicit position on charged categories; neutral proxies still reveal contextual tradeoff priorities.Hair color was used as a neutral proxy attribute to reduce confounds from existing stereotypes.Source: §10 · pp. 12–13

10 Discussion

• Technical abstractions of human values do not sufficiently align with affected people, and human oversight alone cannot bridge the mismatch between mutable social concepts and fixed technical metrics.The authors ask whether the limitations of required guardrails and accountability measures outweigh ADM’s efficiencies.Source: §10 · p. 13

11 Conclusion

• Participants generally preferred human decision-making for accountability and handling complex, consequential situations, while privacy, fairness, and robustness priorities varied by domain.They viewed information sensitivity as dependent on its use, and fairness as a feature of system behavior rather than only a statistical measure.Source: §11 · p. 13

B Ethical Considerations

• Researchers used fictional scenarios in real-world domains to elicit lived experiences while limiting distress, but Prolific recruitment excluded some experiences.The authors acknowledged that IRB approval alone was insufficient and described taking additional steps to conduct the study ethically.Source: B Ethical Considerations · p. 16

D Study Questionnaires

• The study questionnaires comprised eight variants covering four scenarios and two tradeoff sets: fairness and security.Demographic questions were also included.Source: D Study Questionnaires · p. 16

D.1 Fairness-Insurance

• Insurance questionnaires presented fairness modifications that traded brunette favoritism against accuracy, disclosure of depression status or women’s data counts, and manipulation resistance.The scenario concerned insurance premiums based on heart-disease risk.Source: §D.1 · pp. 16–17

D.2 Fairness-Job

• Job questionnaires presented fairness modifications trading brunette favoritism against hiring-prediction accuracy, criminal-background disclosure or women’s data counts, and manipulation resistance.The scenario concerned hiring a software engineer based on candidates’ likelihood of success.Source: §D.2 · pp. 17–18

D.3 Fairness-Mortgage

• The mortgage-fairness questionnaire scenario concerned interest rates set according to applicants’ default risk, with higher rates for higher-risk individuals.The provided passage begins describing employee assessment of a client’s non-payment risk.Source: §D.3 · p. 18
• In mortgage lending, participants considered whether reducing an AI’s preference for brunettes justified inaccurate interest rates, revealing minority-group membership or women’s counts, or greater employee manipulation.The vignettes framed each tradeoff as a choice between degrees of bias and the corresponding accuracy, privacy, or manipulation risk.Source: §D.3 · p. 19

D.4 Fairness-Prison

• In prison sentencing, participants weighed reducing an AI’s preference for brunettes against inaccurate sentences, revealing wealth or women’s counts, or greater employee manipulation.The scenarios asked prisoners to choose among different levels of bias and corresponding risks.Source: §D.4 · p. 20

D.5 Security-Insurance

• In health insurance, participants considered whether making an AI harder for employees to manipulate justified less accurate premiums for low-risk applicants.The supplied passage begins a scenario in which employee manipulation could disadvantage blue-eyed applicants despite their low risk.Source: §D.5 · p. 21
• Insurance scenarios varied manipulation resistance against incorrect premiums and the risk of revealing depression status, women’s representation, or training-data information.Respondents were asked to choose among options ranging from no or low disclosure risk to higher risk alongside differing manipulation difficulty.Source: §D.5 · p. 21

D.6 Security-Job

• Hiring scenarios asked respondents to weigh manipulation resistance against incorrect hiring predictions and disclosure of criminal-background status, women’s representation, or training-data information.The AI could be manipulated to disadvantage high-potential blue-eyed applicants; the prompts then presented tradeoffs involving prediction correctness and information disclosure.Source: §D.6 · pp. 22–23

D.7 Security-Mortgage

• Mortgage scenarios asked respondents to weigh manipulation resistance against incorrect interest rates and disclosure of minority-group status or women’s representation in the training data.The AI could be manipulated to give low-risk blue-eyed applicants higher rates; prompts described tradeoffs between manipulation protection and correct rates or privacy.Source: §D.7 · pp. 23–24

E.1 Demographic Distribution

• Tables 3–5 report participant gender, minority-group, and machine-learning-practitioner distributions; the supplied passages do not provide their values.Table 3 is described as covering gender distribution in human versus AI decision-making preference across scenarios.Source: §E.1 · p. 25, Table 3 · p. 25

F Human versus ADM: Quantitative Analysis

• Figure 10 plots mean HDM and ADM acceptability scores with 95% confidence intervals on a five-point scale.The scale runs from completely unacceptable to completely acceptable.Source: F Human versus ADM: Quantitative… · p. 25, Figure 10 · p. 26
• Tables 4 and 5 summarize human-versus-AI preference distributions across scenarios for minority-group participants and ML practitioners.Both tables exclude participants who preferred not to answer.Source: Table 4 · p. 26, Table 5 · p. 26
• HDM and ADM acceptability differed significantly in every scenario except insurance.The reported Wilcoxon signed-rank results use a significance level of 0.005.Source: F Human versus ADM: Quantitative… · p. 26, Table 7 · p. 26

G Tradeoff Preferences: Across Scenarios

• Across scenarios, preference distributions differed significantly only for fairness versus membership inference and fairness versus distribution inference.Tables 10 and 11 present pairwise scenario comparisons for those two tradeoffs.Source: G Tradeoff Preferences: Across Scenarios · p. 26, Table 10 · p. 26, Table 11 · p. 26

H Tradeoff Preferences: Within Scenarios

• For the eight tradeoffs, significant chi-square results were followed by residual analysis to identify options chosen more or less often across the four scenarios.Tables 6 and 8 present pairwise scenario comparisons for HDM and ADM acceptability, respectively.Source: H Tradeoff Preferences: Within Scenarios · p. 26, Table 6 · p. 27, Table 8 · p. 27

H.2 Tradeoff: Fairness versus membership inference

• Participants showed significant preferences among fairness-versus-membership-inference response categories in every scenario except insurance.Table 17 summarizes the chi-square tests.Source: §H.2 · p. 27
• Across all scenarios except insurance, participants showed a significant preference among the four response categories; in the prison and job scenarios, they significantly preferred a response category.Tables 18 and 20 show the prison- and job-scenario results.Source: §H.2 · p. 27
• In mortgage, participants significantly less preferred “Slight bias; moderate chance of MI.”The passage identifies Table 19 for this scenario-level residual result.Source: §H.2 · p. 28

H.4 Tradeoff: Fairness versus security

• In mortgage and prison scenarios, participants significantly preferred choosing among four response categories; across those categories, they significantly preferred the highest protection against adversarial tampering, High bias; No Manipulation Risk.Source: §H.4 · p. 29

H.5 Tradeoff: Security versus accuracy

• For security versus accuracy, mortgage and job participants significantly chose “Moderate manipulation risk; Slight Inaccuracy,” while insurance participants chose the accuracy-maximizing option.The overall preference test was significant in every scenario except prison.Source: §H.5 · p. 29, Table 27 · p. 30

H.6 Tradeoff: Security versus membership inference

• For security versus membership inference, no scenario showed significant preference differences except mortgage, where “Slight manipulation risk; Moderate chance of MI” was chosen less.Table 31 summarizes the tests; Table 32 reports the mortgage residual analysis.Source: §H.6 · p. 30, Table 31 · p. 31

H.7 Tradeoff: Security versus distribution inference

• For security versus distribution inference, participants did not significantly favor or disfavor any response option across the scenarios.Table 33 summarizes the chi-square tests.Source: §H.7 · p. 31, Table 33 · p. 31

H.8 Tradeoff: Security versus data reconstruction

• In mortgage, participants chose “Slight manipulation risk; Moderate chance of DR” significantly less often than the other options.Across the other scenarios, no option was chosen significantly more or less than the others.Source: §H.8 · p. 31, Table 35 · p. 32
• The mortgage residual analysis compared security-versus-data-reconstruction response categories against a Bonferroni-adjusted critical value of z = ±2.50.The threshold corresponded to α = .0125 for four category-level comparisons.Source: Table 34 · p. 32
• Table 35 summarizes the chi-square test for the security-versus-data-reconstruction tradeoff at a significance level of .05.Source: Table 35 · p. 32