PaperlayerView exact revision on arXiv

Source-linked AI summary

Do job seekers value procedure in AI hiring only for error correction? Evidence from a conjoint experiment

Chuyao Wang, Patrick Sturgis, Daniel de Kadt

arXiv:2609.16390v1cs.CY

TL;DR

Prior studies rarely vary procedural features and system performance independently, leaving unclear whether applicants value procedure for its own sake or for errors it corrects. In a preregistered conjoint experiment, 1,919 U.S. job seekers chose between systems with randomized attributes. Human involvement carried weight comparable to reducing wrongful rejections, while appeal, opt-out, and audit value did not rise as errors became more common.

  • Problem

    Prior studies rarely vary procedure and performance independently, leaving unclear whether applicants value procedure for its own sake or for the errors it corrects.

    Source: §1 · pp. 2–3

  • Method

    A preregistered paired-profile conjoint asked 1,919 U.S. job seekers to make eight choices between systems with independently randomized attributes.

    Source: §1 · p. 3

  • Results

    Cutting wrongful rejections from 30% to 10% was worth about as much as putting a human into the decision, while appeal, opt-out, and bias-audit value did not rise as wrongful rejections became more common.

    Source: §5 · pp. 14, 16

  • Takeaways & Limitations

    The findings constrain a simple error-correction account and are consistent with applicants valuing procedure partly for its own sake.

    Source: §5 · p. 14

    The study measures stated choices in a hypothetical task rather than application behavior, and offers no status quo or reject-both option.

    Source: §5 · p. 16

Abstract

from arXiv · show

Employers increasingly delegate initial screening to automated systems, which in many cases reject an application before any human reads it. Acceptance of such systems plausibly depends both on how well they perform and on the procedure that produces the decision. Prior studies rarely vary procedure and performance independently, leaving it unclear whether applicants value procedure for its own sake or for the errors it corrects. In a preregistered paired-profile conjoint experiment, 1,919 United States job seekers made eight choices between systems with independently randomized levels of decision authority, error rate, explanation, opt-out, appeal, and independent bias audit. The value of the appeal, the opt-out, and the bias audit did not rise as wrongful rejections became more common, each staying within a preregistered equivalence bound. Human involvement carried more weight than any procedural feature, moving stated choice about as much as cutting wrongful rejections from 30% to 10%. These patterns constrain a simple error-correction account and are consistent with applicants valuing procedure partly for its own sake, so that improving a system's performance does not substitute for a right applicants can invoke.

1 Introduction

As employers increasingly use AI in hiring, applicants’ reservations raise the question of what makes these systems acceptable and whether acceptance depends on performance. This study tests whether procedure carries value of its own by independently varying system performance and procedural features in choices made by job seekers.

1 Introduction

  • Public reservations persist even as employers adopt AI for hiring, prompting the study’s question about what makes these systems acceptable to applicants.The introduction notes broad doubts about hiring algorithms and continued worker apprehension about workplace AI.Source: §1 · p. 2
  • Prior research leaves unclear whether applicants value procedure for its own sake or for the mistakes it can detect or reverse.The two accounts differ in whether procedural value should grow as wrongful rejections increase.Source: §1 · p. 3
  • Earlier applicant studies largely measured fairness perceptions, while conjoint evidence on human versus algorithmic decision-makers often came from observers rather than decision subjects.The study instead asks job seekers—the people judged by hiring systems—to choose between systems.Source: §1 · p. 3
  • The preregistered conjoint asked 1,919 U.S. job seekers to make eight choices between systems with independently varied authority, error rate, explanation, opt-out, appeal, and bias audit.The design also uses preregistered equivalence tests to assess whether procedural-feature value changes with error rates.Source: §1 · p. 3

2 Relevant literature and hypotheses

The literature section distinguishes instrumental error correction from performance-independent reasons for valuing procedure, then derives hypotheses about human involvement, recourse, and audits. It motivates testing these attributes while varying error rates independently.

2 Relevant literature and hypotheses

  • The study contrasts an error-correction account, in which procedural features help detect or reverse mistakes, with a performance-independent account in which they retain value across accuracy levels.Distinguishing the accounts requires varying decision authority, error rate, and applicant recourse independently.Source: §2 · pp. 3–4

2.1 Why human involvement may matter more than procedural features

  • Human involvement may convey standing and allow attention to individual circumstances; the hypotheses predict it raises choice probability and has the largest effect among four confirmatory attributes.The other confirmatory attributes are appeal, opt-out, and bias audit.Source: §2.1 · p. 4

2.2 The interaction between the confirmatory attributes and the error rate

  • The error-correction hypothesis predicts that confirmatory attributes gain value as wrongful rejections rise, whereas performance independence predicts no increase beyond a prespecified equivalence bound.The prediction is sharpest for appeal, which offers human re-review; an audit targets group-level error distribution rather than the overall error rate.Source: §2.2 · pp. 4–5

2.3 Why appeal and the opt-out gain from a human

  • Appeal is framed as voice and opt-out as exit; both are hypothesized to gain value when a human is already involved because human authority can act on them.They could instead substitute for human involvement when AI decides alone.Source: §2.3 · p. 5

2.4 Why a system-level audit ranks below individual attributes

  • An independent bias audit targets group-level disparate impact and offers no individual re-review, motivating the hypothesis that it ranks below human involvement, appeal, and opt-out.Its benefit is described as diffuse and shared, unlike individual voice or recourse.Source: §2.4 · p. 6

3 Data and methods

The study recruited current U.S. job seekers for a preregistered paired-profile conjoint experiment. It randomized system attributes across repeated choices and used regression and equivalence tests to estimate effects and assess moderation.

3.1 Sample recruitment, ethics, and data quality

  • The sample comprised 1,919 U.S.-resident current job seekers recruited through Prolific; the preregistered analysis did not condition on completion time, response patterns, or the post-task manipulation item.Ninety-two percent answered the manipulation check correctly, identifying that error rate varied across tasks.Source: §3.1 · p. 6

3.2 Conjoint experimental design and instrument

  • In eight tasks, respondents chose between two hiring systems described on six attributes, with attribute levels independently randomized subject to profiles differing on at least one attribute.Authority had three levels, error rates were 10%, 20%, or 30%, and procedural features included explanation, opt-out, appeal, and independent bias audit.Source: §3.2 · p. 7

4 Results

Procedural features retained their value as errors rose, while human involvement was valued more than any individual procedural feature. Applicants’ stated intention to apply exceeded their belief in AI hiring’s legitimacy, and the main findings were robust to several alternative specifications.

4 Results

  • As the system erred more often, procedural features’ value held steady: preregistered equivalence tests bounded change within ±0.05 on the choice-probability scale, while human involvement fell outside that bound.Source: §4 · p. 9

4.1 Human involvement outweighs the procedural features

  • Reducing the error rate from 30% to 10% raised choice probability by 0.285, a benchmark for other attribute effects and nearly equal to the pooled human-involvement effect (0.272).The difference was not distinguishable from zero (95% CI [−0.007, 0.033]).Source: §4.1 · p. 10

4.2 The value of the procedural features holds as errors rise

  • At the lowest tested error rate (10%), all four confirmatory attributes had positive conditional effects; as errors rose, the opt-out and bias audit held their value.Source: §4.2 · pp. 10–11

4.3 How appeal and the opt-out combine with human involvement

  • Appeal was valued more when a human was involved (+0.025, 95% CI [0.004, 0.047]) on the linear probability scale, but the opt-out interaction was effectively zero (−0.0003).The appeal interaction was not significant under logit, and the design does not identify why the opt-out interaction was near zero.Source: §4.3 · p. 12, Figure 3 · p. 12

4.4 Legitimacy and intention to apply

  • Intention to apply averaged 3.24 on a five-point scale, above legitimacy belief at 2.85; the measures were strongly associated (r = 0.65).7.8% reported below-midpoint legitimacy belief alongside above-midpoint intention to apply.Source: §4.4 · p. 13, Figure 4 · p. 13

4.5 Robustness to specification and scale

  • Appeal and the opt-out passed the stricter ±0.03 equivalence bound, while human involvement failed the ±0.05 bound and passed only at ±0.07.The alternative acceptability outcome had the same signs and a similar ordering (rank correlation 0.89).Source: §4.5 · p. 14, Figure 5 · p. 14

5 Discussion

Applicants valued appeal, opt-out, and bias audit without those features gaining value as wrongful rejections became more common, constraining a simple error-correction account. Human involvement carried weight comparable to reducing wrongful rejections from 30% to 10%, while the study leaves open why procedure retains value.

5 Discussion

  • Appeal, opt-out, and bias-audit value did not rise as wrongful rejections became more common, with each moderation inside the preregistered equivalence bound.These results do not support a simple error-correction account.Source: §5 · p. 14
  • Decision authority determines who makes the call, while appeal and opt-out give applicants recourse and an audit examines outcomes across cases.Removing re-review while adding an audit changes what applicants can do, not only what employers can document; nominal human review may offer no route to invoke.Source: §5 · p. 15
  • Applicants valued individual rights and system-level oversight separately, but preference evidence does not establish whether any legal provision is adequate.The discussion distinguishes oversight requirements, rights to human intervention and contestation, and bias-audit requirements across regulatory frameworks.Source: §5 · pp. 15–16
  • The study measures hypothetical stated choices by U.S. Prolific job seekers, not application behavior, and its sample is not representative of all job seekers.Features were binary and costless, leaving quality and implementation unspecified despite real review involving delay, effort, and uncertain outcomes.Source: §5 · p. 16
  • The design cannot identify why procedure retains value or which feature combination best protects applicants or improves decision quality.Standing, dignity, and distrust of unaccountable systems are possible explanations; testing them and the pattern where applicants cannot apply elsewhere remains open.Source: §5 · p. 16
  • Cutting wrongful rejections from 30% to 10% was worth about as much as putting a human into the decision.Applicants also valued human involvement more than any procedural feature.Source: §5 · pp. 14, 16

Declarations

The declarations record the study’s ethics approval, preregistration, and replication materials. The appendix covers the instrument, sampling, estimates, robustness, power, heterogeneity, and plan deviations; its instrument measures legitimacy, application, and withdrawal intentions.

Declarations

  • The pre-analysis plan was posted before outcome data were examined, and replication data and code are available.The appendix also summarizes deviations from the plan.Source: Declarations · p. 17

Appendix

  • The appendix documents the instrument, sampling and data quality, estimates, equivalence and interaction tests, robustness, power, human-preference heterogeneity, and plan deviations.Source: Appendix · p. 24

A Instrument and measures

  • Three five-point items measured perceived legitimacy, intention to apply, and intention to withdraw; withdrawal intention correlated negatively with legitimacy (r = −0.46) and application intention (r = −0.67).Source: A Instrument and measures · p. 24

B Sampling, exclusions, and data quality

The analytic sample comprised 1,919 respondents, and the estimation reports choice effects for human involvement and error rates. The passage also notes limits in demographic-field availability.

B Sampling, exclusions, and data quality

C Estimation and full estimates

D Equivalence tests

Appeal, opt-out, and bias-audit effects met the preregistered equivalence criterion across the tested error range, while human involvement did not. Additional interaction analyses include exploratory and scale-sensitive findings.

D Equivalence tests

  • Appeal, opt-out, and bias audit met the preregistered ±0.05 equivalence criterion for moderation; human involvement did not.At the stricter ±0.03 bound, appeal and opt-out remained equivalent, but the explanation and bias-audit intervals exceeded the bound before rounding.Source: D Equivalence tests · p. 27, Table D1 · p. 27

E Interaction and subgroup estimates

F Further robustness

Robustness checks found small changes in estimated attribute effects under sample restrictions and added controls. The power analysis targeted approximately 1,400 respondents using pilot-based simulations.

F Further robustness

  • Dropping all three data-quality flags at once moved an AMCE by at most 0.005 (n = 1,687), while individual respondent restrictions changed an AMCE by at most 0.001–0.003.Source: F Further robustness · pp. 28–29

G Power analysis

  • The target sample was approximately 1,400 respondents, set by the error-rate-moderation and opt-out-by-human tests using conservative pilot effect sizes.Pilot participants were excluded from the analytic study.Source: G Power analysis · p. 29
  • The plan was posted before analytic-study outcomes were examined, and no interim outcome analysis was conducted.Source: G Power analysis · p. 29

H Heterogeneity in the human preference

Most respondents favored human-involved over AI-alone profiles, though the within-respondent statistic was noisy. Observed dispersion only modestly exceeded a homogeneous-preference benchmark, which did not identify model-free individual-level variance.

H Heterogeneity in the human preference

  • 77% of the 1,911 respondents exposed to both process types had a positive within-respondent difference favoring human-involved over AI-alone profiles.With only eight tasks per respondent, this statistic is noisy.Source: H Heterogeneity in the human preference · p. 29
  • Prior AI-evaluation experience did not significantly moderate human involvement; unregistered analyses found no detectable age interaction and a small gender interaction (women +0.037, p = 0.003).Source: H Heterogeneity in the human preference · p. 29
  • The human-involvement statistic correlated only weakly with legitimacy, application-intention, and withdrawal-intention measures (|r| ≤ 0.14).Source: H Heterogeneity in the human preference · p. 29
  • Observed dispersion was 0.30 versus a 0.27 standard deviation under the homogeneous-preference benchmark (Figure H1), modestly exceeding it without identifying model-free individual-level variance.The benchmark held each respondent’s numbers of human-involved and AI-alone profiles fixed.Source: H Heterogeneity in the human preference · p. 29, Figure H1 · p. 30

I Deviations from the pre-analysis plan

The registered positive interaction was not supported, and a post hoc equivalence test was reported descriptively rather than as support for H3. The paper documents other plan-to-paper changes as wording or exploratory additions while noting where estimands, models, and decision rules remained unchanged.

I Deviations from the pre-analysis plan

Loading 2609.16390v1…