PaperlayerView exact revision on arXiv

Source-linked AI summary

The Initial Screening Order Problem

Jose M. Alvarez, Antonio Mastropietro, Salvatore Ruggieri

arXiv:2307.15398v5cs.LGcs.CY

TL;DR

Candidate screening often relies on an initial screening order (ISO), yet its effects on selection optimality and fairness have received limited attention. The paper formalizes two search formulations and compares algorithmic and human-like screeners; its analysis identifies fairness and optimality risks associated with position bias.

  • Problem

    The initial screening order, often supplied through information access systems, has received little attention despite potentially affecting selection optimality and fairness, especially with human screeners.

    Source: §1 · p. 1

  • Method

    The paper defines best-k and good-k search formulations, compares algorithmic and fatigue-affected human-like screeners, and uses simulations across screening settings.

    Source: §1 · p. 1, §5 · p. 8

  • Results

    Under good-k, a human-like screener can violate individual fairness while meeting the quota; position bias harms optimality, and algorithmic procedures may miss the best candidate.

    Source: §1 · p. 1

  • Takeaways & Limitations

    For practitioners, both the problem formulation and the screener's behavior while searching the ISO matter.

    Source: §5 · p. 8

    The results are limited by the functional assumptions used to formulate the problems and screeners, particularly the human-like screener.

    Source: §5 · p. 8

Abstract

from arXiv · show

We investigate the role of the initial screening order (ISO) in candidate screening. The ISO refers to the order in which the screener searches the candidate pool when selecting $k$ candidates. Today, it is common for the ISO to be the product of an information access system, such as an online platform or a database query. The ISO has been largely overlooked in the literature, despite its impact on the optimality and fairness of the selected $k$ candidates, especially under a human screener. We define two problem formulations describing the search behavior of the screener given an ISO: the best-$k$, where it selects the top $k$ candidates; and the good-$k$, where it selects the first good-enough $k$ candidates. To study the impact of the ISO, we introduce a human-like screener and compare it to its algorithmic counterpart, where the human-like screener is conceived to be inconsistent over time. Our analysis, in particular, shows that the ISO, under a human-like screener solving for the good-$k$ problem, hinders individual fairness despite meeting group fairness, and hampers the optimality of the selected $k$ candidates. This is due to position bias, where a candidate's evaluation is affected by its position within the ISO. We report extensive simulated experiments exploring the parameters of the best-$k$ and good-$k$ problems for both screeners. Our simulation framework is flexible enough to account for multiple candidate screening tasks, being an alternative to running real-world procedures.

1 Introduction

The paper formalizes the initial screening order (ISO) as part of candidate selection, examining how search behavior and position bias affect selected candidates. It develops two search formulations, compares algorithmic and human-like screeners, and situates the analysis in real-world screening practices and related fairness research.

1 Introduction

  • Under good-k, a human-like screener can violate individual fairness while meeting the quota; ISO position bias also harms optimality, and algorithmic procedures may miss the best candidate.The paper links the human-like screener's unequal evaluations of similar candidates to fatigue accumulated during the search.Source: §1 · p. 1
  • The paper formalizes best-k selection of the top candidates and good-k selection of the first good-enough candidates, then studies consistent algorithmic and fatigue-affected human-like screeners.Good-k permits a partial search of the candidate pool.Source: §1 · pp. 1–2

1.1 Qualitative Background

  • The motivating HR setting involved screener-chosen ISOs, full or partial searches, minimum requirements, representation quotas, and a target of one minute per candidate.These company-specific practices were presented as salient aspects likely to hold in similar human screening settings.Source: §1.1 · p. 2

1.2 Related Work

  • Prior work studies fair ISO construction and candidate-pool creation, while this paper focuses on how screeners search a given ISO; related screening studies report order-linked biases.The paper also connects its screener-centered approach to click models of user search.Source: §1.2 · p. 2
  • Unlike secretary-problem analyses of online selection, these formulations model offline set selection with individually evaluated candidates and irrevocable decisions, emphasizing the ISO's role.The good-k formulation is described as unusual relative to previously analyzed settings.Source: §1.2 · p. 2

2 Searching the Pool of Candidates

The section formalizes candidate selection under an initial screening order (ISO), distinguishing full-pool selection of the best k candidates from partial search for the first k good-enough candidates. It defines a protected-group quota and describes search procedures that model sequential access in ISO order.

2.1 Setting

  • The setting treats candidates as attribute vectors with a binary protected attribute, scored individually without using that protected attribute, and searched in an ISO chosen or supplied through an information access system.The screener may search the pool fully or partially but must respect the ISO.Source: §2.1 · pp. 2–3, §2 · p. 2

2.2 Two Problem Formulations

  • The quota q requires the selected set’s protected-candidate fraction to be at least q; it constrains composition, not the ordering of candidates within the set.The unconstrained case is q = 0.Source: §2.2 · p. 3
  • Best-k selects the fair top k candidates after scoring and ranking the complete pool, whereas good-k selects the first k candidates meeting the minimum score and quota.Good-k can stop searching after finding k eligible candidates; best-k requires evaluating the entire pool.Source: §2.2 · p. 3
  • Under good-k, changing the ISO can change which candidates are selected, even when the screener itself is not specified.With two candidates above the minimum score, reversing their order reverses which one is selected when k = 1.Source: §2.2 · p. 3

2.3 Two Search Procedures

  • The ExaminationSearch and CascadeSearch procedures model, respectively, best-k selection and good-k search while respecting the quota and the ISO.ExaminationSearch ranks candidates by score; CascadeSearch follows the ISO and checks the minimum score requirement before adding candidates.Source: §2.3 · p. 4
  • The procedures are intended to model sequential screening behavior under the ISO, not to introduce novel optimal algorithms.Applicants are disclosed in ISO order rather than score order, unlike the cited prior set-selection work.Source: §2.3 · p. 4

3 The Human-Like Screener

The human-like screener models fatigue as error that changes with search time. Because that error depends on a candidate’s position in the ISO, it can undermine individual fairness and optimality even when group fairness is met.

3 The Human-Like Screener

  • The algorithmic screener evaluates consistently, while the human-like screener’s fatigue makes candidate evaluation inconsistent over time.Evaluation proceeds along the ISO in discrete time steps; fatigue accumulates linearly in the model.Source: §3 · p. 4, §3.1 · p. 4

3.1 Fatigue and Fatigued Scores

  • Fatigue enters the human-like screener’s score as random error: it can increase variance alone or also introduce a decreasing mean.The first model permits over- or underscoring; the second produces a negative bias over time.Source: §3.1 · pp. 4–5

3.2 Position Bias Implications

  • Both screeners meet the group-fairness quota, but fatigue can violate individual fairness because candidates’ positions produce unequal evaluation error.The analysis assumes the ISO is independent of protected status and that the individual scoring function is fair and truthful.Source: §3.2 · p. 5
  • The algorithmic screener reaches the optimal solution in both formulations, whereas fatigue can cause the human-like screener to choose the wrong candidates.The paper states that relaxing its assumptions about the ISO and truthful scoring worsens these results.Source: §3.2 · p. 5

4 An Experimental Framework

The framework uses Monte Carlo simulations to compare algorithmic and human-like screeners across candidate pools, screening orders, and search procedures. Results show that order, score distributions, selectivity, and fatigue shape how closely selections match the full-search baseline.

4 An Experimental Framework

  • The framework is a flexible R-based Monte Carlo simulation tool for studying different screening scenarios with either screener under a given ISO.The authors present it as a way to inform practitioners without running real-world screening scenarios.Source: §4 · p. 5

4.1.1 Generating the sample.

  • Score pools vary from symmetric, where top candidates are rare, to asymmetric and increasing, where top candidates are more common; ψ controls good-k selectivity.The ISO is either independent of candidate scores or correlated with them; ρ = −1 represents descending score order.Source: §4.1.1 · pp. 5–6

4.2 Experiments without Fatigue

  • Without fatigue, good-k approaches best-k as screening becomes more selective when top scores are rare, and a negatively correlated ISO makes the solutions coincide at ρ = −1.With more top candidates, partial search can still miss the best candidates; increasing k/n improves good-k’s approximation of best-k.Source: Figure 2 · p. 6, §4.2 · p. 7

4.1 Setup

  • Experiments use 10,000 runs per parameter set, with n = 120, k = 6, and q = 0.5 in Sections 4.2 and 4.3; unsolvable runs are discarded.The sample comprises candidate scores, an ISO, and protected-status indicators; good-k may be unsolvable when too few candidates meet ψ.Source: §4.1.1 · pp. 5–6

4.1.2 Fatigued scores. •

  • The fatigue experiments compare centered error with increasing variance against negatively biased error with smaller variance, while assuming truthful candidate evaluation.The study evaluates solutions against the algorithmic best-k baseline using utility ratio and candidate-set overlap.Source: §4.1.2 · p. 6

4.3 Experiments with Fatigue

  • With a human-like screener, fatigue reduces utility in some settings: high selectivity with few top scores is vulnerable to variable fatigue, while adequate top scores or lower fatigue variability can preserve good-k utility.For the symmetric distribution, fatigue also lowers best-k utility; at ψ ≥ 0.5, good-k with fatigue has better utility than best-k with fatigue.Source: Figure 3 · p. 7, §4.3 · p. 8

5 Conclusion

The paper formalizes ISO effects through best-k and good-k formulations, compares algorithmic and human-like screeners, and supplies a flexible simulation framework. It concludes that search formulation and screener behavior matter, while noting that its functional assumptions constrain the results.

5 Conclusion

  • The paper formalizes best-k and good-k search, introduces a human-like screener, and provides a flexible simulation framework for studying ISO effects.The analysis identifies fairness and optimality impacts associated with position bias for both algorithmic and human-like screeners.Source: §5 · p. 8
  • The conclusions are limited by the chosen utility models and screener formulations, especially the human-like fatigue model.The authors identify alternative utility models and fatigue terms as directions for future work.Source: §5 · p. 8
  • Practitioners should first clarify the search formulation, then assess how the screener behaves while searching the ISO.The paper states these takeaways apply regardless of screener type.Source: §5 · p. 8

Ethical Considerations

The authors report no ethical challenges in drafting the paper and state that the results use simulated data. They also emphasize that candidate screening is human-dependent and should not be treated solely as an automated decision problem.

Ethical Considerations

  • The authors report that the paper’s results are based on simulated data and that no company data was used.They say their earlier collaboration with company G followed its ethical guidelines and concluded with an internal report discussed with stakeholders.Source: Ethical Considerations · p. 9
  • The authors report receiving no monetary compensation from company G and state that the views expressed are their own.Source: Ethical Considerations · p. 9
  • The authors distinguish human-like from algorithmic screening to emphasize human users, not to endorse automated screening.They describe candidate screening as a human-centered process.Source: Ethical Considerations · p. 9

A Collaborating with G

The collaboration with G inspired stylized facts about candidate screening, but the paper was not a G-specific deliverable and used no G data. The observations focused on HR’s second hiring phase, where officers reduce the pool to suitable candidates.

A Collaborating with G

  • The four-month hybrid collaboration used interviews, screening-session shadowing, and access to Taleo to understand HR officers’ tasks and practices.The resulting stylized facts were intended as inspiration for the ISO problem, not hard facts from an observational study.Source: A Collaborating with G · p. 11
  • Five HR officers screened technical-role applications under time constraints, often handling pools of hundreds and multiple pools with similar deadlines.The officers included screeners for roles such as data scientist and front-end developer.Source: A Collaborating with G · p. 11
  • No G data was used in the paper, which was not a collaboration deliverable.A separate report to G’s Advanced Analytics team formalized its screening process and assessed fairness implications and automation risks.Source: A Collaborating with G · p. 11
  • The collaboration examined phase two of hiring, when HR reduces the candidate pool to suitable candidates using minimum basic requirements.The described pipeline also included building the pool in phase one and interviewing selected candidates in phase three.Source: A Collaborating with G · p. 11

B.1 Additional Discussion on the Utility Model in Section 2.2.2

The discussion examines how utility choices affect solutions to the good-k problem. With only a minimum quota, utility-maximizing solutions can differ in their protected-group fractions, and the saved-effort alternative does not capture the intended search behavior.

B.1 Additional Discussion on the Utility Model in Section 2.2.2

  • With f(S_k) ≥ q, the good-k utility model can have multiple solutions with different protected-group fractions.In the example, the two solutions have fractions 0.5 and 1.Source: §B.1 · p. 11
  • The alternative utility (9) measures saved effort as the number of candidates after the last screened candidate, but is unsuitable for the intended good-k problem.It favors finding k eligible candidates quickly, yet does not represent the variant that continues evaluating non-protected candidates after their quota is met to keep the best candidates found so far.Source: §B.1 · pp. 11–12
  • In the example, both {c1,c3} and {c2,c3} maximize utility (9) and satisfy quota q, although the discussion questions why c2 should be considered after c1 meets the minimum requirement.The paper introduces the penalty function in utility model (5) rather than considering the continued-search variant.Source: §B.1 · pp. 11–12

B.2 The Two Search Procedures under a Human-Like Screener

The human-like versions of both search procedures incorporate fatigued candidate scores. The fatigue component is stochastic, and the algorithms can accommodate alternative formulations for its distribution.

B.2 The Two Search Procedures under a Human-Like Screener

  • HumanExaminationSearch and HumanCascadeSearch incorporate the fatigued-scores formulation into the two search procedures.They are updates to ExaminationSearch and CascadeSearch for the human-like screener.Source: §B.2 · p. 12
  • In both algorithms, ε is a random variable that depends on the human-like screener’s accumulated fatigue Φ.The paper discusses ε1 and ε2 as possible models.Source: §B.2 · p. 12
  • Other ε formulations are compatible if a random variable is drawn each time the screener evaluates a candidate, using the intended fatigue-dependent probability distribution.This requirement applies to both human-like algorithms.Source: §B.2 · p. 12

C Additional Experiments

These additional experiments vary the representational quota under score and ISO independence. They find that the quota changes protected representation but not the relative strength of the best-k and good-k solutions.

C Additional Experiments

  • The additional figures and discussion analyze the algorithmic screener while varying the quota parameter q.The experiments consider the quota after it had been set to 0.5 with protected-group fraction pr=0.2.Source: C Additional Experiments · p. 12
  • For ISO, the average protected-group fraction in the best-k and good-k solutions is min{q, pr}; Figure 5 (left) illustrates this result for good-k.Source: C Additional Experiments · p. 13
  • Under θ ⊥⊥ s, the quota q does not affect the relative strengths of the best-k and good-k solutions.Figure 5’s center and right panels show RtB across quotas in two experimental settings.Source: C Additional Experiments · p. 13
Loading 2307.15398v5…