PaperlayerView exact revision on arXiv

Source-linked AI summary

FAIR: Fairness-Aware Information Retrieval Evaluation

Ruoyuan Gao, Yingqiang Ge, Chirag Shah

arXiv:2106.08527v2cs.IR

TL;DR

Fairness-aware IR lacks a standard evaluation measure that captures both user utility and fair information exposure. The paper proposes FAIR, which integrates standard IR and fairness measures, and a ranking algorithm that optimizes it; the authors report that FAIR reflects both dimensions and that their algorithm improves fairness while closely approximating ideal utility.

  • Problem

    Traditional IR metrics capture relevance, diversity, and novelty but not fair exposure, while fairness metrics do not adequately account for user utility.

    Source: §1 · pp. 1–2

  • Method

    FAIR integrates a standard IR utility metric with a fairness measure, and the authors develop a ranking algorithm that jointly optimizes the integrated metric.

    Source: §1 · p. 2, §3.1 · p. 4

  • Results

    The authors report that FAIR reflects utility, fairness, and their balance, while their algorithm improves fairness and closely approximates ideal utility.

    Source: §1 · p. 2, §7 · p. 12

  • Takeaways & Limitations

    FAIR is presented as a way to evaluate fairness-aware rankings across utility and fairness dimensions, with an algorithm that does not require gold-standard relevance judgments.

    Source: §1 · p. 2

    The metric needs further testing across datasets and applications to establish whether it captures user models in retrieval and recommendation tasks.

    Source: §6 · p. 11

Abstract

from arXiv · show

With the emerging needs of creating fairness-aware solutions for search and recommendation systems, a daunting challenge exists of evaluating such solutions. While many of the traditional information retrieval (IR) metrics can capture the relevance, diversity, and novelty for the utility with respect to users, they are not suitable for inferring whether the presented results are fair from the perspective of responsible information exposure. On the other hand, existing fairness metrics do not account for user utility or do not measure it adequately. To address this problem, we propose a new metric called FAIR. By unifying standard IR metrics and fairness measures into an integrated metric, this metric offers a new perspective for evaluating fairness-aware ranking results. Based on this metric, we developed an effective ranking algorithm that jointly optimized user utility and fairness. The experimental results showed that our FAIR metric could highlight results with good user utility and fair information exposure. We showed how FAIR related to a set of existing utility and fairness metrics and demonstrated the effectiveness of our FAIR-based algorithm. We believe our work opens up a new direction of pursuing a metric for evaluating and implementing the FAIR systems.

1 | INTRODUCTION

Search rankings can overrepresent prominent aspects, while user clicks may reinforce those rankings and limit minority-aspect discovery. Existing utility and fairness measures do not provide a unified way to assess both dimensions, motivating FAIR and a ranking algorithm that optimizes them together.

1 | INTRODUCTION

  • Prominent aspects can dominate results, and click-driven ranking can reinforce that imbalance, leaving users with skewed discovery of minority aspects.The introduction describes this as a positive feedback loop and vicious cycle of unfairness.Source: §1 · p. 1
  • Fairness and diversity have distinct aims: diversity serves user needs, while fairness constrains the exposure subtopics receive from the system.They can compete; a ranking may provide poor exposure to a minority subtopic and still be fair.Source: §1 · p. 2
  • Existing metrics assess individual factors but lack a standard measure of overall performance that captures both fairness and user utility.The paper argues that this gap makes algorithm comparisons difficult.Source: §1 · p. 2
  • The authors propose FAIR, a metric combining fairness and user utility, and report that it highlights results and algorithms that perform well on both dimensions.Their experiments also show FAIR penalizes algorithms skewed toward one dimension.Source: §1 · p. 2
  • The authors develop a FAIR-based ranking algorithm that does not require gold-standard relevance judgments and report improved fairness while closely approximating ideal utility.The introduction presents this as a detailed case-study result.Source: §1 · p. 2

2 | RELATED WORK

The related work distinguishes user-oriented utility and diversity metrics from fairness measures, then positions FAIR as a unified evaluation metric and its algorithm as direct optimization of that metric.

2.1 | Evaluation metrics in IR

  • Traditional IR metrics assess relevance, diversity, and novelty for user utility, but do not measure fair information exposure.Examples include MAP, DCG, RBP, α-nDCG, and intent-aware metrics.Source: §2.1 · p. 3

2.2 | Fairness notions and measures

  • Topical aspects are treated as sensitive groups, connecting fairness measurement with the diversity context of IR.The paper distinguishes user-serving diversity from socially oriented fairness in information exposure.Source: §2.2 · p. 3
  • Fairness metrics vary in their criteria, and FAIR integrates fairness with standard IR measures rather than measuring fairness alone.Its fairness component can use distribution-, ratio-, or error-based measures.Source: §2.2 · p. 3
  • FAIR is presented as a static-ranking evaluation metric, unlike related work focused on optimization rewards or stochastic rankings.It is intended for use cases that do not assume stochastic ranking models.Source: §2.2 · p. 3

2.3 | Fairness ranking algorithms

  • Prior ranking approaches re-rank items or regularize objectives, while FAIR's algorithm directly optimizes its integrated metric to balance utility and fairness.The algorithm is also described as generalizable to recommendation settings.Source: §2.3 · p. 3

3 | FAIRNESS-AWARE IR (FAIR) METRIC

FAIR combines a standard IR utility score with a fairness component and normalization, with design choices that adapt the metric to different utility models and fairness notions.

3 | FAIRNESS-AWARE IR (FAIR) METRIC

  • FAIR combines standard IR utility with a fairness measure to evaluate ranking performance without assuming utility and fairness are always competing or independent.The metric is designed to balance trade-offs when the dimensions are orthogonal and optimize both when they are not.Source: §3 · pp. 3–4

3.1 | Overall design

  • The overall design uses a rank-specific IR metric and a position-discounted fairness term based on the distance between observed and desired group distributions.The normalization term M is an ideal score used to bring the metric into [0,1].Source: §3.1 · p. 4

3.2 | Encoding utility

  • For diversity and novelty, the utility component can use α-nDCG, which models query aspects as distinct information needs.Other utility choices include MAP, RBP, ERR-IA, and NRBP, depending on the intended focus.Source: §3.2 · p. 4, §3.1 · p. 4

3.3 | Encoding fairness

  • The fairness component discounts the reciprocal of KL divergence by rank, so fairer rankings better approximate optimal α-nDCG.The paper cautions that position discounts should not be duplicated when the IR metric already includes one.Source: §3.3 · p. 5

3.4 | Capturing different fairness notions

  • The desired group distribution can encode parity, relevance-proportional treatment, or demographic parity, depending on the selected fairness notion.Parity gives subtopics equal exposure; demographic parity can match their proportions in the candidate collection.Source: §3.4 · pp. 5–6

4 | FAIRNESS-AWARE RERANKING ALGORITHM

FAIR ε-greedy re-ranks retrieved or candidate items by alternating between utility-oriented and fairness-oriented choices. The method supports different trade-offs and can operate without gold-standard relevance labels under stated assumptions about the default ranking.

4 | FAIRNESS-AWARE RERANKING ALGORITHM

  • FAIR ε-greedy is a re-ranking algorithm for top-k results that optimizes the FAIR metric.The paper uses an α-nDCG-based FAIR example and notes that an RBP-based version is also possible.Source: §4 · p. 6
  • The algorithm alternates between choosing documents for diversity-and-novelty gain and exploring documents that minimize KL divergence.Its ε parameter controls the balance between these choices; ε = 0 is greedy FAIR optimization, while ε = 1 greedily minimizes KL divergence.Source: §4 · p. 6
  • The method does not require gold-standard relevance labels, but assumes default-ranking positions indicate relevance or utility.For recommendations, top-ranked items are assumed to have higher utility, such as clickthrough rate or purchase probability.Source: §4 · p. 6

5 | EXPERIMENTS

The experiments compare FAIR with existing metrics, examine metric behavior in a case study, and evaluate a FAIR ε-greedy ranking algorithm.

5 | EXPERIMENTS

  • The experiments examine how FAIR relates to commonly used utility and fairness metrics.They also include a case study of how FAIR changes when different metrics are optimized and evaluate the FAIR ε-greedy algorithm.Source: §5 · pp. 6–7
  • The FAIR ε-greedy algorithm is evaluated as part of the experiments.The section describes this evaluation after the metric comparisons and case study.Source: §5 · p. 7
  • The experimental section includes a case study examining how FAIR changes when different metrics are optimized.Source: §5 · p. 7

5.1 | Comparison between FAIR and other metrics

The experiments compare FAIR across datasets and ranking algorithms, finding that it reflects both utility and fairness and distinguishes cases where those dimensions diverge.

5.1.1 | Datasets

  • The experiments use Google Search, ClueWeb09, and MovieLens; MovieLens contains 10,000 users, 100 movies, and 46,515 interactions at about 95.35% sparsity.Google Search has 100 queries and 7,410 webpages, while ClueWeb09 covers 50 topics with 3–8 subtopics each.Source: §5.1.1 · p. 7

5.1.2 | Fairness constraint and algorithms

  • The fairness setting uses demographic parity, with the desired group distribution set to each query's or user's full collection distribution.The study evaluates Google ranking, DetCons, and FOE, using rating scores in FOE for recommendation.Source: §5.1.2 · p. 7

5.1.3 | Evaluation metrics

  • The evaluation includes FAIR, nDCG, RBP, KL, and nDRKL at k = 10, 20, and 50.nDCG and RBP represent utility, KL measures distributional difference, and nDRKL modifies nDKL using reciprocal KL to emphasize fair results at earlier ranks.Source: §5.1.3 · p. 8

5.1.4 | Results

  • MovieLens's extreme sparsity coincided with the lowest utility and FAIR scores, while Google and Google+DetCons had higher FAIR scores than ClueWeb09 despite its slightly better relevance.On ClueWeb09, DetCons had nDCG and RBP scores of 1.Source: §5.1.4 · p. 8, Table 2 · p. 9, Table 1
  • At larger rank cutoffs, FAIR correlated more with nDCG and RBP; at top ranks, it differed most from individual utility and fairness metrics.The reported pattern indicates a monotonic but nonlinear correlation with the other metrics.Source: §5.1.4 · pp. 8–9, Table 3 · p. 9

5.2 | Case study on FAIR ε-greedy algorithm

The case study compares FAIR ε-greedy with Google, DetCons, and a utility-optimized ranking, examining both metric behavior and the effect of relevance-judgment availability. FAIR scores reward rankings balancing utility and fairness, and the reported comparisons show utility gains alongside reduced bias.

5.2.1 | Algorithms and metrics

  • The case study compared Google, DetCons, greedy IDCG, and FAIR ε-greedy using α-nDCG, KL, nDRKL, minSkew, maxSkew, and feasibility.At ε = 1, FAIR ε-greedy minimized KL-divergence at each rank; IDCG greedily maximized gain G[i].Source: §5.2.1 · p. 9

5.2.2 | Assumption on availability of relevance judgment

  • On Google Search, position-aware FAIR, α-nDCG, and nDRKL algorithms outperformed Google even without relevance judgments; FAIR ε-greedy increased utility while reducing bias.The results were averaged over 1,000 runs to account for FAIR ε-greedy randomization.Source: Figure 1 · p. 10, §5.2.3 · p. 11
  • Google Search comparisons used NoRJ (Google's ranking only) and RJ (gold relevance judgments); both sets of reranked results were evaluated using gold judgments.Google and IDCG were infeasible, while DetCons and FAIR ε-greedy were feasible only through rank 20.Source: §5.2.2 · p. 10, §5.2.3 · p. 10
  • FAIR rewards rankings combining high utility and low bias; high utility or fairness alone need not produce a high FAIR score, as IDCG and DetCons illustrate.The paper notes positive correlation with α-nDCG and nDRKL, while emphasizing that neither utility nor fairness alone determines FAIR.Source: §5.2.3 · p. 11

6 | DISCUSSION

The discussion bounds FAIR's applicability and interpretation: its inputs, normalization, and integrated scores have important constraints, and its fairness and utility notions rely on stated assumptions.

6 | DISCUSSION

  • FAIR currently accepts effectiveness metrics expressible as sums over rankings; unbounded relevance or fairness metrics require future work.The authors say only a limited set of metrics can currently be plugged in, especially variants of precision and diversity- or novelty-based metrics.Source: §6 · p. 11
  • When perfect fairness is unattainable, FAIR's current normalizer still penalizes rankings that differ from the desired distribution.The proposed direction is to normalize using an algorithm that first minimizes the difference from that distribution.Source: §6 · p. 11
  • FAIR is not a one-size-fits-all metric: interdependent utility and fairness measures need careful handling, and known trade-offs are better modeled directly.Scores computed with different utility or fairness definitions should not be compared; equal FAIR scores can also conceal sharply different utility-fairness balances.Source: §6 · p. 11
  • FAIR focuses on group rather than individual fairness and uses relevance as utility under the paper's stated classic assumption.The discussion also uses fairness and lack of bias interchangeably for simplicity.Source: §6 · pp. 11–12

7 | CONCLUSIONS

The conclusion presents FAIR as an integrated metric for evaluating fairness-ranking algorithms and reports that FAIR-based optimization improved utility and fairness over a baseline. The authors frame integrated metrics as decision-support tools, not prescriptions for a fixed trade-off.

7 | CONCLUSIONS

  • FAIR combines relevance, diversity, and novelty with position- and diversity-bias discounting to evaluate fairness-ranking algorithms.The conclusion says it reflects utility, fairness, and their balance, which existing utility or fairness metrics did not achieve together.Source: §7 · p. 12
  • Compared with the baseline algorithm, FAIR-based optimization achieved better utility as well as fairness.Source: §7 · p. 12
  • The authors present integrated metrics as tools for designers and policymakers to understand overall performance and make choices, rather than as prescriptions for a specific trade-off.Source: §7 · p. 12
Loading 2106.08527v2…