PaperlayerView exact revision on arXiv

Source-linked AI summary

Bias Robustness of Genetic Multi-Criteria Sorting versus Learned Rankers under Proxy Skew and Label Contamination

Vladyslav Shevchuk, Carlos Uziel Pérez Malla, Philipp Wissgott

arXiv:2610.14015v1cs.NEcs.AIcs.LG

TL;DR

Ranking systems may align with undesignated proxies or contaminants when these enter as features or training labels. The paper compares label-free genetic sorting by declared criteria with label-trained rankers under proxy-skew and label-contamination protocols. Genetic sorting shows lower proxy alignment, while contamination sweeps leave its bias curves flat; the authors frame it as a robustness option, not a universal nDCG winner.

  • Problem

    Rankers can overweight structural correlates or absorb popularity, clicks, and position leaked into training grades, motivating comparison of methods under both proxy skew and label contamination.

    Source: §I · p. 2

  • Method

    The paper compares label-free Pikaia genetic sorting of fixed candidate pools by declared criteria C with label-trained LTR under clean-label proxy-skew and contaminated-label protocols.

    Source: §I · p. 2, C. Proxy-skew protocol · p. 4

  • Results

    On WANDS-PS, proxy correlation is 0.59/0.78/0.87 for LightGBM, XGBoost, and RankSVM versus approximately 0.15–0.16 for Genetic AI; contamination evidence shows flat Genetic AI curves and rising LTR bias.

    Source: B. Proxy skew and ablation · p. 5, §V · p. 7

  • Takeaways & Limitations

    Genetic multi-criteria sorting is an architectural option for optimizing declared attributes on a fixed pool when undesignated proxies or label contaminants must be refused.

    Source: §V · p. 7

    Query counts are moderate (n≈30–50), so the results are controlled evidence rather than large-scale online evaluation; demographic protected-attribute fairness is out of scope.

    Source: §V · p. 7

Abstract

from arXiv · show

Learning-to-rank (LTR) models can align rankings with undesignated signals when those signals appear as features or leak into training labels. We compare label-free genetic multi-criteria sorting (open Pikaia strategy variants), which reranks fixed candidate pools using an explicit criteria vector C, to label-trained LTR (LightGBM, XGBoost, RankSVM) under two controlled protocols on MSLR-WEB10K, Baidu-ULTR, WANDS, and Expedia. In the main proxy-skew study, labels remain clean while a calibrated proxy P is available as a feature: Genetic AI rankings show lower Spearman rank-proxy correlation than the LTR peers (e.g. about 0.15-0.16 vs. 0.59-0.87 on WANDS), with both families evaluated jointly on clean nDCG@10 on a bias-utility frontier. In a supporting label-contamination study, training labels are mixed with contaminant B: because Genetic AI methods do not use labels, their bias and utility curves stay flat, whereas LightGBM and XGBoost increase rank-B correlation as contamination rises (OLS bias_slope about 0.58-0.65 on WANDS/Baidu). The results support genetic sorting as an architectural option for optimizing declared attributes on a fixed pool when undesignated proxies or label contaminants must be refused.

I. INTRODUCTION

Ranking can track undesignated proxies or absorb contaminants leaked into training labels, even when evaluation measures only retrieval utility. This paper compares label-free sorting by declared criteria with label-trained rankers and defines bias relative to the task’s declared target.

I. INTRODUCTION

  • Clean-label tree and linear rankers can overweight structural correlates, while label-trained LTR can absorb popularity, clicks, or position leaked into grades.The paper distinguishes proxy skew from label contamination as separate failure modes.Source: §I · p. 2
  • The main question is whether refusing to fit proxy features or biased labels improves the bias–utility trade-off versus label-trained LTR.Genetic multi-criteria sorting reranks a fixed candidate pool using an explicit criteria vector C, without training on relevance labels.Source: §I · p. 2
  • Prior work spans unbiased LTR, societal bias in neural rankers, and fair exposure allocation, but evolutionary methods are rarely tested as bias-robust alternatives under controlled proxy and contamination protocols.The paper uses two studies with shared candidate lists and metrics, identifying proxy refusal as its main claim.Source: §I · p. 2
  • The proxy-skew study uses clean labels to compare default Genetic AI configurations with LightGBM, XGBoost, and RankSVM, including proxy-removal utility ablations.The supporting contamination study varies training-label contamination; Genetic AI rank–B correlation stays flat while LTR bias rises and clean utility falls.Source: §I · p. 2
  • The comparison uses primary LTR peers and secondary controls on five listwise benchmarks, pairing every bias claim with clean-label nDCG@10 on a bias–utility frontier.Benchmark-specific undesignated signals and default criteria designations are listed in Table I.Source: §I · p. 2, Table I · p. 2

A. Defining bias versus signal

  • Bias means ranking dependence on an undesignated proxy P or contaminant B relative to declared criteria C; P may correlate with clean relevance yet remain outside C.Both signals are recorded for evaluation, and ranking–signal correlation is considered alongside clean nDCG@10.Source: A. Defining bias versus signal · p. 2

II. RELATED WORK

Related work covers correction for position-biased feedback, societal bias, and fair exposure; this paper’s contamination study instead mixes a measurable undesignated signal into training grades.

II. RELATED WORK

  • Unbiased LTR addresses position bias with inverse propensity scoring and related estimators.The paper distinguishes this online-feedback correction literature from its study of contaminant B mixed into grades.Source: A. Bias in learning to rank · p. 3
  • Popularity bias in recommenders and ranking is described as amplifying head items.The passage separates this concern from the paper’s label-contamination protocol.Source: A. Bias in learning to rank · p. 3
  • The label-contamination study tests whether label-free ranking avoids absorbing B mixed into training grades, rather than testing online propensity correction.Source: A. Bias in learning to rank · p. 3

B. Societal bias and fair ranking

The paper distinguishes undesignated-signal robustness from demographic fairness and situates its method among evolutionary and multi-criteria rankers. It evaluates fixed candidate pools using clean-label utility and separate proxy or contaminant bias measures.

B. Societal bias and fair ranking

  • The study concerns dependence on undesignated signals excluded from the optimization target, not demographic fairness or exposure constraints across groups.Its trade-off framing pairs bias measures with utility.Source: B. Societal bias and fair ranking · p. 3

C. Evolutionary and multi-criteria ranking

  • Genetic AI treats discrete multiobjective decisions as evolutionary games over genes and organisms, and this paper applies the paradigm to listwise reranking with Pikaia strategies.Unlike classical genetic-programming rankers that learn scoring formulas from labels, these methods optimize declared-criteria importance weights at inference time on a fixed pool.Source: C. Evolutionary and multi-criteria… · p. 3

D. Shortcut learning and proxies

A. Problem setting and evaluation splits

  • Each evaluation unit is a query with a fixed pool of roughly 20–100 documents; utility uses clean-label nDCG@10, while bias measures ranking dependence on P or B.Train/test splits are by query; learned rankers fit training queries, while Genetic AI, linear, and RRF apply fixed rules to held-out pools.Source: A. Problem setting and evaluation splits · p. 3
  • The proxy study reserves disjoint 50-query selection and evaluation slices, while the contamination study trains on contaminated labels and evaluates on held-out queries.The reporting rule compares clean nDCG@10 against bias, preferring lower bias at matched utility or higher utility at matched bias.Source: A. Problem setting and evaluation splits · p. 3

B. Rankers and controls

The comparison includes label-trained peers and non-learned controls alongside Pikaia strategy variants. Genetic AI arms share criteria and evaluation settings, varying the strategy pair used to rank candidates.

B. Rankers and controls

  • The shared registry identifies LightGBM, XGBoost, and RankSVM as primary learned peers, with input order, sort-by-B/P, linear weighted sum, RRF, and coordinate ascent as secondary controls.Linear tests explicit criteria without Genetic AI search.Source: B. Rankers and controls · p. 3, Table II · p. 4
  • Pikaia is treated as an open strategy framework in which candidates are organisms, declared criteria are genes, and gene- and organism-level strategies determine fitness scores for ranking.Every strategy pair uses the same iterative replicator updates.Source: B. Rankers and controls · p. 4
  • The four Pikaia variants include DOMINANT–BALANCED, REWARD-HARD, VALUATION-BLEND, and SELL-BUY strategy pairings.All arms share the same criteria vector, preprocessing, and evaluation protocol; only the strategy pair changes.Source: B. Rankers and controls · p. 4
  • Proxy-removal ablations are reported as diagnostics alongside the main results for the four Pikaia strategies.Source: B. Rankers and controls · p. 4

C. Proxy-skew protocol

The proxy-skew protocol reranks shared pools using clean-label learned peers and methods that apply ranking rules at evaluation time. It freezes a criteria vector selected without using the evaluation slice and tests whether rankings depend on proxy P.

C. Proxy-skew protocol

  • All methods rerank the same pools; learned peers train on clean labels, while Genetic AI, linear, and RRF apply their ranking rules at evaluation time.Ablations compare full inputs with no-P inputs using ΔnDCG and rank–proxy correlation.Source: C. Proxy-skew protocol · p. 4
  • C is constructed from attributes with low pairwise correlation to P, so ranking by declared criteria is less likely to resemble ranking by the proxy.The protocol targets moderate proxy-label correlation, approximately 0.45 within a 0.3–0.6 band.Source: C. Proxy-skew protocol · p. 4
  • The criteria vector C is selected once per domain on a selection slice, then frozen for evaluation; Expedia retains its hand-specified C.The protocol filters attributes with |corr(attr, P)| > 0.35, sets polarity and weights from train-split evidence, and selects criteria size by held-out validation nDCG.Source: C. Proxy-skew protocol · p. 4, Table III · p. 4

D. Label-contamination protocol

The label-contamination protocol mixes clean grades with a normalized undesignated signal, trains LTR on the resulting grades, and evaluates all methods against clean labels. It contrasts label-trained methods with methods whose criteria exclude the contaminant.

D. Label-contamination protocol

  • LTR methods train on biased labels while Genetic AI, linear, and RRF use a fixed criteria vector C that excludes B; evaluation measures clean nDCG@10 and dependence on B.Point estimates use contamination 0.4, and sweeps over [0, 1] produce bias and utility curves.Source: D. Label-contamination protocol · pp. 4–5
  • Genetic AI rankings cannot change as contamination varies because these methods do not train on labels.The protocol compares their fixed criteria-based rankings with LTR peers trained on contaminated labels.Source: D. Label-contamination protocol · p. 5
  • For each pool, the protocol mixes clean grade y_i with min–max-normalized contaminant b_i at strength s, then maps the mix to training grades.The sweep uses s ∈ {0, 0.2, 0.4, 0.6, 0.8, 1.0}; s=0 recovers clean grades and s=1 is fully contaminant-driven.Source: D. Label-contamination protocol · p. 5

E. Metrics

  • Utility is measured primarily by nDCG@10 on clean labels, while label-contamination bias is Spearman correlation between the ranking and B.The bias slope is the OLS slope of rank–contaminant correlation against contamination strength; it summarizes the bias curve, not utility.Source: E. Metrics · p. 5

F. Benchmarks

The benchmarks provide fixed listwise candidate pools with per-item attributes, and proxy-skew evaluation uses queries separate from those used to select C. The proxy-label calibration target was met on three domains but not WANDS.

F. Benchmarks

  • Every proxy-skew result uses a disjoint evaluation slice of 50 queries, separate from the 50 queries used to select C.Contamination snapshots use 30 synthetic or 50 benchmark queries.Source: F. Benchmarks · p. 5
  • Mean |corr(P, label)| was 0.462 on Expedia, 0.454 on MSLR, and 0.405 on Baidu, but 0.252 on WANDS, below the intended 0.3–0.6 band.WANDS is therefore a weaker proxy-calibration test, though RankSVM still correlates 0.87 with P there.Source: F. Benchmarks · p. 5

A. Hypotheses

  • The proxy-skew hypothesis is that LTR has higher rank–proxy correlation than Genetic AI under multi-attribute C, while LTR utility is more sensitive to removing P.For label contamination, the hypothesis predicts rising LTR rank–B correlation and falling clean nDCG as contamination rises, while Genetic AI remains stable on both axes.Source: A. Hypotheses · p. 5

B. Proxy skew and ablation

On the primary proxy-skew results, Genetic AI rankings align less with P than clean-label LTR, while conceding clean nDCG on the reported frontier. The utility effect of removing P varies by domain.

B. Proxy skew and ablation

  • Removing P reduces LightGBM nDCG more than Pikaia on MSLR (0.370 vs. 0.123) and Baidu (0.162 vs. 0.114), while WANDS is effectively tied (0.022 vs. 0.024).Expedia reverses the ordering (0.128 vs. 0.316); its narrower criteria set means a forced proxy gene supplies proportionally more signal there.Source: B. Proxy skew and ablation · pp. 5–6
  • On WANDS-PS, rank–proxy correlation is 0.59/0.78/0.87 for LightGBM, XGBoost, and RankSVM versus approximately 0.15–0.16 for Genetic AI; the same ordering holds on MSLR-PS.On the graphical-abstract frontier, Genetic AI keeps the preferred upper-right position but concedes clean nDCG to the three LTR peers.Source: B. Proxy skew and ablation · p. 5, Table IV · p. 6, Figure 2 · p. 6
  • Lower proxy alignment for Genetic AI holds across all four domains, but the stronger utility-ablation pattern holds on only two.Linear tracks Genetic AI closely, pointing to explicit criteria rather than evolutionary search as the active ingredient.Source: B. Proxy skew and ablation · p. 6

C. Label contamination

The contamination sweep leaves Genetic AI rankings unchanged, while label-trained LTR becomes more aligned with contaminant B and loses clean utility as contamination rises. On WANDS and Baidu, OLS bias slopes for LightGBM and XGBoost are about 0.58–0.65.

C. Label contamination

  • Across contamination [0, 1], Genetic AI rankings stay flat on rank–B correlation and clean nDCG@10 (DOM-BAL: 0.136 and 0.771).The methods do not train on labels; the figure shows DOM-BAL flat on both axes.Source: Figure 3 · p. 6, C. Label contamination · pp. 6–7
  • On WANDS, LightGBM rank–B correlation rises 0.126→0.296→0.380 and clean nDCG@10 falls 0.802→0.797→0.716 at contamination 0, 0.4, and 1.0.XGBoost rank–B correlation moves 0.155→0.289→0.370, while clean nDCG@10 moves 0.851→0.803→0.714 on the same grid.Source: Figure 4 · p. 7, C. Label contamination · p. 7
  • OLS bias slopes are about 0.58–0.65 for LightGBM/XGBoost on WANDS/Baidu; at contamination 0.4, Genetic AI nearly matches biased tree utility while roughly halving contaminant correlation.Expedia is reported as a less favorable trade-off.Source: C. Label contamination · p. 7

V. DISCUSSION

The discussion frames genetic multi-criteria sorting as a robustness option for optimizing declared criteria, not a universal nDCG winner. The conclusion and availability statement bound the evidence and identify future work and repository materials.

V. DISCUSSION

  • Genetic sorting occupies a more favorable bias–utility operating point than clean-label LTR, with lower rank–proxy correlation at competitive clean nDCG.Expedia has a utility gap, and some domains show large proxy-skew nDCG gaps versus full LTR; these are not elevated to the main claim.Source: §V · p. 7

VI. CONCLUSION

  • The paper concludes that the evidence supports robustness under refusal of undesignated signals, not universal nDCG dominance, and calls for larger query samples, online evaluation, fairness metrics, and hybrid pipelines.The proxy-skew study is the main result; label-contamination evidence is supporting.Source: §VI · p. 7

DATA AND CODE AVAILABILITY

  • Experiment protocols, criteria configurations, and evaluation scripts are maintained in the accompanying research repository.Proxy-skew criteria vectors are also listed in Appendix B; datasets are used under their respective licenses.Source: DATA AND CODE AVAILABILITY · p. 7

Appendix A: Full result tables

Appendix A supplies full benchmark tables, including domains where Genetic AI sorting pays a utility cost. Appendix B lists the declared proxy-study criteria vectors and their selected polarity.

Appendix A: Full result tables

  • Tables VII and VIII provide all-benchmark results for contamination and proxy-skew studies, including domains with visible Genetic AI utility costs.The tables allow the primary-domain summaries to be checked against the complete benchmark picture.Source: Appendix A · p. 10, Table VII · p. 10, Table VIII · p. 11

Appendix B: Declared criteria vectors for the proxy-skew study

  • WANDS class token match is selected as a COST because its train-split correlation with clean grade is sufficiently negative; it has the vector’s smallest weight.The semantic inference would suggest GAIN, but the reported protocol uses the selected direction rather than a manual override.Source: Appendix B · p. 12
  • Table IX lists proxy-skew criteria vectors selected on the train split; no vector contains P, weights sum to one, and polarity is GAIN unless marked COST.These vectors are used by every non-LTR method on each proxy-skew domain.Source: Table IX · p. 12, Appendix B · p. 12

Related papers

Loading 2610.14015v1…