PaperlayerView exact revision on arXiv

Source-linked AI summary

A Fairness Audit of the Duckworth-Lewis-Stern Method: Format-Specific and Gender-Differential Bias, with an Interpretable Calibration Layer for Cricket Target Revision

Soumyadeep Roy

arXiv:2609.04754v1cs.LG

TL;DR

DLS has long been used for rain-interrupted target revision, but its prediction bias has not been audited at large scale, especially across genders. The paper audits Cricsheet matches with synthetic interruptions and temporal splits, then adds an interpretable calibration layer. DLS-Cal reduces overall bias, while its gender-aware variant sharply reduces women’s ODI residual bias.

  • Problem

    The paper addresses the lack of large-scale empirical evidence on DLS prediction bias, including whether one resource table miscalibrates women’s and men’s cricket differently.

  • Method

    The study audits DLS on synthetic interruption scenarios from Cricsheet, benchmarks alternatives, and adds DLS-Cal as an interpretable state-conditioned correction.

  • Results

    DLS-Cal reduces overall bias by 31% on ODI and 19% on T20I, while its gender-aware variant reduces women’s ODI residual bias from +6.19 to +0.65 runs.

  • Takeaways & Limitations

    The results support calibration as a lightweight way to reduce structured DLS bias, including the documented gender differential, within the evaluated scope.

  • Takeaways & Limitations

    The audit relies on synthetic interruptions, reconstructs a published rather than proprietary ICC resource table, and does not directly measure operational target-setting error.

Abstract

from arXiv · show

The Duckworth-Lewis-Stern (DLS) method has been the international standard for revising target scores in rain-interrupted limited-overs cricket since 1999. Despite over two decades of operational use, no large-scale empirical audit of its prediction bias has been published. We conduct such an audit on 8,150 international matches (3,095 ODIs, 5,055 T20Is) from Cricsheet, generating 233,550 synthetic interruption scenarios with temporal splits. We document two structured biases. First, DLS prediction error spans a 137-run range across (overs-remaining, wickets-lost) match-state buckets. Second, DLS exhibits a gender-differential bias on ODIs that has not previously been quantified: on the training split, mean over-prediction is +1.51 runs for men but +7.63 runs for women, a gap of +6.13 runs (F = 195.16, p < 10^-43). We benchmark DLS against five modern alternatives: Bi-LSTM, XGBoost, an enriched XGBoost variant, a deep context-aware model, and a stacking ensemble, and propose DLS-Cal, a lightweight interpretable calibration layer (27K parameters) outputting a state-conditioned correction added to DLS. DLS-Cal reduces absolute bias by 31% on ODI and 19% on T20I, and a gender-aware variant reduces women's ODI residual bias from +6.19 to +0.65 runs while leaving men's calibration unchanged. We release code, models, and data.

1 Introduction

The paper audits DLS for structured prediction bias at scale, focusing on match-state variation and previously unquantified gender differences. It proposes an interpretable calibration layer to address these biases.

  • Audit motivation and contributions: DLS uses one resource table for men’s and women’s cricket despite documented differences in scoring patterns, leaving its gender calibration empirically unresolved.The paper identifies this as a distinct concern from prior critiques of format mismatch and player equivalence.
  • Audit motivation and contributions: Cricsheet’s ball-by-ball coverage enables a larger audit than prior studies, which mainly proposed model extensions or addressed outcome classification rather than revised-target prediction.The paper contrasts its target-revision focus with smaller or differently framed prior machine-learning comparisons.
  • Audit motivation and contributions: A 137-run bias span across match-state buckets provides the paper’s first systematic large-scale empirical audit of DLS prediction error.The audit uses Cricsheet data and tests whether errors vary systematically across states.
  • Audit motivation and contributions: +6.13 runs is the reported gender-differential gap in mean DLS over-prediction between women’s and men’s ODIs on the training split.The contribution is described as the first quantitative documentation of this gender-differential bias, with robustness checks across inference, team strength, and time.
  • Audit motivation and contributions: DLS-Cal is a lightweight interpretable calibration layer, accompanied by a gender-aware variant that reduces the documented gender bias.The calibration contribution is presented as an added layer rather than a replacement for DLS.
  • Audit motivation and contributions: Win-Flip Rate (WFR) is introduced as a fairness-oriented thresholded error metric for cricket target revision.The metric is listed among the paper’s analytical contributions.

2 Data and Setup

The study uses public Cricsheet ball-by-ball data, synthetic interruption points, and temporal splits to evaluate remaining-runs prediction across a broad match-state space. It qualifies this as an audit of the resource-estimation component underlying target revision rather than direct operational target-setting error.

  • Dataset: 8,150 international matches comprise the corpus: 3,095 ODIs and 5,055 T20Is.The dataset is filtered to first innings of at least 30 overs for ODIs or 15 overs for T20Is.
  • Dataset: 5,708 men’s matches and 2,442 women’s matches are included in the corpus.These counts describe the gender composition of the international-match dataset.
  • Target-revision interpretation: The audit evaluates remaining-runs prediction as a resource-estimation component proportional to revised targets, not operational target-setting error directly.Held-out real rain-affected matches are described as the closest approach to direct operational evaluation.
  • Synthetic interruption sampling: Synthetic interruption points require at least 5 overs of context and 5 overs of remainder, with the post-interruption runs as the supervision target.The sampling scheme is chosen for state-space coverage rather than to reproduce the operational distribution of interruptions.
  • Synthetic interruption sampling: 233,550 synthetic interruption examples are generated, split into 140,160 training, 24,300 validation, and 69,090 test examples.Synthetic sampling provides broad coverage of match states because genuine rain-interrupted matches are too few for direct training.
  • Temporal evaluation: A temporal split assigns matches through 2022 to training, 2023 to validation, and 2024–March 2026 to testing.The split is designed to prevent future-match leakage, and player-level features use training-period statistics only.

3 DLS Bias Audit

The audit finds that DLS errors are structured across match states and formats, with especially pronounced gender-differential bias in ODIs. The ODI gender gap remains after clustered inference, bucket matching, implementation checks, and team-strength stratification.

  • Overall bias: DLS’s mean training-set prediction error is +4.79 runs, indicating systematic over-prediction overall.
  • Match-state bias: Bias ranges from −111 to +26 runs in ODIs and −19 to +25 runs in T20Is, producing a 137-run span across match-state buckets.Between-bucket structure is statistically distinguishable from uniform noise in both formats.
  • Match-state bias: The largest absolute biases occur in ODI collapse-state buckets, while T20I over-prediction is largest with 5–10 overs remaining and 6 wickets lost.The worst ODI bucket is −74.4 runs; the worst T20I bucket is +25.1 runs.
  • Gender-differential bias: On ODIs, mean bias is +1.51 runs for men and +7.63 runs for women, a gender gap of +6.13 runs.The gap also differs within matched match-state buckets, including a +26.1-run example.
  • Robustness checks: The ODI gender gap remains positive under match-level bootstrap inference, bucket matching, and Full-Member-only stratification.The match-clustered 95% CI is [+3.70, +10.07] runs; the Full-Member-only gap is +6.72 runs.
  • Format specificity: The gender effect is format-specific: the T20I gap is +0.51 runs and is not distinguishable from zero under clustered inference.The clustered T20I 95% CI is [−1.10, +2.12] runs, whereas the ODI finding persists across implementations.

4 Methods Compared

The paper compares DLS variants, machine-learning forecasters, and an interpretable calibration layer, while introducing WFR to capture thresholded target errors relevant to chase outcomes.

  • DLS variants: Three DLS variants include the original published formulation, a training-period recalibration of Z0, and a canonical resource-ratio formulation blending current run rate with a format baseline.The canonical variant uses α = 0.7, selected on validation data.
  • Machine-learning baselines: The machine-learning baselines comprise a 2-layer Bi-LSTM, XGBoost, XGBoost-Plus with player and interaction features, CAP-Net v2, and a non-negative stacking ensemble.XGBoost-Plus is treated as a richer forecaster and the ML methods as an upper bound rather than drop-in DLS substitutes.
  • DLS-Cal: DLS-Cal adds a learned correction Δ(x) to the DLS prediction using an 18-dimensional input and a 27,265-parameter MLP initialized to zero correction.Its inputs include the DLS prediction, bucket-level training bias, and 16 enriched features.
  • DLS-Cal: Gender-specific DLS-Cal models route inference to male-only or female-only calibrators trained with the same architecture and hyperparameters as the gender-blind model.This design directly addresses the documented gender-bias finding.
  • Evaluation: Win-Flip Rate measures the proportion of predictions whose absolute error exceeds a threshold k, complementing RMSE and MAE for chase-target evaluation.The paper reports WFR at k ∈ {3, 5, 10, 15}, while noting that it remains a proxy rather than a direct measurement of flipped outcomes.

5 Results

On the test split, machine-learning methods outperform DLS variants overall, while DLS-Cal provides the strongest ODI WFR5 result and an interpretable calibration alternative; ablations support a lightweight learned layer.

  • Main comparison: The test-set comparison covers all evaluated methods on N = 69,090 examples, with lower values preferred for every metric.Gender-aware DLS-Cal results are reported separately by gender.
  • Main comparison: The stacking ensemble achieves the lowest overall RMSE of 21.49 and MAE of 16.19, while all machine-learning methods outperform every DLS variant.Replacing original DLS with the canonical formulation reduces RMSE from 37.41 to 28.92, a 22.6% reduction.
  • Main comparison: DLS-Cal records the lowest ODI WFR5 at 83.0%, beating XGBoost at 84.0% and stacking at 85.0%, while trailing stacking on RMSE by 2.23 runs.The result combines a thresholded outcome-oriented metric with an interpretability advantage.
  • Main comparison: Gender-aware DLS-Cal matches gender-blind DLS-Cal on overall accuracy while improving fairness on women’s matches.The gender-aware results are stratified in the paper’s separate fairness analysis.
  • Calibration ablation: A bucket-mean lookup worsens accuracy, a linear correction recovers most gains, and the MLP improves only modestly beyond it, showing that a lightweight learned layer suffices.The bucket-bias prior is retained for interpretability rather than aggregate accuracy.

5.3 Bias Reduction

DLS-Cal reduces structured DLS bias across formats, with Figure 3 showing corrections aligned to the audited bias map and residual structure remaining after calibration.

  • Format-level reduction: 31.0%: ODI mean absolute bias falls from 33.74 to 23.29 runs after applying DLS-Cal.This is the reported reduction in mean |bias| on ODI.
  • Format-level reduction: 18.8%: T20I mean absolute bias falls from 20.58 to 16.71 runs after applying DLS-Cal.The reduction is smaller than the ODI reduction but remains substantial.
  • Figure 3: The correction panel visually mirrors the DLS-bias panel, while the residual panel shows the remaining unexplained bias after calibration.Residual structure remains particularly relevant for interpreting what the calibration does not remove.
  • Figure 3: Figure 3 places ODI in the top row and T20I in the bottom row, with DLS bias, DLS-Cal corrections, and residual bias shown across columns.Cells with fewer than 30 examples are omitted.
  • Real interrupted matches: On 417 real D/L-decided matches, DLS under-predicts ODI runs by 11.1 on average and over-predicts T20I runs by 8.7, corroborating the audit’s format-specific pattern.The real-match evaluation reports RMSE values of 45.8 for ODI and 25.1 for T20I.

5.5 Closing the Gender Bias Gap

Gender-aware DLS-Cal preserves the directional gender difference seen in test-period DLS bias while sharply reducing women’s residual bias. The result is stable across temporal windows and uses separately trained, gender-routed calibrators.

  • The directional finding—higher residual bias for women than men after level adjustment—is preserved on the held-out test split.Test-period canonical biases differ from training-period values because scoring rates continued to rise.
  • The gender-aware model trains separately on men’s and women’s data and routes each inference example to the matching calibrator.
  • +6.19 to +0.65 runs: gender-aware DLS-Cal reduces women’s residual bias by 5.54 runs.Women’s RMSE also improves from 22.92 to 22.11 relative to gender-blind DLS-Cal.
  • +6.20 runs: the mean gender gap across 2021–2025 holdout windows is positive in four of five windows.The 2023 window is marginally negative, while the largest-sample 2021 window is individually significant.

5.7 Calibration

Reliability diagrams show DLS canonical systematically over-predicting, whereas XGBoost, Stacking, and DLS-Cal remain close to perfect calibration. Across interruption phases, machine-learning methods reduce WFR most substantially in late innings.

  • DLS canonical sits above the reliability diagonal, while XGBoost, Stacking, and DLS-Cal track it closely.Each marker represents a quantile bin of predicted remaining runs; the dashed diagonal denotes perfect calibration.
  • Late interruptions: DLS-Cal reduces WFR to 74.1% versus 86.5% for DLS canonical.The passage reports more than a 10-percentage-point WFR reduction for all machine-learning methods in this regime.
  • Mid interruptions: DLS-Cal records 80.2% WFR versus 85.8% for DLS canonical.
  • Early interruptions: DLS-Cal records 85.9% WFR versus 87.7% for DLS canonical.

6 Discussion

The discussion frames gender-aware calibration as a transparent mitigation while emphasizing trade-offs, data constraints, and limits of the audit and modeling choices. It also identifies residual structure and governance considerations for future deployment.

  • 6.1 The Gender Finding and Its Implications: Women’s residual bias falls from +6.19 to +0.65 runs with gender-aware DLS-Cal, without degrading men’s calibration.The authors present this as a tractable mitigation requiring no change to the underlying DLS framework.
  • 6.2 Interpretability vs. Accuracy: DLS-Cal wins on ODI WFR but trails Stacking on RMSE, trading some accuracy for predictions decomposable into DLS plus a state-attributable correction.The discussion assigns interpretability independent value in governance contexts involving public scrutiny.
  • 6.3 Why CAP-Net Underperformed: CAP-Net’s underperformance is attributed to cold-start effects from 4,707 players, especially the long tail with fewer than 50 recorded balls.A controlled ablation restricted to players with at least 100 balls is left for future work.
  • 6.4 Policy Considerations for Governing Bodies: A fully independent women’s resource table requires a sufficiently large, era-current corpus that does not yet exist, making gender-aware calibration a lower-data-cost interim route.The authors present these governance choices as considerations rather than prescriptions.
  • 6.5 Limitations: The audit uses synthetic interruptions because genuine rain-interrupted matches are too few and unevenly distributed, although conclusions are validated on real rain-affected matches.This design is explicitly treated as an approximation of operational DLS use.
  • 6.5 Limitations: Residual structure remains in late-innings, high-wicket states, suggesting that closing the gap may require features beyond overs, wickets, format, and current run rate.

7 Conclusion

The paper concludes that DLS has state-dependent and gender-differential bias, and proposes DLS-Cal as an interpretable correction layer. The gender-aware variant substantially reduces women’s residual bias while preserving the existing DLS infrastructure.

  • 137-run range: DLS prediction bias varies across match-state buckets, while the ODI gender gap reaches +6.13 runs on the training split.
  • 31% on ODI and 19% on T20I: DLS-Cal reduces overall absolute bias across formats.
  • +6.19 to +0.65 runs: the gender-aware variant reduces women’s ODI residual bias.It requires no change to existing DLS infrastructure and is proposed as a transparent calibration layer.
  • The gender finding survives clustered inference, Full-Member matching, temporal windows, and validation on real rain-affected matches.

Reproducibility Statement

The study supports reproducibility through public data, released artifacts, and fixed experimental procedures.

  • Public Cricsheet data underpin the study.
  • Source code, trained checkpoints, training logs, and the audit dataset are released online.
  • All experiments use random seed 42 and a temporal train/validation/test split.
Loading 2609.04754v1…