PaperlayerView exact revision on arXiv

Source-linked AI summary

The BOSARIS Toolkit: Theory, Algorithms and Code for Surviving the New DCF

Niko Brümmer, Edward de Villiers

arXiv:1304.2865v1stat.APcs.LGstat.ML

TL;DR

Very low-error evaluation makes error-rate estimates unreliable and requires large trial sets, while large score files create computational challenges. The paper presents toolkit methods for evaluating and calibrating likelihood-ratio systems across operating points, including normalized Bayes-error plots and efficient computation. These methods provide practical references for calibration and identify regions where small-sample effects may undermine meaningful evaluation.

  • Problem

    Error-rate estimates become unreliable when observed errors are scarce, creating challenges for evaluating systems across operating points and processing large trial sets.

  • Method

    The paper explains BOSARIS Toolkit algorithms for sweeping operating points, plotting normalized Bayes error rates, computing DCF and minDCF efficiently, and calibrating scores with isotonic regression.

  • Results

    The toolkit's plots use dev and eval databases, provide Bayes-error and error-type curves, and mark regions where false alarms or misses become scarce.

  • Takeaways & Limitations

    Normalized Bayes-error plots offer a practical reference for judging calibration and database adequacy across operating points.

  • Takeaways & Limitations

    ROC-based criteria do not test calibration, and reducing ROCCH-EER does not guarantee lower minDCF across all operating points.

Abstract

from arXiv · show

The change of two orders of magnitude in the 'new DCF' of NIST's SRE'10, relative to the 'old DCF' evaluation criterion, posed a difficult challenge for participants and evaluator alike. Initially, participants were at a loss as to how to calibrate their systems, while the evaluator underestimated the required number of evaluation trials. After the fact, it is now obvious that both calibration and evaluation require very large sets of trials. This poses the challenges of (i) how to decide what number of trials is enough, and (ii) how to process such large data sets with reasonable memory and CPU requirements. After SRE'10, at the BOSARIS Workshop, we built solutions to these problems into the freely available BOSARIS Toolkit. This paper explains the principles and algorithms behind this toolkit. The main contributions of the toolkit are: 1. The Normalized Bayes Error-Rate Plot, which analyses likelihood- ratio calibration over a wide range of DCF operating points. These plots also help in judging the adequacy of the sizes of calibration and evaluation databases. 2. Efficient algorithms to compute DCF and minDCF for large score files, over the range of operating points required by these plots. 3. A new score file format, which facilitates working with very large trial lists. 4. A faster logistic regression optimizer for fusion and calibration. 5. A principled way to define EER (equal error rate), which is of practical interest when the absolute error count is small.

1 Introduction

The BOSARIS Toolkit is a MATLAB toolkit for calibrating, fusing, and evaluating binary-classifier scores, developed for speaker recognition with wider biometric and forensic applicability. The document explains its theory, algorithms, and implementation.

  • The toolkit provides MATLAB code for calibrating, fusing, and evaluating scores from automatic binary classifiers.
  • Although developed for automatic speaker recognition, the code is intended to apply more broadly to biometric and forensic likelihood-ratio calibration.
  • The document serves as a user guide covering the toolkit’s theory, algorithms, and code, complementing the user manual.
  • The toolkit’s core implementation originated in the authors’ ABC submission to NIST’s 2010 Speaker Recognition Evaluation and was later developed collaboratively at the BOSARIS Workshop.

2 Theory

The theory section establishes the framework for understanding the BOSARIS Toolkit, from sampling and Bayes decision theory through evaluation criteria, score calibration, fusion, and related measures.

  • The theory section provides the framework needed to understand what the toolkit does and why.
  • Subsection 2.1 addresses running out of errors, while subsections 2.2 and 2.3 review Bayes decision theory and NIST’s DCF criterion.
  • Subsections 2.4 and 2.5 evaluate likelihood-ratio outputs using theoretically optimal Bayes thresholds and develop practical evaluation criteria.
  • Subsection 2.6 examines relationships among ROC/DET curves, EER, and minDCF.
  • Subsection 2.7 discusses methods for fusing and calibrating scores.

2.1 Sampling effects

Sampling effects make error-rate estimates unreliable when observed errors become scarce, so the toolkit marks low-error regions and recommends databases large enough to provide at least 30 errors of each type.

  • At some operating points, observed errors become so scarce that error-rate estimates are unreliable regardless of system accuracy or database size.
  • Confidence-interval and credible-interval analyses can quantify estimate accuracy, but their results depend on modelling assumptions.
  • Doddington’s Rule of 30 recommends at least 30 errors for a probably approximately correct error-rate estimate under independent Bernoulli trials.
  • The toolkit flags plot regions where error counts fall below 30 and cautions users about interpreting results beyond those flags.
  • For SRE’10’s new operating point, participants addressed scarce false alarms by manufacturing more non-target trials, whose possible count grows quadratically with speaker count.
  • Calibration and evaluation databases should each yield at least 30 misses and 30 false alarms at every operating point of interest.

2.2 Bayes decision theory

Bayes decision theory maps score evidence and prior information into accept or reject decisions, while the toolkit distinguishes uncalibrated score potential from calibrated likelihood-ratio performance.

  • The canonical detection problem makes independent accept/reject decisions for trials under mutually exclusive target and non-target hypotheses.
  • Scores conventionally favour the target hypothesis when larger and the non-target hypothesis when smaller.
  • Trial outcomes pair a hypothesis with a decision, with misses and false alarms designated as the two error outcomes.
  • Bayes decisions compare risks formed from costs and posterior probabilities conditioned on the score and prior information.
  • The calibration mapping converts scores into log-likelihood-ratios, which preserve the score direction but support Bayes decisions at the theoretical threshold.
  • The toolkit evaluates uncalibrated score potential, trains calibration mappings, and evaluates calibrated log-likelihood-ratios for Bayes decisions.

2.3 DCF: criterion for goodness of hard decisions

DCF evaluates the goodness of hard accept/reject decisions using empirical miss and false-alarm rates at a fixed operating point. The operating point uses a synthetic target prior and costs known to the evaluee.

  • DCF evaluates hard accept/reject decisions submitted for supervised trials.NIST’s SREs required both decisions and scores, with DCF serving as the primary criterion for hard decisions.
  • Evaluees may threshold calibrated log-likelihood-ratios at the Bayes decision threshold or tune a system-dependent threshold on supervised calibration data.
  • The empirical error rates are Pmiss, the ratio of misses to target trials, and Pfa, the ratio of false-alarms to non-target trials.
  • π is a synthetic target prior that models the application domain and need not equal the target proportion in the evaluation database.
  • The DCF operating point is specified by π, Cmiss, and Cfa, and the evaluation recipe requires it to be fixed and known to the evaluee.

2.4 Bayes Risk: criterion for goodness of log-likelihood-ratios

Bayes risk evaluates submitted log-likelihood-ratios by letting the evaluator make decisions across thresholds and operating points. The criterion supports calibration assessment because sufficiently calibrated systems can outperform a default system across all operating points.

  • Evaluators can apply the Bayes decision threshold to submitted log-likelihood-ratios instead of accepting hard decisions from the evaluee.
  • Sweeping the evaluator’s threshold η maps Pmiss(η) and Pfa(η) into the system’s ROC/DET curve.
  • The empirical Bayes risk is computed from log-likelihood-ratios over all supervised evaluation trials, with target and non-target indices identifying the trial classes.
  • When the operating point is fixed, a shifted score can make Bayes risk numerically equal to DCF.
  • When η is swept, calibration depends on score shifting, scaling, and finer details of the calibration mapping.
  • The evaluation criterion’s proper-scoring-rule interpretation rewards both discrimination and probabilistic calibration.
  • Sufficiently well-calibrated likelihood-ratios achieve risk no greater than the default zero-output system at every operating point.If a system fails this comparison at an operating point, better Bayes decisions are obtained on average by not using it.
  • The three-parameter Bayes-risk space, π, Cmiss, and Cfa, can be represented by an equivalent one-dimensional range that is easier to cover in evaluation.The reparametrization uses an effective prior with unit miss and false-alarm costs while preserving the decision threshold and relative system comparisons.

2.5 Empirical Bayes error-rate: a practical evaluation recipe

The toolkit evaluates calibrated likelihood-ratios across operating points using empirical Bayes error-rate, reference systems, and normalized plots that expose calibration quality and data-sufficiency limits.

  • Empirical Bayes error-rate: The empirical Bayes error-rate E(L|π˜) evaluates likelihood-ratio decisions at a single operating point parameterized by π˜ or threshold −logit π˜.The operating point is one-dimensional, enabling evaluation across a range of thresholds.
  • Reference criteria: Emin provides an ideal lower-bound reference for calibration and a calibration-insensitive measure of potential decision performance across operating points.When E and Emin are close, calibration is considered very good.
  • Reference criteria: A system is badly calibrated where its empirical Bayes error-rate exceeds the default reference E(L0|π˜), because ignoring the system would yield better Bayes decisions.This comparison uses the default system as a reference for calibration failure.
  • Operating-point sweeps: The toolkit sweeps operating points either by integrating them into the scalar Cllr criterion or by plotting error-rate against operating point.The plot uses logit π˜ horizontally and normalizes vertically by E(L0|π˜) to keep low-error regions visible.
  • Normalized Bayes-error-rate plots: Normalized Bayes-error-rate plots compare true and miscalibrated systems, with DR30 markers identifying regions where fewer than 30 misses or false-alarms make evaluation unreliable.The safe evaluation region lies between the miss and false-alarm DR30 boundaries; outside it, small-sample effects can distort comparisons.
  • Normalized Bayes-error-rate plots: For SRE’10 speaker-recognition scores, the plots distinguish development and evaluation databases, actual and minimum Bayes error-rates, and miss and normalized false-alarm contributions.The new operating point is marked at π˜ = 0.001, or threshold −6.91; normalization exposes calibration effects across a wider range than APE-curves.

2.6 ROC/DET and related criteria for goodness of scores

ROC/DET curves, minDCF, and EER provide complementary views of score quality, with calibration distinguishing attainable performance from actual decisions. The ROC convex hull supports efficient evaluation across operating points and motivates ROCCH-EER and unequal-error-rate criteria.

  • ROC/DET and calibration: ROC/DET curves evaluate uncalibrated scores, whereas actual Bayes performance also depends on selecting calibrated thresholds.The ROC can be optimistic because calibration is not tested; actual performance is usually no better than minDCF.
  • The ROCCH is where minDCF lives: The empirical ROC is a discrete steppy curve, while minDCF operating points lie on its lower-left convex-hull boundary, the ROCCH curve.The ROCCH is continuous, piecewise linear, and convex between (0, 1) and (1, 0).
  • The ROCCH is where minDCF lives: The ROCCH vertex set is typically much smaller than the empirical ROC, enabling efficient minDCF computation across many operating points.The toolkit computes the ROCCH and its DET representation; the convex hull can be computed efficiently with the PAV algorithm.
  • EER as upper bound: ROCCH-EER interpolates the ROCCH at Pmiss = Pfa, providing a well-defined practical EER when the empirical ROC lacks an exact equal-error point.Figure 4 demonstrates the equality of ROCCH-EER and max minDCF.
  • EER as upper bound: ROCCH-EER is a tight upper bound on the Bayes error-rate achievable with perfect calibration, but reducing it need not improve minDCF at every operating point.The maximizing operating point can shift laterally, allowing error-rates to increase far from the former maximum.
  • UER: Unequal-error-rate: Unequal-error-rate criteria target selected operating-point regions by fixing an error-rate ratio, such as r = ˜π, rather than requiring Pmiss = Pfa.Varying r maps out the ROCCH curve, and the resulting value forms a tight upper bound on a concave function of π.
  • UER: Unequal-error-rate: ROCCH concavity gives the curve a stiffness property: optimizing one point tends to improve decision-making over a larger operating-point region.This relationship supports generalized criteria while retaining awareness that EER optimization alone does not guarantee uniform improvement.
  • PRBEP: PRBEP is intended for comparing systems on the same database, not for meaningful comparisons across databases of different sizes.ROCCH interpolation can also make the reported value non-integer when expressed as an absolute error count.

2.7 Fusion and Calibration

BOSARIS supports supervised calibration through non-parametric isotonic regression and parametric logistic regression, with logistic regression also enabling fusion. The methods differ in training behavior, test-data performance, and assumptions about score distributions.

  • Calibration methods: BOSARIS provides isotonic-regression and logistic-regression solutions for mapping scores to log-likelihood-ratios.Both mappings are trained on supervised calibration data; logistic regression also generalizes to fusion.
  • Non-parametric calibration: On training data, isotonic calibration is simultaneously optimal for any sensible calibration objective.The stated objectives include proper scoring rules and Bayes risk criteria.
  • Non-parametric calibration: PAV calibration yields DCF equal to minDCF when the same dataset is optimized and evaluated.The procedure also corresponds to using ROCCH slopes as calibrated likelihood-ratios.
  • Method comparison: Isotonic calibration is invariant to monotonic score warping, whereas logistic regression works best for approximately normal score distributions.This contrasts the distributional robustness of the non-parametric method with the parametric method’s stated preference.
  • Parametric fusion: Logistic regression trains combination weights to fuse multiple subsystems into a single output of calibrated log-likelihood-ratios.It can also incorporate certain side-information or quality measures.

3 Algorithms

The toolkit uses sorting, ROC convex-hull computation, PAV, and large-scale convex optimization to process DCF, minDCF, calibration, and fusion efficiently. For million-scale score files, the reported implementations reduce several-minute computations to seconds.

  • Efficient evaluation: For large trial lists, BOSARIS computes DCF and minDCF in a few seconds instead of the several minutes required by traditional implementations.The comparison concerns evaluation across the operating points needed for normalized Bayes error-rate plots.
  • DCF computation: The toolkit computes DCF by jointly sorting scores and operating-point thresholds, then tracking threshold positions to obtain error rates.The miss and false-alarm rates are computed from threshold positions and the numbers of target and non-target scores.
  • minDCF computation: minDCF is computed from ROCCH vertices obtained with PAV, because replacing the full ROC with these vertices preserves minDCF.The method exploits the typically small number of ROCCH vertices.
  • PAV algorithm: PAV assigns monotonically ordered likelihood-ratios and is simultaneously optimal for proper scoring rules and Bayes risk criteria.Its uses include ROCCH construction, minDCF, EER, UER, PRBEP, and non-parametric calibration.
  • Computational complexity: PAV is linear after sorting, while sorting has complexity O(T log(T)); processing a few million scores takes a few seconds.The complexity statement separates the linear PAV step from the preceding sort.
  • Optimization: Logistic-regression fusion and calibration use trust-region Newton conjugate gradient optimization for large-scale unconstrained minimization.The toolkit describes the optimizer as faster and better behaved than the predecessor’s conjugate-gradient optimizer.

4 Code

The code provides MATLAB APIs and platform-independent binary score formats for organizing, processing, plotting, calibrating, and evaluating large trial collections. Binary storage substantially improves feasibility and speed for multi-million-trial development data.

  • API: The toolkit’s MATLAB implementation uses an object-oriented API to organize its functions.The object-oriented structure is described as an organizational choice rather than an essential property of the code.
  • File format: HDF5 binary files improve interoperability while avoiding the size and processing costs of text files.The toolkit also provides MATLAB .mat versions and conversion functions between binary and text formats.
  • Data objects: Indexes describe model–test-segment trial pairs, while keys additionally identify target and non-target trials.Scores store values for trials specified by an index or key, together with the index information.
  • Data objects: Quality measures are represented as scores for models or test segments and can be fused with ordinary trial scores.The fusion of quality measures with ordinary scores is referenced in the toolkit’s fusion section.
  • Score manipulation: The toolkit supports score alignment, subset selection, merging non-overlapping score objects, and external score production using index files.These operations support managing structured score collections and selected trial subsets.
  • Plots and evaluation: Its plotting functions include DET and normalized Bayes error-rate plots, actual and minimum Bayes-error curves, miss and false-alarm contributions, operating points, and DR30 markers.The code also includes functions for EER, minimum DCF, actual DCF, PRBEP, and effective prior.
  • File-format performance: For approximately eight million SRE’10 development trials, one binary file was about 60 times smaller and loaded about 160 times faster than text.The example concerns a tel-tel development file.
  • Score representation: The binary score format represents scores as model-by-test-segment matrices with a logical validity matrix and accompanying name lists.The matrix position identifies the trial formed by a model and test segment pair.
Loading 1304.2865v1…