Source-linked AI summary
Tutorial on logistic-regression calibration and fusion: Converting a score to a likelihood ratio
Geoffrey Stewart Morrison
TL;DR
Forensic comparison systems produce scores that indicate relative evidential support but are not themselves interpretable as log likelihood ratios. The paper provides a practical, low-mathematical-complexity tutorial on logistic-regression calibration and fusion for converting scores and combining parallel systems. It also describes applications in forensic voice comparison and fingerprint/fingermark comparison.
Problem
Scores indicate relative support between same-origin and different-origin hypotheses, but their absolute values are not interpretable as log likelihood ratios.
Method
The paper explains logistic-regression calibration and fusion from a practical forensic-likelihood-ratio perspective with minimal mathematical complexity.
Results
The tutorial describes calibration for converting scores to log likelihood ratios and fusion for converting parallel score sets from multiple forensic-comparison systems to log likelihood ratios.
Takeaways & Limitations
The procedures are presented as applicable beyond forensic voice comparison, including an example using fingerprint/fingermark scores.
Takeaways & Limitations
The presentation assumes familiarity with the likelihood-ratio framework and the basics of calculating a likelihood ratio.
Abstract
from arXiv · showhide
Logistic-regression calibration and fusion are potential steps in the calculation of forensic likelihood ratios. The present paper provides a tutorial on logistic-regression calibration and fusion at a practical conceptual level with minimal mathematical complexity. A score is log-likelihood-ratio like in that it indicates the degree of similarity of a pair of samples while taking into consideration their typicality with respect to a model of the relevant population. A higher-valued score provides more support for the same-origin hypothesis over the different-origin hypothesis than does a lower-valued score; however, the absolute values of scores are not interpretable as log likelihood ratios. Logistic-regression calibration is a procedure for converting scores to log likelihood ratios, and logistic-regression fusion is a procedure for converting parallel sets of scores from multiple forensic-comparison systems to log likelihood ratios. Logistic-regression calibration and fusion were developed for automatic speaker recognition and are popular in forensic voice comparison. They can also be applied in other branches of forensic science, a fingerprint/fingermark example is provided.
Introduction
The paper tutorials logistic-regression calibration and fusion as likelihood-ratio calculation steps, motivated by the need for an accessible forensic perspective. Scores reflect similarity and population typicality but require conversion before their values can be interpreted as log likelihood ratios.
- Scope: The tutorial presents logistic-regression calibration and fusion as potential steps in calculating forensic likelihood ratios.Calibration converts scores, while fusion combines parallel scores from multiple forensic-comparison systems.
- Scores: Scores indicate relative support for same-origin versus different-origin hypotheses while incorporating similarity and population typicality.Higher-valued scores provide more support for same-origin than lower-valued scores.
- Scores: Absolute score values are not interpretable as log likelihood ratios, so an additional conversion step is required.The ordering of scores is informative, but their absolute support and differences are not directly quantified as likelihood ratios.
- Tutorial aim: The paper addresses a lack of introductory, forensic-oriented descriptions by explaining calibration and fusion practically with minimal mathematical complexity.It assumes readers understand the likelihood-ratio framework and basic likelihood-ratio calculation.
- Tutorial structure: The tutorial develops likelihood-ratio calculation from single-Gaussian models before addressing multiple offender data points and score conversion.It also discusses applications to other score-generation methods and provides a fingerprint/fingermark example.
More complex models
More complex probability-density models are needed when the data are not well represented by a single Gaussian. Multiple offender data points create a separate problem because the desired likelihood ratio should characterize the whole recording rather than isolated portions.
- More complex models: Single-Gaussian models can be extended to multivariate data or replaced with arbitrary, potentially complex probability-density models.Gaussian-mixture model–universal background model approaches are identified as common in forensic voice comparison.
- More complex models: Figure 2 illustrates likelihood-ratio calculation using Gaussian-mixture suspect and background models.The example is univariate, but the procedure also applies to multivariate data.
- Multiple data points: Multiple data points are common in forensic voice comparison and make a single likelihood ratio for the whole offender recording necessary.Sequentially calculating likelihood ratios for individual portions does not provide the desired recording-level strength of evidence.
- Multiple data points: Averaging raw data before calculating a likelihood ratio can fail with multimodal or skewed models because the mean may fall in a low-density trough.Two offender values may each align with suspect-model peaks while their mean aligns with a trough.
- Multiple data points: Multiplying individual likelihood ratios would require independence, but data points from the same or overlapping recording sections are expected to be highly correlated.This correlation makes a naïve-Bayes combination inappropriate as a general solution.
Calculating a score
A score summarizes multiple data-point likelihood ratios by averaging their logarithms, but it remains only log-likelihood-ratio-like. Calibration is required before the score can be interpreted as a likelihood ratio.
- Score construction: A typical solution is to define the score as the mean logarithm of the individual data-point-specific likelihood ratios.The score aggregates N likelihood ratios associated with data points from the offender recording.
- Score construction: The logarithm base may be chosen freely, with natural logarithms often used for computation and base-ten logarithms usually preferred for presenting results.The choice of base changes representation rather than the stated aggregation procedure.
- Interpretation: A higher score gives greater relative support for same-origin than a lower score, but neither the absolute score nor score difference has a direct likelihood-ratio interpretation.The score preserves ordering of evidential support without quantifying its magnitude.
- Interpretation: Scores therefore require another conversion step before they become interpretable likelihood ratios.This conversion is the role of calibration discussed in the subsequent procedure.
Converting a score to a likelihood ratio using single-Gaussian models
A score-to-likelihood-ratio procedure can train same-origin and different-origin score models and evaluate their density ratio for a test score. Equal-variance Gaussian score models connect this procedure to a linear relationship in logged-odds space.
- Procedure: Scores can be treated as univariate summaries of relationships between samples before conversion to likelihood ratios.The paper first introduces a single-Gaussian procedure as a precursor to logistic-regression calibration.
- Procedure: Training uses labeled same-origin and different-origin scores to fit single-Gaussian models constrained to share a pooled within-group variance.A test comparison is scored and its value is evaluated under both models to obtain a density ratio.
- Figure 4: Figure 4 maps equal-variance score densities to a probability curve and a straight line in logged-odds space.The figure uses circles for same-origin training scores and triangles for different-origin training scores.
- Model assumptions: Equal-variance constraints are imposed for score models, whereas raw-data models are intended to represent different between-speaker and within-speaker variability.The raw-data modeling rationale concerns population variability versus suspect-specific variability.
- Score models: The score models differ from raw-data models because relative score values encode modeled similarity and typicality between samples.Higher scores support same-origin more strongly, while lower scores support different-origin more strongly.
Mapping between single Gaussians and logistic regression
Logistic regression can be mapped directly from equal-variance Gaussian score models, but it uses equal priors to make logged posterior odds equal log likelihood ratios. The conversion therefore remains valid for forensic likelihood-ratio calculation without changing trial priors.
- Mapping from Gaussian models: The Gaussian score model and logistic regression can be mapped directly because the equal-variance Gaussian relationship is linear in logged-odds space.The Gaussian procedure models category distributions, whereas logistic regression models the boundary between categories.
- Equal priors: Equal priors simplify the posterior-probability relationship so that logistic-regression output can be equated with the logarithm of a likelihood ratio.With equal priors, prior odds are 1, making log likelihood ratio equal log posterior odds.
- Mapping from Gaussian models: Logistic regression models the probability of the same-origin hypothesis given a score, with same-origin and different-origin training scores coded as 1 and 0.The model is trained in logged-odds space using maximum likelihood.
- Equal priors: Equal priors used during score calibration are independent of the priors a trier of fact assigns when combining a likelihood ratio with case hypotheses.The calibration procedure does not constrain the trier of fact’s choice of trial priors.
Logistic regression is more robust
Logistic regression is preferred because it models the category boundary rather than the full score distributions, making it more robust to violations of equal-variance and Gaussian assumptions.
- Robustness mechanism: Logistic regression is less affected by training scores far from the category boundary because it models the boundary’s shape rather than each category’s full distribution.A very high same-origin score can substantially alter the Gaussian model while having almost no effect on logistic regression.
- Practical preference: Logistic regression is more robust to violations of equal-variance and Gaussian-distribution assumptions than the equal-variance Gaussian procedure.This robustness is the reason the paper prefers logistic regression for score-to-likelihood-ratio conversion.
- Practical preference: Logistic regression remains a legitimate likelihood-ratio procedure because it is analogous to a generative model whose category-density ratio can be calculated.The analogy connects discriminative boundary modeling with likelihood-ratio calculation.
Converting a score to a likelihood ratio using logistic-regression calibration
Calibration converts scores into interpretable log likelihood ratios through a linear relationship learned from training data. Shifts alter the intercept, scaling alters the slope, and real data commonly require both adjustments.
- Calibration relationship: The calibrated relationship between a score and its log likelihood ratio is linear in logged-odds space.Training estimates the intercept α and slope β, after which test scores can be converted easily.
- Calibration relationship: A perfectly calibrated system has log(LR) = 0 + 1 × s, so its scores already equal log likelihood ratios.This corresponds to α = 0 and β = 1.
- Shifting: A one-unit leftward shift in training scores yields log(LR) = 1 + 1 × s, requiring one unit to be added during calibration.The shift changes the intercept while leaving the slope at 1.
- Scaling: Doubling within-group variance halves the slope of the logged-odds line to 0.5.The increased variance makes the probability curve shallower and changes the score-to-log-likelihood-ratio scaling.
- Combined effects: Real training data commonly produce both shifting and scaling rather than perfect equal-variance Gaussian relationships.The combined shift-and-scaling example also changes the separation and steepness of the fitted relationship.
Application to scores generated in other ways
Logistic-regression calibration can also convert scores produced by methods other than the paper’s Gaussian construction, provided the scores are suitable for forensic interpretation. Applications include model-based, human-listener, and fingerprint-related systems.
- Scope of application: Logistic-regression calibration applies beyond the motivating Gaussian score procedure when suitable training scores are available and the scores capture relevant forensic information.The paper presents calibration as having wider application than the initial multi-data-point likelihood-ratio problem.
- Examples: Calibration has been applied to multivariate-kernel-density scores from an acoustic-phonetic voice-comparison system when modelling assumptions may be violated.The cited work used formant-trajectory information from vowel recordings.
- Examples: Human-listener similarity judgments have also been scaled as logged odds and converted to likelihood ratios using logistic-regression calibration.The resulting human-listener systems were compared with automatic systems.
- Forensic requirement: Forensic calibration requires scores to account for typicality as well as similarity, because similarity alone does not determine evidential strength.Highly similar samples can provide different support depending on whether that similarity is typical or atypical in the relevant population.
Logistic-regression fusion
Logistic-regression fusion combines parallel scores from multiple forensic-comparison systems and converts them into calibrated likelihood ratios. Training estimates coefficient weights for a weighted combination of scores.
- Each forensic-comparison system must produce a score for every training and test comparison, creating parallel score sets for fusion.
- Fusion can combine systems differing in signal processing, modelling techniques, or the phonetic units they analyse.
- Multivariate logistic regression combines parallel scores while accounting for correlations between systems.
- The trained model produces calibrated likelihood ratios by applying coefficient-weighted scores to parallel test scores.
- In the illustrative two-score example, fusion weights the dimensions to minimise overlap between same-origin and different-origin training scores.
Example of logistic-regression calibration in forensic voice comparison
The voice-comparison example applies logistic-regression calibration to scores from vowel-formant measurements. Training-derived coefficient weights convert test scores into log likelihood ratios, illustrated with a Tippett plot.
- The example analyses recordings from 64 male Standard Chinese speakers recorded on two occasions about one week apart.
- Scores based on /a/ vowel-formant measurements and MVKD were used to train calibration coefficients and convert test scores to log likelihood ratios.
- Cross-validation separated background, training, and test comparisons despite using data from a single database.
- Figure 9 compares pre-calibration scores with post-calibration log likelihood ratios using a Tippett plot.
Example of logistic-regression calibration in forensic fingerprint/finger-mark comparison
The fingerprint/finger-mark example recalibrates existing system outputs by treating their logarithms as scores. Calibration removed bias toward same-origin support and improved the interpretability of the outputs as likelihood ratios.
- The example applies logistic-regression calibration to fingerprint/finger-mark system outputs from Neumann, Evett, and Skerret.
- The system’s reported likelihood-ratio outputs were logarithmized and treated as scores, with cross-validation used because no separate training database was available.
- Figure 10 shows pre- and post-calibration Tippett plots for the 9-minutiae system, whose relative performance changed from worst to best.
- Calibration removed the bias toward greater same-origin support observed in all systems.
- Bias generally increased with the number of minutiae used by the system.
- Calibration can improve likelihood-ratio estimation because score-level modelling uses one dimension and a simple model with fewer parameters to estimate.
Example of logistic-regression fusion in forensic voice comparison
The voice-fusion example combines scores from five vowel-phoneme systems. The paper presents the resulting individual-system and fused-system performance, while stressing that calibration and fusion depend on training data matching the relevant population and test conditions.
- Five forensic voice-comparison systems were constructed, one for each of five vowel phonemes, and parallel score sets were calculated.
- The example compares the best-performing individual-vowel systems with their fusion.
- Figure 11 uses dashed lines for a single /ɔɪ/ system and solid lines for the fusion of five single-vowel systems.
- Calibration and fusion quality depends on training data representing the relevant population and the suspect and offender recording conditions.
- Mismatched training data may produce inaccurate log-likelihood-ratio results for test comparisons.