PaperlayerView exact revision on arXiv

Source-linked AI summary

MASH-Bench: Diagnosing Cross-Source Failure in Mass-Shooting Risk Classification

Neha Sharma, Ritesh Sharma

arXiv:2608.22460v1cs.LGcs.CY

TL;DR

Heterogeneous mass-shooting databases make cross-source risk classification difficult to evaluate and generalize. The paper introduces MASH-Bench and uses LODO, feature masking, domain adaptation, recalibration, and group auditing to diagnose these failures. Performance transfers poorly to GVA, while feature completeness and prevalence differences emerge as central constraints; DANN provides only partial recovery.

  • Problem

    Different database coverage, feature availability, and reporting practices leave cross-source generalization in mass-shooting risk classification insufficiently evaluated.

  • Method

    MASH-Bench harmonizes four databases into 6,968 incidents and evaluates LODO transfer using classifier baselines, feature masking, domain adaptation, recalibration, and per-group auditing.

  • Results

    VeryHigh-risk recall is 0.68–0.89 on curated sources but 0.20 on GVA; feature masking collapses recall to zero, while DANN is the only tested adaptation method producing non-zero GVA signal.

  • Takeaways & Limitations

    Within MASH-Bench, cross-source generalization is constrained more by feature completeness and label prevalence than by classifier choice.

  • Takeaways & Limitations

    The benchmark is not deployable: harmonization and GVA results depend on mapping and imputation choices, all sources retain media-coverage bias, and DANN uses a different base architecture.

Abstract

from arXiv · show

Public mass-shooting databases differ substantially in coverage, feature availability, and reporting practices, creating challenges for machine-learning models that must generalize across data sources. We introduce MASH-Bench, a harmonized benchmark of 6,968 incidents from four U.S. databases: Kaggle, Mother Jones, Stanford MSA, and the Gun Violence Archive (GVA). We evaluate cross-source risk classification using leave-one-dataset-out (LODO) evaluation. Random Forest, XGBoost, and LightGBM achieve VeryHigh-risk recall of 0.68-0.89 on the curated sources but generalize poorly to GVA, where mean recall drops to 0.20 and precision to 0.0004. To investigate the source of this degradation, we conduct a controlled feature-masking ablation that removes the five features unavailable in GVA from the curated sources. The resulting recall collapse to zero provides evidence that feature completeness is a major contributor to the observed cross-source failure. We further evaluate three domain-adaptation approaches: DANN, CORAL, and importance weighting. DANN improves VeryHigh-risk recall on GVA by 0.282 (95% CI [0.11, 0.47], p = 0.003), although precision remains low, whereas CORAL and importance weighting yield zero recall. Oracle prior-shift recalibration likewise fails to recover VeryHigh-risk predictions, indicating that label-side correction alone is insufficient under the observed feature deficiencies. A per-group audit further identifies substantial disparities associated with media-attributed mental-health labels. Overall, these results indicate that, in MASH-Bench, cross-source generalization is constrained more by feature completeness and label prevalence than by classifier choice. The benchmark provides a controlled setting for diagnosing these effects in cross-source risk classification.

I. INTRODUCTION

MASH-Bench addresses limited systematic evaluation of cross-source mass-shooting risk classification by harmonizing four databases and diagnosing transfer failures. The study combines LODO benchmarking, controlled feature masking, domain adaptation, dependence-aware inference, and group auditing.

  • VeryHigh-risk recall reaches 0.68–0.89 on curated sources but falls to 0.20 recall and 0.0004 precision when transferring to GVA.
  • 6,968 incidents from four public U.S. databases form MASH-Bench, using a common 12-variable schema for cross-source risk classification.
  • Masking the five features unavailable in GVA collapses curated-source recall to zero, providing evidence that feature completeness contributes to transfer failure.
  • DANN, CORAL, and importance weighting are compared under a common cross-source protocol to characterize domain-adaptation benefits and failure modes under feature heterogeneity.
  • A per-group audit examines VeryHigh-risk recall and precision across media-attributed mental-health-label groups, identifying substantial differences in prediction behavior.MASH-Bench is intended as a diagnostic benchmark rather than a deployable risk-prediction system.

II. RELATED WORK

Prior work has largely emphasized within-source analyses, while cross-source studies show that performance can degrade under heterogeneous data collection. MASH-Bench combines harmonization with controlled analyses of feature completeness, label shift, domain adaptation, and group disparities.

  • Prior mass-shooting machine-learning studies largely analyze individual sources and do not directly test transfer across heterogeneous incident databases.
  • Cross-source evaluation in adjacent domains shows that within-source performance can overestimate performance after transfer to differently collected data.
  • Public mass-shooting databases differ in definitions and inclusion criteria, motivating a common representation for cross-database evaluation.
  • CORAL, importance weighting, and DANN address different forms of distribution shift through statistic alignment, example reweighting, and adversarial representation learning.
  • Source-level prevalence differences make class-imbalance handling important when interpreting recall and precision for rare VeryHigh-risk outcomes.
  • MASH-Bench extends prior tools by combining feature harmonization with controlled feature-masking, label-shift, domain-adaptation, and per-group analyses.

A. Sources

MASH-Bench harmonizes four heterogeneous public databases into a shared schema and defines severity-based risk strata without using outcome variables as predictors. GVA differs structurally through broader lower-severity coverage and missing participant-level features.

  • MASH-Bench expands the benchmark to 6,968 incidents by adding GVA to Kaggle, Mother Jones, and Stanford MSA.
  • GVA records incident_area, gender, race, mental_health, and age as “Unknown” for every incident because participant-level fields are unavailable.
  • GVA has lower mean casualty counts than curated sources, with 5.07 victims versus 7.53–13.80, reflecting broader lower-severity coverage.
  • The shared schema contains 12 variables spanning source, incident context, demographics, mental health, and multiple shooters.
  • Risk labels are Low, Medium, High, and VeryHigh, derived from total_victims using rule, standard-deviation, or quartile thresholds.Under LODO, distribution-derived thresholds use training-fold statistics, and risk denotes severity strata rather than future-attack occurrence.
  • Quartile stratification is excluded from four-source experiments because pooled boundaries make the Medium class empty.

D. Empirical Distributional Shift

The study measures source heterogeneity and evaluates classifiers and domain-adaptation methods under LODO with leakage controls. GVA shows strong marginal shifts and missing participant features, while joint shift alone does not explain transfer failure.

  • D. Empirical Distributional Shift: Wasserstein-1, total-variation, and MMD2 quantify numeric, categorical, and joint-distribution shifts across six source pairs.
  • D. Empirical Distributional Shift: 90.5% of Kaggle incidents match Stanford MSA on year, fatalities, and injured, rising to 93.1% within their shared time window.
  • D. Empirical Distributional Shift: GVA has the largest marginal shifts against curated sources, with Wasserstein 7.77–9.41 and total variation 0.48–0.69.
  • D. Empirical Distributional Shift: The largest joint MMD2 is Kaggle–Mother Jones at 0.243 rather than a GVA pair, so joint distance alone does not explain GVA transfer failure.
  • Experimental Setup: The contextual and full feature sets differ in available demographic and incident attributes, while classifiers include Random Forest, XGBoost, and LightGBM.
  • Experimental Setup: DANN uses adversarial domain training, whereas CORAL aligns covariance statistics and importance weighting reweights source examples.
  • Experimental Setup: LODO trains on three sources and tests on the fourth across five seeds, while held-out data is excluded from label construction, fitting, oversampling, and threshold estimation.

C. Inference Protocol

The protocol uses paired, dependence-aware inference to compare feature sets and domain-adaptation methods while reporting descriptive best-configuration transfer results.

  • ∆ denotes paired VeryHigh recall differences evaluated on the same test source and seed.
  • Inference Protocol: 10,000 clustered-bootstrap resamples estimate 95% confidence intervals for mean paired effects.
  • Inference Protocol: Cluster-robust regression accounts for dependence among shared experimental configurations.
  • Inference Protocol: Per-source domain-adaptation comparisons report unadjusted p-values and assess Bonferroni robustness at p < 0.0125.
  • Inference Protocol: Table IV maxima summarize the highest mean recall over five seeds and evaluated configurations, rather than unbiased deployment estimates.

V. CROSS-SOURCE GENERALIZATION RESULTS

Models transfer well among the three curated sources but fail on GVA, where missing contextual features and much lower VeryHigh prevalence coincide with poor ranking performance.

  • Cross-Source Generalization: 0.68–0.89 VeryHigh recall is achieved on curated sources, versus 0.20 ± 0.45 on GVA.
  • Cross-Source Generalization: 0.0004 VeryHigh precision on GVA indicates that predicted VeryHigh cases are almost always false positives.
  • Cross-Source Generalization: GVA has entirely Unknown participant fields, removing contextual and demographic features used by the classifiers.
  • Cross-Source Generalization: 0.03% versus 1.8–5.1% under rule stratification and 0.18% versus 11–36% under std stratification describes GVA’s lower VeryHigh base rate.
  • Cross-Source Generalization: Modern classifier families do not reach the prior-work 0.81 minimum-across-source recall floor on the extended four-source pool.
  • Cross-Source Generalization: 0.007 VeryHigh PR-AUC on GVA confirms poor class ranking beyond any single threshold choice.

B. Feature-Set Effect Under Dependence-Aware Inference

Dependence-aware analysis finds no reliable full-feature advantage, while domain adaptation produces a selective GVA gain: DANN helps the hardest target but with very low precision and lower median recall.

  • Feature-Set Effect: ∆ = +0.018 with 95% clustered-bootstrap CI [−0.030, +0.067] shows no statistically detectable full-feature advantage.
  • Domain Adaptation: 0.28 mean GVA recall and 0.47 best-configuration recall make DANN the only adaptation method with non-zero GVA recall.
  • Domain Adaptation: 0.00 recall is produced by CORAL and importance weighting on GVA, while both hurt curated-source performance.
  • Domain Adaptation: 0.282 DANN improvement on GVA is statistically robust, with 95% CI [+0.114, +0.469] and p = 0.003.
  • Domain Adaptation: 0.0019 mean DANN precision on GVA remains near zero despite the recall improvement.
  • Domain Adaptation: 0.82 baseline median recall falls to 0.65 with DANN, exposing a trade-off between curated-source performance and GVA coverage.

VI. MECHANISM ANALYSIS

The mechanism analysis separates two possible explanations for GVA failure: incomplete feature availability and label-prevalence mismatch.

  • Mechanism Analysis: Feature masking tests whether incomplete feature availability can reproduce the observed cross-source failure.
  • Mechanism Analysis: Prior-shift recalibration tests whether label-prevalence mismatch alone is sufficient to explain the failure.

A. Feature-Masking Ablation

The controlled masking experiment isolates feature availability as a major contributor to cross-source failure, while oracle prior-shift recalibration cannot recover GVA signal.

  • Feature-masking design: Masking gender, race, mental_health, incident_area, and age on all sources tests feature completeness while holding other properties constant.The baseline LODO pipeline uses XGBoost, the standard strategy, full features, and five seeds.
  • Feature-masking results: Zero VeryHigh recall on curated sources follows masking the five features unavailable in GVA.The collapse provides controlled evidence for the feature-completeness hypothesis.
  • Feature-masking results: 0.745 GVA recall under masking is degenerate predict-all-VeryHigh behavior, with precision of only 0.002.The apparent recall gain does not represent recovered discriminative information.
  • Prior-shift recalibration: Oracle prior-shift recalibration leaves GVA VeryHigh recall and precision at exactly zero across all seeds and configurations.This indicates that label-side correction cannot rescue the feature-side collapse.
  • Prior-shift recalibration: Kaggle recalibration shifts precision from 0.25 to 0.37 while recall falls from 0.34 to 0.25.This confirms the expected precision–recall trade-off when features carry signal.

VII. FAIRNESS AND RESPONSIBLE EVALUATION

The fairness audit finds uneven mental-health-associated prediction behavior, but sparse or missing demographic information limits interpretation and group-wise analysis.

  • Audit scope: The audit examines VeryHigh recall and precision across gender, race, and media-attributed mental-health groups under the best modern LODO configuration.The contribution is diagnostic rather than corrective.
  • Coverage constraints: Gender auditing is uninformative because approximately 95% of perpetrators are male, while GVA is 100% Unknown for gender and race.Racial variation is substantial only in Mother Jones, the smallest source at n = 157.
  • Mental-health-associated disparities: Stanford MSA recall is 0.88 for mental_health=Yes versus 0.43 for No, while Kaggle shows 0.89 versus 0.86.The corresponding gaps are 45 points and 3 points.
  • Mental-health-associated disparities: Precision for the mental_health=Yes group is 0.34 on Stanford MSA and 0.31 on Kaggle despite higher recall.The pattern is described as over-prediction of VeryHigh risk for this group.
  • Mental-health-associated disparities: Mother Jones has a mental health=Yes rate of 51%, compared with 33% in Kaggle and 29% in Stanford MSA.The passage attributes this difference to editorial focus rather than clinical prevalence, weakening causal interpretation.
  • Uncertainty and interpretation: Stanford’s Yes-versus-No recall gap has overlapping confidence intervals only at extremes, while precision intervals are strictly disjoint.The reported intervals are 0.88 [0.69, 0.97] versus 0.43 [0.10, 0.82] for recall, and 0.34 [0.23, 0.47] versus 0.07 [0.02, 0.20] for precision.
  • Uncertainty and interpretation: Reverse causality cannot be distinguished from classifier-side disparate impact because labels may be assigned retrospectively after severe incidents.The finding should therefore be read as a joint labeling-plus-classifier failure mode.

C. Fairness Implications of Cross-Source Failure

Cross-source failure creates a fairness concern because GVA’s Unknown-dominated features and lower VeryHigh prevalence undermine both prediction and meaningful group auditing.

  • Fairness implications: A model that fails on most of its target population cannot be meaningfully audited for group-level disparities there.The GVA collapse therefore extends fairness concerns beyond individual-attribute gaps.
  • Failure mechanism: GVA participant fields are systematically Unknown, while its VeryHigh base rate is lower than in curated sources under both stratification strategies.The reported base rates are 0.03% versus 1.8–5.1% under rule and 0.18% versus 11–36% under std.
  • Implications for evaluation: Single-source models trained on curated data may appear more performant than they are on the broader mass-shooting population.The passage links this boundary to differences in casualty distributions, participant coverage, and inclusion criteria.
  • Failure mechanism: Three lines of evidence support feature completeness as a contributor despite co-varying temporal and distributional differences across sources.These include MMD comparisons, successful DANN transfer, and the controlled feature-masking result.
  • Domain-adaptation interpretation: CORAL and importance weighting require informative target features, whereas DANN can retain signal if a domain-invariant subspace preserves label information.GVA’s Unknown-dominated feature matrix undermines the assumptions behind CORAL and importance weighting.
  • Domain-adaptation interpretation: All domain-adaptation methods and baseline classifiers use identical feature sets, with outcome variables excluded from every pipeline.DANN’s difference is the adversarial domain-invariance objective, not an informational feature advantage.
  • Domain-adaptation interpretation: 0.47 is DANN’s best GVA recall, leaving more than half of true VeryHigh incidents missed.The result is the only tested method producing any GVA-transfer signal, not a resolution of cross-source generalization.

C. Limitations

MASH-Bench’s conclusions are bounded by choices in harmonization, imputation, model comparison, labeling, and correction methods. The benchmark also measures generalization across media-biased sources rather than fidelity to the underlying phenomenon, and operational use could amplify documented disparities.

  • Harmonization depends on source-specific mapping decisions, particularly for incident_area and mental_health, and reasonable choices may differ across teams.
  • GVA results depend on the specific imputation strategy used for its Unknown-heavy feature landscape.The Unknown-heavy landscape also reflects a real information constraint.
  • All four sources inherit media-coverage biases, so the benchmark measures generalization across biased sources rather than fidelity to the underlying phenomenon.
  • DANN uses a two-layer MLP whereas CORAL and importance weighting use XGBoost, partially confounding adaptation effects with base-learner architecture.The masking ablation provides indirect support that the observed mechanism is not purely architectural.
  • Quartile-based labeling degenerates on the extended pool, limiting prior-work comparison to rule and standard-deviation strategies.
  • The study evaluates one label-shift correction but not several related methods or richer missingness treatments.The oracle prior-shift result indicates that pure label-side correction is insufficient under the observed feature deficiencies.
  • Operational deployment could amplify the documented disparate prediction behavior for media-flagged mental-health cases.
Loading 2608.22460v1…