PaperlayerView exact revision on arXiv

Source-linked AI summary

Evidence-Anchored Overdose Death Prediction from Death Investigation Reports

Raavi Gupta, Ziqing Yang, Karli R. Hochstatter, Talia Nadel, Nicole D'Anna, Jason K. Graham, Hannah Johnson, Kelsey Opozda, Nabila El-Bassel, Anubhav Jangra, Smaranda Muresan

arXiv:2610.19556v1cs.CLcs.AIcs.LG

TL;DR

Death investigation reports may provide early information for suspected-overdose prediction, while confirmation can be delayed and existing approaches rely on manual input or later death certificates. This paper compares expert-guided explanation and prediction designs; its best pipeline with report-grounded rationales improves on an adopted rule-based system.

  • Problem

    Overdose confirmation may be delayed, and existing early detection relies on manual investigator input or death certificates that may not be available for months.

    Source: §1 · p. 1

  • Method

    The system predicts suspected overdoses from death investigation reports using expert-guided feature explanations, including verbatim evidence sentences, and compares pipeline with self-rationalizing models.

    Source: §1 · pp. 1–2, §3.2.2 · p. 5

  • Results

    15% improvement in overdose prediction over an adopted expert rule-based system is reported for the best pipeline setup with feature-guided rationales and open-source models.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The pipeline lets experts refine extracted explanations before classification, and expert feedback improves rationale extraction F1 by 13.6%.

    Source: §6 · p. 8

    Evaluation uses a single jurisdiction, so transfer to offices with different documentation practices remains an open question.

    Source: Limitations · p. 9

Abstract

from arXiv · show

Confirmation of overdose deaths in the United States is often delayed due to the complexities of certifying drug-involved deaths. Death investigation reports represent a rich but underutilized data source that could be used to predict whether a death is a suspected overdose (i.e., a death preliminarily identified as an overdose prior to the final cause and manner of death determination). We present an automated framework that classifies suspected overdoses from death investigation reports, while producing supporting explanations guided by expert knowledge. Through an expert-guided design process, we investigate 1) what type of explanations are best for model prediction performance, and 2) which architecture is better suited for this task: pipeline models that rationalize first and then predict, or self-rationalizing models that jointly predict and rationalize? Our experiments show that a pipeline model with explanations grounded in the input death investigation reports outperforms the other settings. Our best model also outperforms a rule-based system currently in deployment by 15%, while allowing expert interventions for final decisions.

1 Introduction

Delayed confirmation of overdose deaths limits timely surveillance, while death investigation reports offer early narrative evidence. This paper develops an expert-guided system to predict suspected overdoses from those reports and compare explanation and model designs.

1 Introduction

  • Death investigation reports are available shortly after a fatality and contain physical findings, substance involvement, and witness observations.Confirmation of overdose deaths can lag other causes by more than six months, delaying surveillance and public health interventions.Source: §1 · p. 1
  • Existing early-detection systems rely on manual investigator input, while prior automated methods using death certificates may wait months for those records.Manual approaches are time-intensive and depend on a small, highly trained workforce.Source: §1 · p. 1
  • The study compares feature verbalizations with report-extracted feature-guided rationales, and pipeline models with self-rationalizing models.The design process is guided by domain experts and produces explanations informed by expert knowledge.Source: §1 · p. 1
  • 15% improvement in overdose prediction over an adopted expert rule-based system is achieved by the best pipeline setup using feature-guided rationales and open-source models.Experts can refine explanations during inference rather than starting from scratch, which the authors say could save time per report.Source: §1 · p. 2

2 Related Work

Prior work motivates explanations grounded in input evidence and expert feedback, alongside medical NLP systems that extract signals from clinical text. This paper applies those ideas to early overdose identification from death investigation reports.

2.1 Explanation generation

  • Input-extracted rationales offer auditable, human-readable evidence, addressing concerns that free-form rationales may not faithfully reflect model reasoning.Prior work also finds that free-form rationales are not faithful by construction.Source: §2.1 · pp. 2–3

2.2 Medical NLP and Explanation-Driven Decision Support

  • Earlier overdose systems using death investigation reports require extensive expert annotation or return only a global flag, whereas this framework preserves criterion-level evidence.Prior overdose classification and NLP work has also relied on death certificates, which may not be available for more than six months.Source: §2.2 · p. 3

3 Methods and Experimental Setting

The method constructs expert-feature explanations from death investigation reports and compares prediction architectures that use those explanations. The experiments use annotated and weakly annotated reports, with separate procedures for feature verbalization and report-grounded rationales.

3.1 Dataset

  • The dataset contains 9,263 reports split into feature-annotated reports with expert features and weakly annotated reports with overdose labels only.Overall, 64.7% of reports are labeled as overdoses, and 39.8% lack expert feature annotations; the study reports macro-F1.Source: §3.1 · p. 4

3.2 Explanation Generation and Prediction Models

  • For weakly annotated reports, five-shot question answering with Llama-3.1-8B infers missing feature labels, while available expert labels are retained.The completed feature values are used to generate both explanation types.Source: §3.2 · p. 4

3.2.1 Feature Verbalization

  • Feature verbalizations map present expert-defined features to natural-language statements, which are concatenated into an explanation.The feature indicators come from expert labels when available and model-inferred labels otherwise.Source: §3.2.1 · p. 4

3.2.2 Feature-guided Rationales

  • Pipeline models separate explanation generation from prediction, while self-rationalizing models produce the explanation and prediction in one pass.For feature-guided rationales, the pipeline classifier can use the extracted rationales alone or alongside the original report; self-rationalization is not considered for this explanation type.Source: §3.2.1 · p. 4, §3.2.2 · p. 5
  • Feature-guided rationales are verbatim report sentences classified as evidence for expert-defined features, and the rationale set includes sentences supporting at least one feature.Sentence-level spans are chosen for simplicity and robustness rather than arbitrary spans.Source: §3.2.2 · p. 5
  • Because no gold evidence spans are available, teacher-model labels are aggregated by majority vote into silver labels for training a sentence-level student classifier.At inference, the student extracts rationales directly from the report without the teacher ensemble.Source: §3.2.2 · p. 5

3.2.3 Implementation Details

  • Decoder-only models use LoRA fine-tuning with 4-bit quantization, while Clinical-ModernBERT is fine-tuned in full precision for sentence-level multi-label classification.The decoder-only models are trained with a maximum sequence length of 4,096 tokens.Source: §3.2.3 · p. 5

4 Results

The strongest configuration combines the report with feature-guided rationales, outperforming the expert rule-based baseline. Results also favor grounded rationales, show differences between model regimes and data splits, and identify where teacher agreement and expert feedback vary.

4 Results

  • 0.831 macro-F1 for Qwen3 with I, RG →O exceeds SPOT’s 0.721 macro-F1; the configuration uses no expert knowledge.Table 2 compares variants across test splits; SPOT is the expert-human-annotated rule-based baseline.Source: §4 · pp. 5–6, Table 2 · p. 6
  • Feature-guided rationales outperform feature-verbalized explanations on every split, with the best model ahead by 2.3% macro-F1.A logistic regression using binary feature indicators reaches 0.621 macro-F1, substantially below the corresponding best Qwen3 model; feature presence alone is insufficient.Source: §4 · pp. 5–6
  • For feature-verbalized explanations, pipeline models outperform self-rationalizing models: Llama-3.1 scores 0.812 vs. 0.779 macro-F1 and Qwen3 scores 0.776 vs. 0.768.This comparison is limited to feature-verbalized explanations; the feature-guided rationale model cannot jointly generate explanations and predictions.Source: §4 · p. 6
  • The best model’s macro-F1 is higher on the weakly annotated split than on the feature-annotated split.The authors suggest that feature prevalence or class imbalance may explain the difference.Source: §4 · p. 6

4.1 Teacher Agreement for Silver Data Construction

  • Teacher agreement is moderate overall (κ = 0.361) and varies by feature, with higher agreement for observable scene evidence than for substance-use history.Prescription Medication(s) has κ = 0.171; Powder/Residue on Scene has κ = 0.497.Source: §4.1 · p. 6

4.2 Expert driven online testing

  • Expert feedback raises aggregate rationale-extraction F1 from 0.602 to 0.684, a 13.6% gain, with the largest improvements for rare or underspecified features.Cocaine and Ketamine remain stable; PCP has 0.000 F1 in both conditions, while feedback marginally hurts Injection Equipment and Straw.Source: §4.2 · p. 7

5 Case Study

Three representative cases distinguish aggregation errors, where surfaced evidence is weighed incorrectly, from extraction errors, where evidence fails to reach the classifier and the downstream prediction is also wrong.

5 Case Study

  • The cases separate errors of aggregation from extraction failures, the latter preventing evidence from reaching the classifier and causing downstream prediction errors.The analysis examines three representative cases from the feature-annotated test set.Source: §5 · p. 7

6 Discussion

Feature-guided rationales complement the death investigation report, and together they produce the best reported configuration. The discussion connects this approach to expert correction of extracted evidence and local use of open-weight models.

6 Discussion

  • Combining the report with feature-guided rationales yields the best configuration at 0.831 macro-F1; feature-guided rationales alone do not match the full report, while logistic regression over the same features reaches 0.621 macro-F1.Source: §6 · p. 7
  • In a representative case, the rationale clarifies that flagged substance-use history refers only to marijuana, enabling the model to correctly reject the case.The qualitative cases distinguish correct evidence extraction from a failure that propagates to prediction.Source: §6 · p. 8, Table 4 · p. 8
  • Experts can correct explanations before classification; incorporating their feedback improves rationale-extraction F1 by 13.6%.Experts often remove on-scene evidence misread as use history and add missed synonyms; without correction, such errors can propagate to prediction.Source: §6 · p. 8
  • The framework requires no manual annotation at inference and uses open-weight models that run locally, keeping sensitive data in the practitioner’s environment.It also allows experts to refine the evidence behind each prediction.Source: §6 · p. 8

7 Conclusion

The paper targets delays in overdose mortality data and the manual extraction burden of existing report-based systems. It finds that report-grounded feature-guided rationales in a pipeline model provide an earlier signal that experts can review and adjust.

7 Conclusion

  • Delayed finalized mortality data make timeliness the practical bottleneck the system targets.Confirmation depends on toxicology findings, autopsy reports, and sometimes other investigative information.Source: §7 · p. 8
  • The pipeline model with report-grounded feature-guided rationales is the best setup, offering experts machine-extracted rationales to refine rather than requiring them to read each report from scratch.The conclusion presents reviewable, adjustable outputs as a way to reduce manual extraction burden while keeping experts accountable for determinations.Source: §7 · p. 8
  • The conclusion frames correctable explanations as a way to support expert review and adjustment while reducing the burden that limits surveillance throughput.It describes this as an earlier signal at lower human cost, without requiring practitioners to trust an opaque model.Source: §7 · p. 8

Limitations

The evaluation and evidence design leave transferability, feature coverage, rationale-label quality, and prospective feedback-loop design unresolved.

Limitations

  • Evaluation in one jurisdiction leaves transfer to offices with different documentation practices an open question.This is the clearest stated boundary on generalizing the learned models.Source: Limitations · p. 9
  • Expert-defined features may not capture all overdose-relevant evidence across jurisdictions or changing drug landscapes.The limitation concerns the breadth and continued coverage of the feature set.Source: Limitations · p. 9
  • Silver-labeled rationale-extraction data are noisy, with moderate teacher agreement (κ = 0.361) and possible systematic errors for the lowest-agreement variables.The authors note that some sentence-level labels are incorrect.Source: Limitations · p. 9
  • The feedback loop was evaluated retrospectively; prospective deployment would require interface design that keeps reviewer corrections efficient and consistent.Source: Limitations · p. 9

Ethical considerations

The study describes safeguards for confidential death-investigation reports, including deidentification before research-team access and secure, compliant data environments.

Ethical considerations

  • The study was approved by the institutions’ IRB, and data sharing used a secure SharePoint environment configured for HIPAA and FDA compliance.Source: Ethical considerations · p. 9
  • Model development and training took place in a certified cloud environment approved for PII, PHI, and HIPAA data.Source: Ethical considerations · p. 9
  • Reports were deidentified with the Presidio analyzer before the research team accessed them to prevent unnecessary use and release of identifiable data.Source: Ethical considerations · p. 9

AI use disclosure

The appendix reports that logistic-regression baselines using evidence variables performed below language-model approaches, and describes feature extraction, explanation construction, and teacher-model evaluation procedures.

AI use disclosure

  • The disclosure says generative AI aided metric-generation code and table generation, but not core experiment implementation, ideation, literature review, figure generation, or paper writing; authors reviewed the assisted work.Source: AI use disclosure · p. 9

A Feature Extraction for the Weakly-annotated Partition

  • For the weakly annotated partition, the feature-extraction system answered a separate presence question for each feature and achieved average macro-F1 of 0.967 on the feature-annotated split.Source: A Feature Extraction for the… · p. 11

A.1 Generation Setting

  • Feature extraction used Llama-3.1-8B-Instruct with greedy decoding and five in-context examples per feature, querying each report section separately and marking a feature present if any section yielded Yes.Source: §A.1 · p. 11

A.2 Question for Each Feature

  • Table 5 reports per-feature extraction accuracy on the feature-annotated test split, distinguishing human annotations from the system.Source: Table 5 · p. 12

B Feature-verbalized Statements

C Teacher Ensemble Prompt

D Comparison with Logistic regression

  • The appendix compares logistic regression across three experiments: human-annotated features on the feature-annotated subset, model-extracted features on matching cases, and model-extracted features pooled across splits.All models use balanced class weights.Source: D Comparison with Logistic regression · p. 12
  • Logistic-regression baselines remained subpar to language-model approaches, indicating that evidence and report context are required for the final prediction.The appendix describes three baselines varying annotation source and training split; Table 12 reports macro-F1.Source: D Comparison with Logistic regression · p. 12

E Analysis where the evidence extracted is correct but the final prediction is not

The analysis examines prediction discrepancies despite correct feature extraction, expert feedback on feature-guided rationales, and additional extraction and prediction results. The supplied passages identify the scope of these analyses but do not provide most table contents.

E Analysis where the evidence extracted is correct but the final prediction is not

G Additional Model Results

  • Additional prediction-model results and logistic-regression baseline Macro-F1 results are reported, with the latter varying evidence-variable source and training distribution.Table 12 distinguishes three training setups and marks evaluations unavailable by construction.Source: Table 11 · p. 13, Table 12 · p. 13
  • The appendix lists per-feature questions for MLI reports and predefined natural-language statements mapped to feature variables for feature-verbalized explanations.Source: Table 6 · p. 14, Table 7 · p. 14
  • Teacher-model agreement is reported for sentence-level evidence extraction, and per-variable extraction accuracy is evaluated against human labels across 5,580 feature-annotated cases.Table 8 covers agreement across 18 forensic variables; Table 9 reports per-variable accuracy and notes PPV = 1.000 for all variables and models.Source: Table 8 · p. 15, Table 9 · p. 15

F Expert feedback types

Related papers

Loading 2610.19556v1…