PaperlayerView exact revision on arXiv

Source-linked AI summary

Calibrated Decisions at Scale: Converting Police Crash Narratives into Probabilistic Crash Variables with a System One Model (Jev)

Amir Rafe, Subasish Das

arXiv:2609.24052v1cs.CL

TL;DR

Crash narratives contain circumstances that coded fields omit, but coding them at scale raises cost, verification, and human-review questions. This paper tests a typed decision model through large-scale Texas coding and independent probability audits. Against human labels, the model reaches an F1 of 0.908, and adding narrative variables changes the safety diagnosis.

  • Problem

    Crash coded fields omit circumstances recorded in investigator narratives, while generated model outputs can be difficult to verify.

    Source: §1 · p. 2

  • Method

    The study codes Texas narratives with gated, typed decisions and audits returned probabilities against coded fields and human judgments.

    Source: §3 · p. 7, §6 · p. 31

  • Results

    Against human judgment, the typed model reached F1 0.908; coded-field agreement understated human agreement by a median 0.26 kappa, and narrative variables added 10,747 injury and fatal crashes per year across nine factors.

    Source: §5 · p. 28, §6 · p. 31

  • Takeaways & Limitations

    Adding calibrated narrative variables to coded contributing-factor fields changes the safety diagnosis those fields support.

    Source: §5 · p. 28

    The model is closed and commercial, its training data are undisclosed, and its behavior may change without notice, making the analysis non-reproducible if the vendor withdraws it.

    Source: §6 · p. 31

Abstract

from arXiv · show

Crash datasets that carry an investigator narrative hold information the coded fields omit. Coding those narratives at scale has been blocked by three obstacles. Frontier large language models are costly at that scale, their generated text cannot be verified, and no rule says how much output a human must check. This paper formulates narrative coding as gated, typed decisions answered by Jev, a System One model that returns probabilities over analyst-defined options and generates no text. A screen covered 499,500 Texas narratives and 195,857 were coded with a 27-question schema. Cost is governed by schema size rather than narrative length. The probabilities are audited against coded fields and against 2,416 blinded human judgments drawn under a stated sampling design. Two frontier large language models are benchmarked on the same records. Against human labels the typed model attains an F1 of 0.908. One frontier model gains 0.059 and the other is indistinguishable from it. Calibration varies by model rather than by paradigm, so each model must be audited. Recalibration on the same labels reduces calibration error by a factor of 3.3. Agreement with coded fields understates fidelity to the narrative by a median of 0.26 in kappa. A resolution-floor bound covers any model that reports probabilities on a discrete grid. A review budget over flagged records gives the records a human must read per variable and per year. Adding the calibrated variables to the coded fields raises the injury and fatal crashes attributed to nine factors by 10,747 per year.

1. Introduction

Crash narratives record circumstances that fixed coded fields omit, but their use at scale has been limited by cost, unverifiable generated output, and the lack of a calibrated human-review budget. This paper evaluates a typed-decision approach through population-scale coding, probability audits, and analysis of how narrative-derived variables change safety diagnoses.

1. Introduction

  • Frontier-model cost, unverifiable generated outputs, and the absence of a calibrated rule for human checking are the three obstacles to routine narrative coding.Without trusted confidence values, an agency must either check everything or check nothing.Source: §1 · pp. 2–3
  • Jev returns an answer from a fixed option set and a probability for each typed question in one pass, without generated text.The paper independently tests whether its probabilities are calibrated rather than relying on the developer's claims.Source: §1 · p. 3
  • No prior study combined non-generative typed coding, calibration audits against administrative and weighted human references, a flagged-record review budget, and measured population-scale cost.The nearest studies evaluated generative models on attributes already coded in the database and reported agreement.Source: §1 · p. 3
  • The study audits probabilities against coded fields and weighted human judgments, benchmarks two frontier models on the same records, and measures review needs and changes to safety diagnoses.Its four research questions cover population-scale coding and cost, calibration, annual review workload per variable, and changes when narrative variables are added to coded contributing-factor fields.Source: §1 · pp. 3–4
  • The study codes narratives as gated, typed decisions and measures each design rule rather than assuming its effect.Presence questions open detail questions, and named entities come from a closed option set rather than generated text.Source: §1 · p. 3
  • 195,857 Texas narratives were coded with a 27-question schema at $0.154 per thousand; the fitted cost model says spend depends on schema size, not text length.A 10-question screen covered 499,500 narratives; $774 is the projected cost for all 5,018,080 narratives, which were not coded.Source: §1 · p. 3

2. Background

Prior narrative-coding research shows that narratives can expose circumstances missing or misclassified in coded fields, while calibration and selective-prediction methods supply tools for evaluating probabilities and review decisions. This paper adapts those tools to rare variables and discrete probabilities, where standard assumptions do not hold.

2.1. Crash narratives and narrative coding in safety research

  • Crash-narrative studies have recovered factors omitted or misclassified by coded fields, but comparisons that use those fields as the reference inherit their documented incompleteness.The literature includes rule-based, topic-model, encoder, and generative approaches; injury-surveillance work routed low-confidence cases to human review without testing whether routing confidence was calibrated.Source: §2.1 · pp. 4–5
  • A typed interface prevents invalid option values, but it does not establish that option order or criteria wording leave returned probabilities unchanged.The study treats output validity separately from possible order and wording effects.Source: §2.1 · p. 5

2.2. Calibration, selective prediction, and label-efficient evaluation

  • Calibration measures whether predicted probabilities match observed frequencies; selective prediction links abstention to coverage, but error-rate targets can be uninformative for rare safety variables.The human reference set uses stratification on model probability to make the informative middle probability range estimable.Source: §2.2 · pp. 5–6
  • The review budget reframes selective prediction around whether a person must read a flagged record, rather than conventional error risk and coverage.On rare-positive variables, a model answering everything can satisfy conventional error targets because negatives dominate.Source: §2.2 · p. 6

2.3. Decision models without text generation and the calibration of language-model confidence

  • System One models return probabilities with typed answers and no generated text, but their probabilities still require independent calibration audits.The paper identifies two setting-specific adaptations: a discrete grid permits an exact calibration error and bound, while rare positives call for a workload measure tied to flagged records.Source: §2.3 · pp. 6–7

3. Data and methods

This section describes the data and methods for converting crash narratives into typed probability outputs, including the corpus, coding process, reference sets, and evaluations of calibration and human review needs. It also states the assumption behind a conditional calibration bound and describes procedures for recalibration and threshold evaluation.

3. Data and methods

  • Jev returns probabilities for typed questions in one call, and only the one-dimensional recalibration map is fitted; CRIS fields are compared against, while human labels fit the map.The framework reports the share of flagged records a person must review to reach a target precision.Source: §3 · p. 7

3.1. Corpus and de-identification

  • The analyzed corpus contains 5,018,080 Texas crash narratives from 2017–2025, while empty narratives fell from 8.24% in 2017 to 0.00% in 2025.The typed-model runs used text redacted by the data owner only; human coders and frontier vendors received additionally redacted and screened text, so the authors recommend screening before the first model call.Source: §3.1 · pp. 8–9

3.2. Schema design and two-stage coding

  • The 27-question schema uses gated details and a two-stage design, coding 499,500 screen records and 195,857 full-schema records.The random second-stage subset supports population estimates, while rare-class enrichment supports per-variable calibration and precision; the two runs cost $25.23 and $30.22. Cost is driven more by question count than narrative length, and a targeted pursuit exclusion reduced aggression positives without materially changing other presence questions.Source: §3.2 · pp. 9–10

3.3. Reference sets

  • The study compares coded CRIS fields, a weighted human reference, and two frontier models scored against the same human labels and criteria.The human sample used probability-cell filling and calibration weights; 2,416 usable judgments remained after unclear answers and ties were excluded. CRIS agreement is bounded by errors in the coded reference, and frontier models supplied verbalized probabilities without access to coded fields or Jev outputs.Source: §3.3 · pp. 10–12

3.4. Evaluation

  • For calibrated outputs, ρ = E[p] ≥ δ(1 − w0); if ρ < δ(1 − w0), no allocation on the attainable grid is calibrated.The audited model never returned zero in 2.4 million answers, with δ = 0.01, so the bound applies to variables whose coded-field base rate is below 0.01; for them, mean probability is not a prevalence estimate.Source: §3.4 · p. 13
  • The review budget counts flagged records that cannot be auto-accepted at a target precision, with thresholds cross-fitted on human labels and annual counts scaled to the analyzed corpus.Tied probability blocks are reviewed or accepted whole; if no threshold reaches the target, all flagged records go to review. The budget is reported both as variable-level decisions and, where relevant, as the union of records across variables.Source: §3.4 · pp. 13–14
  • Under one-sided coded-field error, its reliability curve lies at or below the narrative-based curve, and its calibration error is no greater when the stated condition holds at every weighted grid value.The condition is that (1 − f_v)r_v ≥ v at each grid value; the coded field is assumed to miss stated factors but never record unstated ones.Source: §3.4 · p. 14
  • The bound assumes coded fields never record factors unstated in narratives, an unestablished source-pair condition checked against human judgments in Section 4.4.The reported ratio of the two calibration errors is close to one, so the bound does not explain the difference between the coded-field and human references.Source: §3.4 · p. 14
  • Post-hoc Platt scaling and isotonic regression are fitted on one narrative half and evaluated on the other over 50 repeats in both directions.Probabilities are clipped to [10^-6, 1 − 10^-6] for fitting and evaluation; Platt scaling adjusts intercept and slope, while temperature scaling is its a = 0 special case, and isotonic regression fits a non-decreasing map held constant outside its fitted range.Source: §3.4 · p. 14
  • Maps are fitted per variable when the reference set has at least 20 positive labels, and pooled otherwise; thresholds are retuned per variable to maximize weighted F1.Thresholds are tuned on one random narrative half and scored on the other over 200 repeats, allowing out-of-fold gains to distinguish threshold failure from model failure.Source: §3.4 · p. 14

4. Results

The results assess Jev’s narrative-coding performance and calibration against human judgments, frontier models, and coded fields, then examine how calibrated variables alter factor-attributed crash estimates. They also report variation in human confirmation and the cost of Jev’s gains over keyword rules.

4.1. Population coding, cost, and the probability landscape

  • The run coded 195,857 of 5,018,080 narratives for $30.22, or $0.154 per thousand; at that rate, coding the full corpus would cost $774.At 27 questions, schema content averaged about 3,451 tokens per call versus about 89 tokens for a median-length narrative; the executed screen’s break-even fraction was 0.67, versus the model-predicted 0.58.Source: §4.1 · pp. 14–15, Table 2 · p. 15
  • Across all 16 presence variables, fewer than 3% of narratives fell between probabilities 0.3 and 0.7, while every variable showed mass at both grid ends.The output grid ran from 0.01 to 1.00 in 0.01 steps; across seven gated variables, leakage ranged from 0.03% to 3.99%.Source: §4.1 · p. 15, Figure 3 · p. 18

4.2. Calibration against coded fields and against human judgment

  • Coded-field kappa was lower than human-reference calibration-weighted kappa for all 9 comparable variables, by a median of 0.26, because codes omit factors stated in narratives.For drug involved, kappa was 0.48 against the coded field and 0.96 against human judgment; omission can reduce agreement without reducing calibration.Source: §4.2 · pp. 16–17
  • Pooled human-reference calibration error fell from 0.0231 to 0.0069 with either Platt scaling or isotonic regression, a factor of 3.3.The held-out median error across 11 variables with at least 20 positive labels fell from 0.0261 raw to 0.0034 under variable-specific Platt maps; 10 of 11 improved.Source: §4.2 · pp. 18, 20, Table 6 · p. 21

4.3. Accuracy, selective prediction, and the review budget

  • Jev’s population-weighted human-reference F1 was 0.908, with 11 of 16 variables at 0.85 or higher; results exclude coder-unclear or disputed pairs.Treating all 49 excluded pairs as model errors lowers pooled F1 to 0.875; treating them all as correct raises it to 0.912.Source: §4.3 · pp. 20, 23, Table 5 · p. 21
  • The coded-reference review budget was 68,049 variable-level decisions per year, or 61,994 distinct narratives, while conventional accuracy-based selective risk implied no review for all 9 variables.Eight of nine variables required review of essentially every flagged record to reach 90% precision against coded fields; human-referenced budgets were lower for all nine comparable variables.Source: §4.3 · pp. 22–23, Table 8 · p. 23, Figure 5 · p. 24

4.4. Frontier comparison and value added over the coded fields

  • Calibration varied by frontier model: one had a calibration error of 0.0027 versus 0.0282 and 0.0231 for the other models, showing calibration cannot be inferred from model type or interface.A pricing scenario—not measured costs for the benchmarked arms—put a mid-tier model at $7.95 per thousand narratives versus $0.154 for Jev, a factor of 52.Source: §4.4 · pp. 24–25, Table 9 · p. 25
  • F1 was 0.967 for the first frontier model versus 0.908 for Jev, a paired difference of 0.059; the second scored 0.885 and was not distinguishable from Jev.The second model’s paired difference was 0.023 with an interval of [-0.002, 0.048], which straddles zero.Source: §4.4 · p. 24, Table 9 · p. 25
  • Coded-field agreement ranged from 0.10 kappa for unbelted to 0.91 for animal involved, and seven of nine factors were flagged more often by narratives than by coded fields.Human coders confirmed 59% of narrative-only model flags overall, 77% on variables the model coded well, and 24% on weak variables, showing that coded fields are not an accuracy reference.Source: §4.4 · p. 25, Table 10 · p. 26
  • The typed model recovered more coded positives than keyword rules, raising recall for every variable while precision rose for three and fell for the others.McNemar tests rejected equal correctness for every variable except fatigue; macro-averaged F1 was near one half for keyword rules and somewhat higher for the typed model at two orders of magnitude higher cost.Source: §4.4 · pp. 25–26, Table 11 · p. 26, Figure 6 · p. 27

4.5. What changes in a safety diagnosis when narrative variables are added

  • The annual estimates use a Stage-2 random stratum of 44,403 injury or fatal crashes, summing recalibrated probabilities for narrative-only factors rather than counting model flags.The stratum includes 964 fatal crashes; intervals resample reference narratives and refit the per-variable recalibration map, estimated on 400 narratives.Source: §4.5 · pp. 26, 28
  • 10,747 additional injury and fatal crashes per year were attributed across nine factors; additions ranged from 16% for unbelted to 182% for phone use, with a 39% median.Phone use accounted for 1,334 added crashes per year (interval: 1,091 to 1,522), compared with 732 recorded in coded fields; four factor rankings changed.Source: §4.5 · p. 28, Table 12 · p. 28
  • Human confirmation supported factor-by-factor adoption: it was at least 0.70 for four factors and 0.90 for phone use, but below 0.50 for two factors and 0.15 for unbelted.The paper cautions that low-confirmation additions, especially for unbelted, are mostly model error rather than a finding about crashes.Source: §4.5 · p. 28

5. Discussion

Adding calibrated narrative variables changes the safety diagnosis supported by coded contributing factors, while calibration, reference choice, and transfer conditions constrain how the results should be used. The discussion also develops a review budget for flagged records and a probability-grid calibration bound.

5. Discussion

  • 10,747 additional injury and fatal crashes per year are attributed to nine mapped factors; four change rank, with a median increase of 39%.Phone use rises 182% and moves from the smallest factor in coded fields.Source: §5 · p. 28
  • The typed model’s mean probability was 4.8% against 2.7% prevalence and slope 1.63; Platt recalibration brought the pooled slope to 0.97.Calibration patterns differed across models, so calibration is a property of the individual model and elicitation together.Source: §5 · p. 29
  • Cohen’s kappa against coded fields was lower than weighted kappa against human judgment for all nine comparable variables, by a median of 0.26.The two references had similar calibration error, with a median ratio of 1.0; the difference was in agreement, not calibration.Source: §5 · p. 29
  • A discrete probability grid can make calibration impossible when its smallest positive value exceeds a variable’s base rate, forcing mean probabilities above prevalence.The bound applies to any model reporting probabilities on a discrete grid.Source: §5 · p. 30
  • A review budget defined over flagged positives estimates the human reading needed to reach target precision; the reported cross-fitted bound is finite-sample, not a guarantee.Applying distribution-free risk control would require an exchangeable calibration sample and a preselected error level.Source: §5 · p. 30
  • The schema and cost model are expected to transfer more readily than the coded-field reference, which must be rebuilt for each state’s coding system.The study covers one state, model version, and schema; different prevalence, reporting conventions, or much longer narratives are stated failure conditions.Source: §5 · p. 30
  • Transfer remains a design expectation until tested beyond this study’s one-state setting.The discussion identifies replication in other states as a test of whether the schema and cost model transfer.Source: §5 · pp. 30–31

6. Conclusion

The study combines large-scale typed narrative coding with calibration and review-budget audits, then measures how calibrated variables change crash-factor counts. Its conclusions remain bounded by the references, labels, model, and single-state setting used.

6. Conclusion

  • The typed model coded 195,857 Texas narratives with a 27-question schema at $0.154 per thousand, projecting to $774 for the full corpus.The cost model is governed by schema size rather than text length; two-stage screening pays below a break-even fraction of 0.67.Source: §6 · p. 31
  • Against human judgment, the model reached F1 0.908 and weighted kappa 0.906, but recalibration reduced out-of-fold calibration error to 0.0069 and brought slope to 0.97.Adding calibrated variables raised injury and fatal crashes attributed to nine mapped factors by 10,747 per year and changed four factors’ ranks.Source: §6 · p. 31
  • Accuracy claims rely on a smaller human reference set of cases coders found clear; disputes were excluded, and five variables had too few positives for their own recalibration maps.The coded CRIS fields are incomplete, and recalibration maps fitted on Texas judgments should be refitted rather than transported.Source: §6 · p. 31
  • The proposed next steps are discrepancy adjudication, revised criteria for four variables, replication in other states, and repeated audits across model versions.The revised schema would be evaluated against a fresh reference set.Source: §6 · p. 32

A. The question schema

Appendix A documents the schema sent to the model, including its question types, gates, instructions, and criteria. It also records how the released schema version was developed and distinguishes the separately run residual-identifier screen.

A. The question schema

  • Table A.1 reproduces the full version 1.1 schema sent to the model, including each question’s type, gate where applicable, instructions, and criteria.The criteria text specifies boundary cases and is the complete variable specification used by the model, coders, and frontier models.Source: A. The question schema · p. 32, Table A.1 · p. 32
  • The released version followed a 2,000-narrative development run; it was used for second-stage coding, the reference set, and frontier-model comparisons.The files also include a lean first-stage schema of 10 presence questions.Source: A. The question schema · p. 32
  • Gated questions are interpreted only when the gate probability exceeds 0.5; the residual-identifier question shipped with the schema but ran separately from population coding.It was run as a separate display screen, outside the Stage 1 and Stage 2 calls.Source: A. The question schema · p. 36

B. Keyword families and the coded-field mapping

Appendix B documents the regular-expression baseline and the mapping from narrative variables to coded CRIS references. The baseline families have different development and scoring scopes, while the mapping includes proxy and sensitivity-only references.

B. Keyword families and the coded-field mapping

  • The rule baseline uses case-insensitive regular-expression families, with its first ten tuned only on development narratives to over-find and enrich rare classes.Three further families cover variables with coded references and were scored on the working set only.Source: B. Keyword families and the… · pp. 36–37, Table B.1 · p. 37
  • Table B.2 maps narrative variables to coded CRIS fields, with crash-level references true when the rule holds on any person or unit row.The mapping was implemented over the crash-unit-person extract.Source: B. Keyword families and the… · pp. 36, 38, Table B.2 · p. 37
  • Hydroplaning uses a proxy because CRIS has no hydroplaning field; the opposite-direction collision type is retained only as a wrong-way sensitivity check.That proposed wrong-way proxy proved a factor of 17 too broad.Source: B. Keyword families and the… · pp. 37–38

C. Coder protocol and additional reliability diagrams

The human reference labels only what the narrative states, using independent coders and a fixed assignment protocol; the appendix details the weighting and reliability displays used to assess that reference.

C. Coder protocol and additional reliability diagrams

  • The protocol labels whether a narrative states a factor—not what probably happened—and uses a mixed panel of three independent readers to limit over-inference.Each narrative was coded by two readers, with the shared calibration block coded by all three; unclear verdicts and ties were recorded as disputed and excluded.Source: C. Coder protocol and additional… · p. 38
  • Calibration weights scale labeled pairs to population counts within five probability strata for each of 16 presence variables.Cells with fewer than five labeled pairs were merged with an adjacent cell for that variable; the weights have a design effect of 1.95 and effective sample sizes of 64–93 per variable.Source: C. Coder protocol and additional… · pp. 38–39
  • Forty-nine unclear or disputed pairs are excluded from every Section 4.3 rate, making those rates conditional on pairs coders found clear.The appendix notes that routing these pairs to review leaves F1 undefined because human-reviewed records are not scored as model decisions.Source: C. Coder protocol and additional… · p. 39, Table C.1 · p. 39
  • Figure C.1 plots weighted human positive rates by sampling stratum against model reliability for seven presence variables without coded counterparts.It also marks the probability-grid floor and approximate intervals based on Kish effective sample size.Source: Figure C.1 · p. 40
  • Figure C.2 compares precision among accepted records with the share of flagged records auto-accepted, reporting in-sample budgets of 10.1% and 0.0%.The prose distinguishes these from the cross-fitted budgets of 10.0% and 0.8% reported in Section 4.3.Source: Figure C.2 · p. 40

D. Proofs and the implementation map

The appendix formalizes calibration and cost results for weighted data and discrete probability outputs, then states the assumptions behind its weighting estimator and points to implementation details.

D. Proofs and the implementation map

  • The appendix’s proofs use weighted empirical distributions, and its implementation map identifies where each formalized quantity is computed.The stated results carry over to the population when weighted means are replaced by expectations.Source: D. Proofs and the implementation map · p. 41
  • Grouping records by their discrete probability values gives an exact Murphy decomposition of the Brier score without binning error.The reliability term bounds the exact calibration error; the appendix defines the observed positive rate at each grid value and the model’s weighted base rate.Source: D. Proofs and the implementation map · p. 41
  • A discrete probability grid imposes a calibration-error floor when the base rate is below the mass implied by the smallest positive grid value.The bound applies regardless of the assignment of positive grid values; recalibration can send a positive value below the grid floor, so the floor concerns the grid rather than ranking.Source: D. Proofs and the implementation map · p. 41
  • The coded-field reference is modeled as narrative positives lost through omission, but if fields also record unstated factors, the direction of measured bias is not determined by omission alone.Under the omission model, the coded field cannot be positive unless the narrative reference is positive; omission can inflate measured error on parts of the probability grid.Source: D. Proofs and the implementation map · pp. 41–42
  • The break-even fraction decreases with narrative length, with elasticity equal to minus the share of full-schema tokens contributed by text.When question-processing cost dominates the text term, the two-stage design’s break-even point depends on the questions dropped by screening rather than text length.Source: D. Proofs and the implementation map · p. 42
  • The weighting is a calibration estimator, not an inverse-probability estimator, because the greedy draw and identifier screen leave no expressible inclusion probabilities.Its estimator is design-consistent if labeling is uninformative about outcomes within cells; the ratio of weighted sums is consistent but not unbiased.Source: D. Proofs and the implementation map · p. 42

Data and code availability

The authors release code and aggregated outputs, while narrative-level and crash-linked inputs remain restricted; a de-identified audit table is available only if the data owner permits it.

Data and code availability

  • The released repository includes the schema, runner, metric implementations and tests, figure and table generators, dependency file, release manifest, and aggregated outputs.Runs use a pinned model identifier recorded per call.Source: Data and code availability · p. 42
  • Narratives, crash-identifier files, and the reference frame are not released under the data agreement; a de-identified pair-level audit table is prepared for release on request pending data-owner approval.The table omits crash identifiers and narrative text while carrying model probabilities, consensus labels, dispute status, design information, and frontier predictions.Source: Data and code availability · p. 42
  • Population-scale results and downstream counts cannot be recomputed from releasable files because they require Stage-2 outputs joined to coded fields at crash level.Aggregated result files are released instead, and source reproduction requires controlled access through the data owner.Source: Data and code availability · pp. 42, 44

CRediT authorship contribution statement

The contribution statement assigns conceptualization and methodology to both authors, with Rafe also responsible for software, formal analysis, data curation, validation, visualization, and the original draft.

CRediT authorship contribution statement

  • Amir Rafe handled software, formal analysis, data curation, validation, visualization, and the original draft; both authors contributed to conceptualization, methodology, and review.Subasish Das is additionally credited with supervision.Source: CRediT authorship contribution statement · p. 44
Loading 2609.24052v1…