Source-linked AI summary
Looking Complete Is Not Being Grounded: Evaluating LLM Sitreps Against Human-Authored Situation Reports
Sachin Kumar
TL;DR
Evaluation of crisis reports should reflect the information humanitarian practitioners need, including figures and provenance, rather than relying only on convenient text comparisons. The paper derives a 15-slot schema from professional reports and evaluates machine reports against human reporting and source availability. Machines can look complete by slot presence, but the quantitative record remains constrained by what the input stream supplies.
Problem
Common evaluation starts with reference overlap, embedding similarity, or general-purpose model judgments, while operational sitreps need current needs, actions, gaps, figures, and provenance-bearing information.
Source: §1 · p. 1Method
Results
Machines nominally exceeded the human ceiling under Gemini on Idai (0.827 vs. 0.821), but the difference was statistically indistinguishable from zero; across events, figures and provenance remained limited.
Source: §6.1 · p. 5, §9 · p. 9Takeaways & Limitations
Crisis-report evaluation should jointly measure source availability, figure grounding, and judge-indexed human-relative completeness.
Source: §9 · p. 9The schema reflects model-led coding and author review without human coding for comparison, so the authors present it as one defensible description of Idai reporting practice.
Source: §4 · p. 3
Abstract
from arXiv · showhide
Evaluation for automatic crisis reporting should measure what practitioners need, not what is convenient to compare. Through LLM-assisted, author-approved content analysis, we derive a 15-slot information schema from professional humanitarian situation reports using qualitative content analysis, then evaluate 121 human and 261 LLM-generated reports across five disasters and three hazard types. Under google/gemini-3.7-flash on Cyclone Idai, the best machine arm is nominally above the empirical human ceiling (0.827 vs. 0.821), but the difference is statistically indistinguishable from zero (95\% bootstrap CI $[-.049,.059]$; permutation $p=.852$), and the ordering does not replicate under openai/gpt-5.6-luna. On a blind 100-item Idai sample, the author agrees with the Gemini judge at 0.910 accuracy and unweighted Cohen's $\kappa=.799$. All four judges clear the plan's raw-agreement criterion of .7; only Gemini reaches .7 on chance-corrected $\kappa$. Across the three primary events, machine reports reproduce 5.4-11.0% of same-day official figures. Across those same events, raw exact-number matching links 14.3-19.6% of official figures and 85.4-90.1% of machine figures to the social stream; manual audit confirms 45/50 declared absences, 4/25 official matches, and 24/25 machine matches semantically. The issuing-provenance gap is positive in all 12 observed event-judge cells: report date, organisation, series, and authority were not supplied at professional-report coverage in any cell. On the three primary events, ROUGE-L is uninformative, BERTScore is negatively associated with completeness after multiplicity correction, and a generic LLM judge changes sign.
1 Introduction
The paper asks what professional sitreps contain, how much information systems recover from social media, and whether common metrics recognize that recovery. It proposes a practice-derived schema and evaluates completeness against human reporting and source availability.
1 Introduction
- The paper induces a 15-slot schema from professional reports, defines human-relative completeness and source visibility, and tests ROUGE-L, BERTScore, and a generic LLM judge against operational completeness.Its central result is that machine sitreps can look complete by slot presence while their quantitative picture remains bounded by what the input stream contains.Source: §1 · p. 1
2 Related Work
- Unlike message-classification work, the study compares generated operational products with reporting practice and measures information absent from the input stream.Social media is treated as a source with noise, brevity, and uneven coverage.Source: §2 · pp. 1–2
- The study adds slot-level comparisons with independently published humanitarian sitreps, a practice-derived schema, and an audit of which official information was visible in the input.It contrasts this combination with prior crisis-summarisation benchmarks, including CrisisFACTS, which used official ICS-209 forms without a practice-derived schema or input-visibility audit.Source: §2 · p. 2
- Operational completeness uses practice expectations induced across reports, allows partial credit, and takes an empirical human ceiling as its reference point.The schema is an LLM-assisted, author-approved content-analysis description, rather than an NLP event-role schema or a prompt-token optimisation method.Source: §2 · p. 2
- The paper tests lexical overlap, contextual similarity, and LLM judging against a domain-derived criterion, rather than relying on general ratings.Prior work has found that overlap and similarity can track topical similarity rather than information quality, while LLM judges can favor their own generations.Source: §2 · p. 2
3 Data
The study assembles professional humanitarian reports and event-day social-media inputs across disasters, then generates reports from sampled posts using crossed model and prompt conditions.
3 Data
- Professional reports were manually collected from ReliefWeb after API approval was unavailable and scripted requests returned a bot challenge; automated scanning and content hashes supported bookkeeping and duplicate removal.Three Idai rows were exact SHA-256 duplicates.Source: §3 · p. 2
- The evaluated Idai set contains 56 professional reports, including flash updates, snapshots, and IFRC appeal documents; Puebla’s evaluated products are mostly infographics or snapshots and are not used as a sitrep ceiling.The collection supplement records titles, sources, verified dates, locators, hashes, and exclusion rules for 125 products.Source: §3 · p. 2
- For each eligible event-day, generation used at most 400 stratified HumAID posts, while source availability was measured against the complete day; days needed at least 30 posts.Kerala used only its first five eligible days because of the stated cost limit.Source: §3 · p. 2
- The study crossed strong and fast DeepSeek generators with generic and schema-guided prompts, and excluded empty generations from judges and metrics.The schema-guided prompt supplied the 15 slot names and definitions; calls ran on 30–31 August 2026 UTC.Source: §3 · pp. 2–3
4 Inducing a Practice-Derived Schema
The authors use model-led coding and clustering, followed by author review, to describe recurring information types in professional Idai reports. They characterize the resulting schema as one defensible account of reporting practice, not the only one.
4 Inducing a Practice-Derived Schema
- The schema’s conceptual space was shaped before human review: instructions seeded five conventions, a model proposed codes from truncated text, and no human coding comparison was available.The authors therefore treat the schema as one defensible description of Idai reporting practice, not the only one.Source: §4 · p. 3
- All 56 evaluated Idai reports formed the induction set; reports longer than 30,000 characters were coded from their first 24,000 and last 6,000 characters, leaving 13% of induction text uncoded.The reports were mostly early-response products, and the instructions’ examples echoed project-plan conventions that later became prominent code names.Source: §4 · p. 3
- Embedding and clustering yielded 382 clusters, of which the top 25 became candidate slots; the other 357 clusters, containing 57% of codes, never reached review.Ranking clusters by report count could penalize inconsistently named concepts; protection and hedging were split, and no final slot records hedging.Source: §4 · pp. 3–4
- The author reviewed every candidate, merged overlapping concepts, renamed categories, rewrote definitions, and finalized 15 slots covering metadata, attribution, affected people, sectors, needs, and response.The frozen schema records its approver and approval time, and the audit trail retains report identifiers, codes, and consolidation decisions.Source: §4 · p. 4
5 Measurement
The measurement framework scores reports against a 15-slot schema, compares machine coverage with an empirical human ceiling, and tests figure grounding and common quality metrics against operational completeness. It also specifies content checks and human validation for judge decisions.
Operational completeness
- Operational completeness averages a document’s usable judged slot scores; missing judgements are dropped rather than scored zero, and documents with none are excluded.Slot judgements use SENT, PARTIAL, or PRESENT, and each decision requires an evidence span.Source: §5 · p. 4
Coverage and human ceiling
- Slot coverage is calculated separately for each event and exact generator/prompt arm, while the human ceiling is mean completeness across evaluated professional reports.The best machine arm is compared with humans without pooling machine arms; Idai’s ceiling is in-sample because its reports induced the schema.Source: §5 · p. 4
Content gate and judge reliability
- A deterministic content gate caps empty or explicitly no-information matched headings at ABSENT and short non-negated bodies at PARTIAL.Bodies with fewer than eight alphabetic tokens receive the PARTIAL cap; other scores remain unchanged.Source: §5 · p. 4
- Judge validation uses a blind, slot-stratified sample of 100 Idai document–slot items and reports exact-match accuracy, unweighted Cohen’s κ, and weighted κ.The plan’s ≥.7 acceptance rule was applied to raw agreement in pre-annotation code, though some notes described it as a κ threshold.Source: §5 · pp. 4–5
Figure grounding and source availability
- Figure-grounding analysis compares machine figures with same-day human reports and the social stream, with separate tests for traceability and official-figure availability.It uses exact numeric matches, event-day or machine–official-pair resampling, and a manual audit of sampled matches and declared absences.Source: §5 · p. 5
Metric meta-evaluation
- Meta-evaluation correlates ROUGE-L, BERTScore, and a schema-blind LLM quality score with operational completeness on Idai, Irma, and Maria.Puebla and Kerala are excluded because each has only eight same-day pairs and weaker or genre-contaminated references; the nine tests receive joint Holm–Bonferroni correction.Source: §5 · p. 5
6 Results
Across the primary events, the results distinguish apparent completeness from source-grounded reporting and show that common reference metrics do not reliably track schema completeness. The apparent Idai machine lead is statistically unsupported, while metadata gaps and weak figure grounding persist.
6.1 Presence produces a nominal inversion
- 0.827 vs. 0.821: api-strong/schema-guided nominally exceeded Idai’s human ceiling under Gemini, but the +.0061 gap was nonsignificant and did not replicate under Luna.The difference CI was [−.0490, .0591], with permutation p = .852.Source: §6.1 · p. 5, Figure 2 · p. 6
6.2 One gap is structural
- Metadata gaps were positive in all 12 observed cells, while displacement gaps varied across Idai, Irma, and Maria (+0.288, −0.035, +0.070), distinguishing medium-driven from event-driven gaps.The provenance concern covers report organisation, number, publication date, and stated authority.Source: §6.2 · p. 6
6.3 The ceiling is in the source
- Official-number recall was 11.0%, 5.4%, and 8.9% across Idai, Irma, and Maria; raw matching linked 14.3–19.6% of official versus 85.4–90.1% of machine figures to the stream.Manual review found 4/25 official matches and 24/25 machine matches were semantic; 45/50 declared official absences were confirmed.Source: §6.3 · p. 6
6.4 Standard metrics fail differently
- ROUGE-L was nonsignificant and changed sign; BERTScore was significantly negative across all three events after Holm correction (ρ = −0.43 to −0.47), while the generic judge changed sign across events.On Idai, the BERTScore association remained negative after controlling for length (partial correlation −0.248).Source: §6.4 · p. 6
- Arm-level ρs across Idai, Irma, and Maria were −1.0/−1.0/−.8 for BERTScore, −.8/.0/−.8 for ROUGE-L, and +.4/−.6/−.6 for Gemini’s quality judge.With four systems per event, these are descriptive order summaries rather than inferential evidence.Source: §6.4 · pp. 6–7
6.5 Human validation is judge-specific
- All four judges met the .7 raw-agreement threshold, but only Gemini reached .7 on chance-corrected κ; Luna’s κ fell from .762 on professional items to .470 on machine items.The blind validation supports Gemini-based pilot measurement more strongly than Luna-based replication, not LLM judging in general.Source: §6.5 · p. 7
7 Implications for Humanitarian Practice
The implications focus on treating machine drafts as unverified signals and checking their figures and provenance. The authors also limit their conclusions to the evaluated setting and recommend event-specific and product-specific checks.
For information managers and report authors
- Reviewers should check draft figures, dates, and sources rather than headings; the authors recommend treating drafts as unverified signals and not circulating them under an agency’s name.The authors also call for screening repeated phone numbers and email addresses from crisis posts.Source: §7 · pp. 7–8
For organisations piloting or funding reporting tools
- Pilots and procurement can test tools on past emergencies by checking figures against same-day official reports, comparing like report types, and asking which inputs were used.Similarity to professional reports and generic LLM quality ratings did not track completeness in this study.Source: §7 · p. 8
For tool builders
- Social-media inputs often lack official figures, so the authors suggest connecting drafting to official-figure channels and using social posts to flag needs and locations for verification; they did not test this approach.Displacement coverage differed across events, so the authors recommend re-checking gaps in each emergency.Source: §7 · p. 8
For professional reporting itself
- A practitioner-reviewed slot list could help authors check for unintended omissions, but it should be adapted by product type and evaluated for operational value.Short products such as snapshots and infographics are not intended to cover every slot, and the schema lacks or only partly covers categories such as protection.Source: §7 · p. 8
8 What the Shared-Task Community Should Adopt
The shared-task protocol should distinguish topic presence from operational completeness, grounding, and source availability. The proposed changes add human calibration, uncertainty, and judge-specific checks alongside existing similarity-based scores.
8 What the Shared-Task Community Should Adopt
- Across three primary events and two judges, human reports score 0.71–0.85, so systems need an empirical human ceiling rather than a perfection-based interpretation.The current protocol does not define human calibration.Source: §8 · pp. 8–9
- Slot coverage can place machine reports nominally above professional reports without significant difference or cross-judge replication; report alignment and confidence-label agreement beside similarity scores.The guidelines specify confidence-label agreement, but the official evaluator does not compute it.Source: §8 · p. 9
- Report precision, recall, and casualty recall against official records alongside slot coverage, and add number-specific exactness beyond text similarity and bullet alignment.Table 2 gives primary-event casualty recall of 33.0/52.1/12.5%.Source: §8 · p. 9
- Report Gj with intervals rather than Bj: the 0.6 inequality falls within the observed .520/.540/.615/.680 spread, with two judges flagging it and two not.The proposed change adds uncertainty to point-score aggregation.Source: §8 · p. 9
- Extend evaluator coverage checks to per-judge parse-failure rates; unparsable outputs were 0/19/159/0 across four judges, with the largest loss in the slot carrying the largest gap.The proposal retains existing coverage and metric-status checks.Source: §8 · p. 9
- Use a human-validated cross-family judge or disclose its absence, and quantify the share of reference facts present in social-media inputs.Cross-family judges are not interchangeable, and same-family agreement can masquerade as validity.Source: §8 · p. 9
- Keep topic naming, operational-reference matching, and evidence sufficiency separate rather than collapsing them into one evaluation question.The proposed additions extend input-to-reference auditing beyond output-to-reference scoring.Source: §8 · p. 9
9 Conclusion
The conclusion distinguishes apparent completeness from operational grounding. Across measured events, reports reproduce little of the official quantitative record and consistently fall short on issuing provenance.
9 Conclusion
- Across primary events, the nominal machine advantage over the empirical human ceiling is statistically indistinguishable, does not replicate under Luna, and is smaller than measured truncation bias.Systems reproduced little of the official quantitative record, while provenance coverage lagged in all 12 observed event–judge cells.Source: §9 · p. 9
- Across measured primary events, systems reproduced little of the official quantitative record and supplied issuing provenance less often than professionals in all 12 observed event–judge cells.The provenance fields are report date, organisation, series, and authority.Source: §9 · p. 9
- Crisis-report evaluation should jointly measure source availability, figure grounding, and judge-indexed human-relative completeness.A social-media draft largely restates figures already in that day’s posts and cannot stand in for the official record or carry an agency’s authority.Source: §9 · p. 9
Limitations
The completeness findings are bounded by judge validation, schema construction, corpus scope, and measurement choices. Figure grounding and source availability do not rely on the schema or a model judge.
Limitations
- Blind judge validation uses one annotator who approved the schema and only Idai, so agreement may overstate independent-practitioner agreement and does not establish transfer to other events.All judges meet the raw-agreement threshold, but only Gemini reaches .7 on chance-corrected κ; consistency measures do not establish correctness.Source: Limitations · p. 9
- The schema comes from 56 Idai reports, 44 from OCHA, making it chiefly reflective of OCHA practice; its Idai ceiling is in-sample.Held-out professional reports on Irma and Maria score 0.71–0.76, below Idai’s 0.82–0.85.Source: Limitations · pp. 9–10
- Schema construction used seeded conventions, model-led coding, capped candidate review, and no independent second coder, so completeness covers 15 measured slots rather than all reporting practice.Omissions could favor either professional reports or stream-derived drafts, depending on the reporting category.Source: Limitations · p. 10
- The source ceiling describes an English-language labelled HumAID sample, not social media at large; one generator family supplies all drafts, and multilingual claims are limited.Irma and Maria have 19 and 25 evaluated human reports, while Kerala has seven, making those ceiling estimates noisier than Idai’s.Source: Limitations · p. 10
- Regex number matching can misclassify coincidental matches or miss equivalent expressions; a machine figure absent from an official report is unverified, not necessarily false.Original judge inputs truncated reports at 20,000 characters; the full-length re-judging shift covers a seed-17 sample of 12 affected reports, not the whole corpus.Source: Limitations · p. 10
Ethical Considerations
The paper frames the system as evaluation infrastructure, not an autonomous reporting tool. It warns that reports which appear complete while remaining operationally incomplete could misdirect resources or falsely convey authority.
Ethical Considerations
- Social-media text is not redistributed, user identifiers are omitted, and no post is quoted verbatim.ReliefWeb reports are public.Source: Ethical Considerations · p. 10
- The system is intended as evaluation infrastructure, not an autonomous reporting tool.The paper states this deployment boundary explicitly.Source: Ethical Considerations · p. 10
- An operationally incomplete report that appears complete could misdirect scarce resources or falsely convey authority.The statistically unsupported nominal ordering illustrates this deployment hazard.Source: Ethical Considerations · p. 10
A Reproducibility and Audit Trail
The artifact makes the study’s schema, data manifest, prompts, measurement procedures, and analysis tables inspectable and reproducible. Machine outputs are withheld for privacy, so numerical results and functions are auditable, but fresh generation from mutable model aliases is not guaranteed to be deterministic.
Reproducibility artifacts
- The release includes verbatim prompts, the frozen 15-slot schema, model identifiers and call dates, a 125-report manifest, matcher-audit labels, and the schema-induction audit trail.The audit trail includes 382 clusters, 25 candidates, and merge rationales.Source: A Reproducibility and Audit Trail · p. 12
- Publication tables and released text-free rows reproduce the coverage, validation, agreement, and other reported claims, with Figure 2 reading the corrected Gemini table.Per-judgement rows include failures and duplicate keys, allowing the last-row and observed-case rules to be checked.Source: A Reproducibility and Audit Trail · p. 12
Measurement specification
- The measurement specification publishes exact number-matching rules and pins ROUGE-L, BERTScore, and embedding-clustering implementations to package and model revisions.Exact matching uses (unit, round(value,6)) with no tolerance; percentages remain separate and recognised dates and bare years are excluded.Source: A Reproducibility and Audit Trail · p. 12
Privacy and provider-version boundary
- Machine-output text is withheld because generations can repeat contact details from crisis posts, limiting public verification of text while preserving bit-for-bit auditability of numerical tables and functions.Fresh generation from mutable aliases is not promised to be deterministic.Source: A Reproducibility and Audit Trail · p. 12
Implementation repairs
- Four implementation defects were repaired, including ignored token caps, broken reference pairing, generation truncation, and incorrect judge labels; regression tests cover all four repairs.Corrected labels and code did not change numeric scores.Source: A Reproducibility and Audit Trail · p. 12
Decoding parameters
- Table 6 records LLM decoding parameters, including provider-default temperature and unsent top-p, while distinguishing pre-fix calls from corrected-cap replication runs.The artifact disaggregates event–judge caps and records effort.Source: Table 6 · p. 13