PaperlayerView exact revision on arXiv

Source-linked AI summary

AI4Fire: Evaluating Large Language Models on Wildfire Tasks

Yue Zhao, Xiyang Hu, Zuobin Xiong, Zhangyu Wang, Ruolin Li

arXiv:2610.10946v1cs.CLcs.LG

TL;DR

Wildfire-management evaluations matter because overstated scores can influence systems used in decisions with consequences for property and lives. AI4Fire surveys fire-task research and evaluates models on five wildfire tasks in paired bare and grounded runs. It finds that grounding’s impact depends on the task, while simple comparators and public-source hazards complicate score interpretation.

  • Problem

    Existing evaluations do not combine a broad model roster, multiple wildfire task families, and paired bare-grounded runs, despite the stakes of evaluating systems entering wildfire management.

    Source: §1 · p. 1

  • Method

    AI4Fire surveys fire-task work and runs six core models zero-shot, bare and grounded, on five wildfire tasks, with a 29-model sweep and source audits.

    Source: §1 · pp. 1, 3

  • Results

    Grounding’s effects varied by task: a read-only SQL tool raised core-model database accuracy from at most 0.160 to at least 0.885, while no core model beat persistence on staffing forecasts.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Simple rules and trained comparators were hard to beat, and public-release hazards can affect what wildfire benchmark scores measure.

    Source: §1 · pp. 1–2

    The six-model core, zero-shot prompts, single stored run per arm, task-specific untuned additions, and publicly indexed sources limit generalization and leave contamination unmeasured.

    Source: Limitations · p. 9

Abstract

from arXiv · show

Large language models (LLMs) are entering wildfire management, where overstated evaluations can cost property and lives. How do they perform on wildfire tasks, with and without grounding? Bare means a model receives the task input alone. Grounded means it also receives one task-specific addition: for smoke detection, a smoke-free reference frame from the same camera. AI4Fire runs six core models bare and grounded on five wildfire tasks, zero-shot; a sweep adds 29 more. Our literature search on fire tasks found 138 works; none combines this roster, task coverage, and paired bare and grounded runs. We report three findings. (1) Grounding helped most where the addition carried the answer: a read-only SQL tool lifted every core model's database accuracy from at most 16 to at least 88 percent. (2) Simple rules were hard to beat: no core model outperformed repeating today's staffing count, and two open-weight models mostly copied the median of similar earlier fire-days, a worse forecast. (3) Public releases carry hazards: a fire-danger column separates the holdout perfectly, and 67 aerial fire frames carry smoldering or fire-free labels read from a clipped thermal maximum. We release prompts, responses, scores, code, and the survey record.

1 Introduction

Wildfire evaluations matter because overstated scores can mislead systems trusted with decisions affecting property and lives. AI4Fire pairs bare and task-specifically grounded runs across wildfire tasks, finding that grounding’s effects vary and that simple comparators and source defects matter.

Research Question and Setup

  • The paper asks how frontier LLMs perform across wildfire tasks and whether task-specific grounding changes their performance.Bare runs receive only the task input; grounded runs add one task-specific input, so the study estimates no average grounding effect.Source: §1 · p. 1

Existing Work and Gap

  • The survey covers 138 works, while the authors report that no surveyed evaluation combines their model roster, task coverage, and paired bare-grounded runs.The benchmark’s 35 models from twelve vendors exceed the surveyed maximum of 22, and its tasks span four of ten survey categories versus at most three.Source: §1 · pp. 1–2

Key Findings

  • A read-only SQL tool raised core-model database accuracy from at most 0.160 to at least 0.885, while two open-weight models’ analogue-median forecasts were worse than persistence.The analogue-median rule had normalized error 0.249 versus persistence’s 0.146.Source: Figure 1 · p. 2, §1 · p. 2
  • No core-model run beat persistence, and a trained classifier outscored every prompted fire-danger run.After prompts stated the sample’s positive rate, no proprietary run’s Brier score was resolvably worse than the prior, while both open-weight models remained worse.Source: §1 · p. 2
  • Public-release hazards include a fire-danger column that separates the holdout perfectly and 67 aerial fire frames labeled smoldering or fire-free from clipped thermal maxima.Fifteen of the core six’s 34 grounded misses on 48 closed-form aerial items fell on those frames.Source: §1 · p. 2, Figure 1 · p. 2

Primary Contributions

  • The authors pair a multi-task benchmark with a survey, protocol and source audits, failure-pattern analysis, and a release of prompts, responses, scores, code, and survey records.The protocol also tests non-LLM comparators on every task and clusters uncertainty by dependence unit.Source: §1 · pp. 2–3

2 Fire Tasks and What Existing Work Shows

The survey organizes fire-related work by task and finds that existing evaluations do not combine a re-checked survey with paired bare-grounded runs across fire tasks. It also identifies differences between wildfire and structure-fire task families and a narrow direct forecasting comparison.

2.1 Scope and Task Categories

  • The survey spans LLMs, VLMs, MLLMs, and agents across wildfire and structure-fire domains, while AI4Fire itself evaluates five wildfire tasks with one model at a time.AI4Fire does not evaluate multi-agent systems or structure-fire tasks; the survey search retained 138 works and re-checked claims in a 31-work summary.Source: §2.1 · p. 3

2.2 Closest Works and the Gap

  • No surveyed work combines a re-checked literature survey with paired bare-grounded evaluation across fire tasks, according to the comparison in Table 1.The table scores listed works on task categories, evaluated models, paired arms, and non-LLM comparators.Source: Table 1 · p. 3, §2.2 · p. 3
  • Among 25 fire records, wildfire and structure-fire studies address disjoint task families; the one direct forecasting comparison favored domain-specific tabular deep learning over a GPT-4-based agent.The comparison concerned fire radiative power and did not test the current models evaluated in AI4Fire.Source: §2.2 · p. 3

3 The AI4Fire Benchmark

AI4Fire evaluates paired bare and grounded runs across five wildfire tasks, using six core models and a 29-model sweep. Its design includes task-specific additions, non-LLM comparators, clustered uncertainty estimates, and audits that identify properties affecting what benchmark scores measure.

3 The AI4Fire Benchmark

  • The benchmark evaluates six core models across five wildfire tasks and 1,446 paired items, with a 29-model sweep and a release of prompts, manifests, scores, and raw responses.The sweep brings the roster to 35 models from twelve vendors.Source: §3 · pp. 3–4

3.1 Tasks

  • Two of the five task additions can carry an item’s reference answer, while the other three provide context; the benchmark withdrew Fire360 because its released prompts lacked visual-evaluation mapping.The answer-carrying additions are the SQL tool and thermal block; the smoke task has 196 paired items from 224 frames.Source: Table 2 · p. 4, §3.1 · p. 4

3.2 Conditions to Compare

  • Each task uses the same models in bare and grounded zero-shot runs, but each task has a different addition, so the design estimates no average grounding effect.Some additions can supply answers directly, while others provide context to reason from.Source: §3.2 · p. 4

3.3 Model Set

  • The core set spans vendor, generation, and open-versus-closed weights; the sweep adds 29 models, including text-only and multimodal configurations.Four core models are proprietary and two are open-weight; ten sweep models ran all five tasks and nineteen were text-only.Source: §3.3 · p. 4

3.4 Metrics and Scoring

  • Scores are automatic because reference answers are exact or numeric, and every task is compared with a simple or trained non-LLM baseline.Allocation error is scaled by mean filed staffing for the relevant fire, and grounding is a paired difference on the same items within a model and task.Source: §3.4 · pp. 4–5

3.4 Metrics and Scoring; 3.5 What Building These Tasks Revealed About the Sources

  • Clustered intervals resample related items together, but may be too narrow with few clusters; reported contrasts condition on one stored run per arm and are descriptive.The source audit found a holdout-separating Mesogeos column and 67 FLAME 3 fire frames whose clipped 187.424 °C maximum led to smoldering or fire-free release labels.Source: §3.4 · p. 5, §3.5 · p. 5

4 Results, Insights, and Future Directions

Across five wildfire tasks, grounding effects depended on what the added input supplied, and simple comparators remained difficult to beat. The results also expose limits in the aerial labels and public benchmark data.

4.1 Results by Task

  • 4 to 21 of 112 smoke frames were recovered by the reference frame with no smoke-frame losses, but four models increased clear-frame false positives.In the first design, the reference-to-target gap alone separated labels; the matched-gap rerun found pooled recall rose by 0.063.Source: §4.1 · p. 5, Figure 2 · p. 7
  • On aerial QA, no core model resolved above the 0.627 majority, while the thermal block raised accuracy on 48 closed-form items and lowered it on the other 360.Every paired grounded-minus-bare interval for pooled accuracy included zero; grounded accuracy fell by 0.006 to 0.031 on the 360 other items.Source: §4.1 · p. 5, Figure 3 · p. 8
  • Only two of six core models beat the 0.654 temperature rule on fire-danger forecasting; monthly climatology lowered AUPRC for all four proprietary models without a resolved interval.The task’s 0.339 positive rate reflects dataset selection rather than regional prevalence.Source: §4.1 · p. 6, Figure 2 · p. 7
  • 0.000–0.160 bare database accuracy rose to 0.885–1.000 with the SQL tool, and only this tool cleared zero for all six core models.The tool arm could query a frozen SQLite snapshot; bare models scored 0.000 to 0.160.Source: §4.1 · p. 6, Table 3 · p. 6
  • 0.249 normalized error for the displayed-analogue-median rule exceeded persistence at 0.146 by 0.044, while no core model beat persistence.Analogues raised error on five of six core models; movement-matched retrieval reduced grounded error relative to rule v1 on all six.Source: §4.1 · p. 5, §4.2 · p. 7

4.2 Cross-Task Insights and Future Directions

  • Zero-shot models rarely beat trivial or trained comparators across tasks, although all twelve core-model smoke arms beat the always-smoke accuracy baseline.On smoke with matched gaps, all twelve also beat the frame-difference detector.Source: §4.2 · p. 7
  • Fire-danger calibration, decision rates, and rankings were sensitive to prompt information and wording, so leaderboard scores alone miss important evaluation dimensions.Stating the positive rate improved proprietary-model calibration; two single-run paraphrases reordered standings.Source: §4.2 · pp. 7–8
  • Public-release hazards included a fire-danger feature with average precision 1.000 on holdout labels and 67 aerial fire frames labeled from a clipped thermal maximum.The benchmark discussion calls for auditing leakage, broken links, and labels computed from clipped thermal values.Source: §4.2 · p. 8

5 Conclusion

AI4Fire evaluates shared model sets bare and grounded across five wildfire tasks, alongside a literature survey. It finds task-dependent grounding effects, weak performance against simple comparators, and public-release hazards that alter what scores measure.

5 Conclusion

  • AI4Fire pairs bare and grounded runs across five wildfire tasks, using six core models and a 29-model sweep on 1,446 paired items.The paper reports 77,644 scored responses and a survey of 138 works.Source: §5 · p. 8
  • A read-only SQL tool lifted every core model to 0.88 or above on database questions, while two open-weight models copied an analogue median that forecast worse than persistence.Where an addition held no answer, scores moved in both directions; zero-shot models rarely beat trivial or trained comparators.Source: §5 · p. 8
  • The paper releases prompts, item sets, harnesses, run manifests, and raw responses, and calls for paired multi-step agent evaluations with trivial and trained comparators.The central next step is multi-step agent tasks over the same public fire databases.Source: §5 · p. 8

Limitations

The paper’s limits arise from the survey’s coverage and the benchmark’s scope, sampling, and evaluation design.

Limitations

  • The literature searches were unsaturated, targeted works first released between 2025-01-01 and 2026-09-13, and did not systematically measure coverage outside English.The claim-by-claim re-check covered a nonrandom set of 31 records.Source: Limitations · p. 9
  • Six core models, mostly single-run configurations, and zero-shot prompts limit transfer claims; weights are also confounded with serving path.Grounding additions differ by task and were untuned apart from the second allocation rule, so null or negative effects bound only that addition.Source: Limitations · p. 9
  • Fire-danger results describe a negatively subsampled population with a 0.339 positive rate, not regional prevalence; aerial references are largely model-verified and include conflicting hotspot labels.Fourteen hotspot references conflict with the prompt, and nine items use clipped-maximum frames, all scored as released.Source: Limitations · p. 9
  • The five tasks omit structure fire and simulation-coupled multi-agent coordination, and exact-match or numeric scoring leaves judge error and operational-advice risks unmeasured.Unpinned proprietary snapshots may also prevent reproducing outputs.Source: Limitations · p. 9

Ethical Considerations

This section documents the survey’s collection and classification, re-checks a subset of prior-work records, and states ethical safeguards for interpreting the study.

Ethical Considerations

  • Zero-shot scores on past incidents do not qualify any model for operational use; staffing, fire-danger, or detection outputs require expert review before action.The study recruited no participants or annotators, and the release describes its data access and licensing conditions.Source: Ethical Considerations · p. 9

A.1 How the Literature Was Collected

  • 138 works first publicly released between 2025-01-01 and 2026-09-13 were retained after 13 search units swept eight topics and five publication sources.An unchecked venue sweep found three additional fire-perception works, which enter the survey as citations only.Source: §A.1 · p. 13

A.2 Profile of the 138 Kept Works

  • Keyword screening tagged 109 of 138 works as involving an LLM, VLM, or agentic LLM framework and 90 as carrying an evaluation; these tags were not re-checked.Source: §A.2 · p. 13

A.3 Re-Check Outcomes and Corrections

  • All 31 scoping-summary records existed, but ten had at least one problem; the 25 records with fire tasks supplied the survey’s fire evidence.Issue groups overlapped, and a second fetch caught an undercounted model list for Chen et al. (2025a).Source: §A.3 · p. 13

A.4 Category Boundaries and Open Questions

  • The survey uses ten task categories, assigning fire investigation to document understanding and code-generating Earth-observation agents to geospatial analysis.Multi-hazard suites count for their fire items.Source: §A.4 · p. 13

A.5 Terms

  • Grounding means adding task data, retrieved documents, tools, or deterministic rules at inference time; the bare arm receives only the task input.The survey distinguishes wildfire from structure fire as its two fire domains.Source: §A.5 · p. 13

A.6 Search-Unit Counts per Closest Work and Confidence in the Negative Answer

  • Repeated independent search-unit finds support the gap claim, while single-unit finds indicate that the searches were not saturated.For example, Qi et al. (2026) appeared in five units, while several listed works appeared in only one.Source: §A.6 · p. 14

B Data Availability Check

The availability check classifies candidate tasks by readiness, while benchmark construction exposed dataset defects that affect what some evaluations measure.

B Data Availability Check

  • Three of 34 candidate tasks were ready, 20 needed work, and 11 were not feasible; unreleased data explained seven infeasible candidates.Ready means reachable and scorable as released; “needs work” requires building items or a harness.Source: B Data Availability Check · p. 14

C Defects Found While Building the Tasks

  • Building the benchmark exposed hazards in three released datasets that change what their scores measure; the authors do not claim prior published evaluations are wrong.For the Mesogeos, Fire360, and closed-form findings, checks re-derived them from raw source files.Source: C Defects Found While Building the Tasks · p. 14

C.1 The Mesogeos Track A Release Ships Columns That Separate Its Own Holdout

  • Average precision 1.000: burned_area_has is nonzero for 100 percent of positives and 0 percent of negatives, so Track A evaluations must withhold all three label columns.The separate columns burned_areas and ignition_points also distinguish some positives from negatives; the holdout row counts differ from reported counts.Source: §C.1 · p. 14

C.2 The Released Fire360 Questions Cannot Be Matched to the Reported Evaluation

This section documents the evaluation and release audit of Fire360 and WildFireVQA, including benchmark selection, grounding, exclusions, and the core model roster. It also reports the model sweep’s recurring baseline-like and false-positive behavior and reviews benchmark data-source and licensing conditions.

C.2 The Released Fire360 Questions Cannot Be Matched to the Reported Evaluation

  • None of Fire360’s 100 released text prompts names a video, frame, or timestamp, so the released items cannot verify a grounded perception evaluation.The answer key has option 1 correct on 25 of 50 multiple-choice items and option 4 on none; none of the five reported accuracies is a multiple of 100/50 or 100/100.Source: §C.2 · p. 14

C.3 Rules on the Thermal Block Reproduce Four WildFireVQA Question Types

  • Fixed rules on thermal summaries reproduce all 24,388 labels across four WildFireVQA question types, covering 48 of 408 benchmark items.The rules use temperature cuts and threshold checks not stated in the prompt; the majority baseline is 0.627 across 408 items, and the vehicle-count question matches on 99.66 percent of frames.Source: §C.3 · pp. 14–15

C.4 A Clipped Thermal Maximum on 67 WildFireVQA Fire Frames

  • 67 Sycan Marsh frames labeled Fire share an exact 187.424 °C maximum, yet release labels mark them smoldering or fire-free across question types.The core six models miss 15 of 30 grounded answers on five closed-form items and all 24 on four no-fire-branch items; the frames show smoke plumes and burning lines.Source: §C.4 · p. 15

C.5 Fire360 as a Withdrawn Task

  • Withdrawing Fire360 leaves five wildfire tasks and no structure-fire arm because other candidate sets remain unreleased or restricted.The paper identifies two remaining routes to structure-fire evaluation: compliance on England Approved Document B and causal reasoning over NIOSH firefighter fatality reports.Source: §C.5 · pp. 15–16

D.1 Selection Criteria and What They Selected

  • Five criteria applied to 34 candidates, selecting FIgLib, WildFireVQA, Mesogeos, ICS-209-PLUS, and FPA-FOD; Fire360 was withdrawn after its prompt-to-frame mapping failed audit.Source: §D.1 · p. 16

D.2 The Grounding Source of Each Task

  • Grounding adds task-specific inputs: same-camera clear frames for smoke, thermal summaries for aerial questions, climatology for forecasting, analogues for allocation, and read-only SQL for tool use.Analogue input and outcome days must both precede target report days; the frozen FPA-FOD SQLite snapshot returns up to 50 rows and 4,000 characters per query, with at most eight calls per question.Source: §D.2 · pp. 16–17

D.3 What the Benchmark Leaves Out

  • The benchmark leaves out structure-fire code evaluation because no redistribution route for NFPA or ICC text was open, and 11 candidate sources were infeasible.Two routes remain open through England Approved Document B and NIOSH fatality reports; multi-agent suppression was deferred because trajectory costs were unmeasured.Source: §D.3 · p. 17

D.4 The Model Set in Full

  • The benchmark evaluates six core models from five vendors: four proprietary models through an API gateway and two open-weight models through Bedrock.Source: §D.4 · p. 18

D.5 The Model Sweep of 2026-09-18

  • The sweep tested 29 added models, but probe failures, tool-call interface differences, and output truncation constrained which models and responses could be evaluated.The adapter enabled Gemma 3 tool calls by rendering declarations in the system prompt; Gemma 3 27B followed the convention on all 156 items (0.827), while 12B and 4B scored 0.000 as interface failures. MiniMax M2.5 truncated 261 responses even at 8,192 tokens, so its allocation and fire-danger scores use only parsed answers; tool accuracy counts all 156 items and unanswered responses as wrong.Source: §D.5 · p. 19
  • 5 of 35 models returned the previous day’s count on at least 290 of 300 bare allocation items; grounding raised smoke false positives in 10 of 16 models.Bare smoke false-positive rates ranged from 0.000 to 0.119, while the four highest grounded rates were Gemma 3 27B (0.250), Gemma 3 12B (0.202), Gemma 3 4B (0.179), and Mistral Large 3 (0.157). Bare fire-call rates ranged from 0.044 for gpt-6-astra to 1.000 for Gemma 3 12B and 4B, against a 0.339 sample positive rate.Source: §D.5 · p. 19

D.6 Data Sources and Licensing

  • The release documents source-specific terms, including CC BY-NC-ND 4.0 attribution conditions for FIgLib and conflicting license statements for WildFireVQA.It releases FIgLib sequence identifiers, labels, and fetch scripts rather than raw JPEGs; FLAME 3 imagery is CC BY 4.0, and the listed Mesogeos and ICS-209-PLUS data are also CC BY 4.0.Source: §D.6 · p. 20

D.7 Reference Answers and Scoring Hazards, by Task

These appendices test how scoring choices, data defects, and alternative baselines affect wildfire-task results. They also document sensitivity to prompts, retrieval rules, run variation, and uncertainty methods.

D.9 Calibration After a Prior-Shift Correction and Recalibration

  • After the sample rate is stated, proprietary models’ ECEs fall to 0.025–0.148, while Qwen3-VL remains above 0.30 and Llama 4 Maverick’s Brier scores remain above the shifted prior.The proprietary four’s Brier scores improve resolvably in five of eight runs, and none is resolvably above the prior shifted to the same rate.Source: §D.9 · p. 24, Table A8 · p. 25

D.10 Trained Non-LLM Baselines on the Same Items

  • The fire-danger boosted classifier scores 0.851 AUPRC, exceeding gemini-3.1-pro bare by 0.154; a separate allocation regressor matches persistence at 0.144 normalized error.The classifier uses the 121 numbers printed in the bare prompt; the allocation regressor uses 23 bare-prompt features and scores 0.144 on the 300 evaluation items.Source: §D.10 · p. 24

D.11 Prompt Sensitivity on Fire Danger

  • Two fire-danger prompt paraphrases reorder model standings: gemini-3.1-pro leads under p0, claude-opus-5 under p1, and gpt-6-astra under p2.The corresponding AUPRCs are 0.697, 0.731, and 0.711; under p2, gpt-6-astra and claude-opus-4.8 call fire on only 0.003 and 0.010 of items.Source: §D.11 · pp. 24–25, Table A10 · p. 26

D.12 A Second Retrieval Rule for the Analogues

  • Under v2, the analogue-only allocation baseline reaches 0.190 normalized error versus persistence at 0.146, while grounded error falls across all six models.False-move rate falls from 0.40 under v1 to 0.20 under v2; the rule changes eligibility, distance, and the random step together, so their separate effects are not isolated.Source: §D.12 · pp. 25–26, Table A11 · p. 27

D.13 A Defect in This Benchmark: The Output Cap Chose Who Got Scored

  • The 1,536-token cap may lower bare tool-use scores: gemini-3.1-pro reached the cap before its ANSWER line on 61 of 156 items, and fallback scoring credited 3.Aerial-prompt thermal fields are percentages, not fractions; fraction readings matched four and three correct bare answers from Qwen3-VL and Llama 4 Maverick on closed-form items.Source: §D.13 · p. 26

D.14 Sensitivity to Image Width on FIgLib

  • On 43 frames captured at least 25 minutes post-plume, smoke recall rises from 0.535 at 1024 pixels to 0.628 at 1568 pixels, gaining four frames with none lost.Unpaired scoring gives the bare arm 28 pre-plume clear frames absent from grounded scoring; paired results supersede that comparison.Source: §D.14 · p. 26, §D.15 · p. 27

D.16 Tool Use on FPA-FOD: Harness and Failure Detail

  • On 156 SQL items, open-weight models’ 35 tool-use errors mostly reflect missed calendar windows, while proprietary models completed 624 items with one refusal.Thirty open-weight errors came from date formatting; five were Python snippets parsed as percentages.Source: §D.16 · pp. 27–28

D.20 Aerial Question Answering: Build, Harness, and Strata

  • Across aerial-question resampling designs, the pooled grounded-versus-bare contrast includes zero for all six models, while Llama 4 Maverick’s separation from the majority depends on clustering choice.The 408 items cover 390 frames and 34 question types; the benchmark also reports repeat-run variation and cluster-based intervals, with widened small-cluster intervals explicitly not claiming nominal coverage.Source: §D.18 · p. 28, §D.20 · p. 29, §D.19 · p. 29

E By-Task Detail

The appendices provide the by-task survey record, supporting floats, and search-corpus scope. They also document a withdrawn structure-fire source and the resulting cross-domain gap.

E By-Task Detail

G.1 Scope and Search Corpus

  • The search retained 138 works first released between 2025-01-01 and 2026-09-13, with one dating exception, and categorized them by task.Broader disaster or Earth-science work was in scope when it included at least one fire task.Source: §G.1 · pp. 29–30

G.2 Survey Patterns Driving Benchmark Design

The re-checked fire literature spans perception, forecasting, operations, and other task families, but its evaluations are often narrow in task, model, or comparator coverage. Its findings include benefits from grounding, limits in perception and forecasting, and gaps in evaluation hygiene.

Survey patterns

  • Prior evaluations often omit model runs, focus on vision or sequence models, or assess LLMs on one task family, one base model, or fire as one hazard among several.The survey patterns describe the re-checked records, not prevalence across all fire-related work.Source: §G.2 · pp. 30–31

Grounding

  • Grounding improved results in six re-checked studies, each against baselines chosen by its authors; retrieval also assisted stronger tested MLLMs on aerial wildfire questions.The studies covered wildfire risk, damage assessment, personnel and cost allocation, fire-code compliance, and regulation retrieval.Source: §G.2 · pp. 30–31

Perception and tuned models

  • Fire-imagery models scored highly on binary recognition and large smoke plumes, but localized early smoke poorly, erred on quantities, and showed deficits in temperature-grounded reasoning.Small domain-tuned models matched larger general models in some forest-fire, disaster-reasoning, and resource-allocation evaluations.Source: §G.2 · p. 31

Forecasting and multi-agent teams

  • The one direct fire-radiative-power forecast comparison favored tabular deep learning over an LLM agent; chain-of-thought improved out-of-distribution wildfire-risk generalization in FireScope.A separate simulation study found multi-agent coordination broke down on large suppression and rescue problems.Source: §G.2 · pp. 31–32

Evaluation hygiene

  • The 25 re-checked records include examples of evaluations using one system and self-chosen baselines, but that count is not a majority estimate.The records were not coded for this feature.Source: §G.2 · p. 35

H Closest Works, the Positioning Table, and the Late Find

The closest-works review finds no study varying grounding strategies across task families on one shared model set, while later-released works cover related but distinct evaluation settings. The failure-mode analysis separates model behavior from benchmark or source properties that invite errors.

H Closest Works, the Positioning Table, and the Late Find

  • No study found varies grounding strategies across task families on one shared model set; later works separately evaluate simulated wildfire missions, disaster-response tasks, and FLAME 3 imagery.DORA includes wildfire as one of ten disaster types, while FlameVQA evaluates two open-weight MLLMs on questions built from RGB and thermal pairs.Source: H Closest Works, the Positioning… · pp. 36, 38

I.1 Failure Modes in Detail

  • Qwen3-VL and Llama 4 Maverick copied the displayed analogue median on 284 and 259 of 300 items, while the analogue-only rule scored 0.249 normalized error against 0.146 for persistence.Under retrieval rule v2, the two open-weight models still copied on 283 and 258 items.Source: §I.1 · p. 39
  • Thirty of 35 open-weight tool failures used zero-padded or ISO date filters against unpadded m/d/yyyy dates, returning zero results.The date format was unstated in the schema and question; the proprietary four answered all 26 date-window items correctly.Source: §I.1 · p. 39
  • A prompt glossing thermal percentage fields as pixel shares introduced nine closed-form misses, including a Qwen3-VL answer above 15 percent when only 0.703 percent of pixels exceeded 200 °C.The hottest-3-percent mean of 172.6 °C ruled out that reading, making the gloss a harness property.Source: §I.1 · pp. 39–40
  • Clipped thermal maxima and ambiguous wording contributed to aerial-task errors, including nine of 34 closed-form misses that fit an area reading of “dominant fire behavior”.The release clips 67 fire frames at exactly 187.424 °C and labels them smoldering or fire-free on temperature-derived questions.Source: §I.1 · p. 40
  • For smoke detection, the reference-to-target time gap separates labels in the original design; fire-danger results also depend on addressing the unstated 0.339 sample rate.The appendix reports a rerun matching smoke and clear-frame gaps and a prior shift to the sample rate.Source: §I.1 · p. 40
  • Plain-word schema rewrites cost Qwen3-VL 0.175 and Llama 4 Maverick 0.126 on 143 clean items, while the proprietary four answered identically item for item; an output cap truncated 61 of gemini-3.1-pro’s 156 bare answers.The omitted ambiguous count_county_year family leaves the 143-item clean subset.Source: H Closest Works, the Positioning… · p. 38, §I.1 · p. 40

I.2 Extended Analysis: Daily Personnel Allocation

In personnel allocation, prompted models did not beat persistence, and retrieved analogues often raised error; in smoke detection, a reference frame increased recall, but the original design tied labels to time gaps. A matched-gap rerun retained a recall gain while narrowing that ambiguity.

I.2 Extended Analysis: Daily Personnel Allocation

  • No prompted model beat persistence in normalized error: model scores were 0.157 to 0.248 bare and grounded against 0.146 for persistence, while a trained regressor matched persistence at 0.144.The regressor used the same report fields and training data from 1,499 other incidents.Source: §I.2 · p. 41
  • Retrieved analogues raised normalized error on five of six models; differences excluded zero for gpt-6-astra (0.058), Qwen3-VL (0.087), and Llama 4 Maverick (0.053).Whether the proprietary models used analogues selectively or mostly ignored them remained unresolved.Source: §I.2 · p. 41
  • Under retrieval rule v2, grounded error fell for all six models, but the analogue-only prediction’s 0.190 normalized error still exceeded persistence by 0.044.The four proprietary models were within 0.015 of bare error, with no v2-minus-bare interval excluding zero.Source: §I.2 · p. 41

I.3 Extended Analysis: Wildfire Smoke Detection

  • The reference frame raised smoke recall on every model, turning 4 to 21 of 112 smoke frames from missed to found; false-positive rates rose on four models.Recall intervals excluded zero for four models and spanned zero for two.Source: §I.3 · p. 42
  • Accuracy was lowest within ten minutes of plume appearance, and the reference added 13 correct decisions among 168 pooled pairs in that window.It added 26 decisions from 10 to 25 minutes and 17 after 25 minutes.Source: §I.3 · p. 42
  • The original design’s reference-to-target gap separates all 196 items: clear targets are 9–30 minutes after reference, while smoke targets are 44–80 minutes after it.Every frame prints its capture date and time in the top strip.Source: §I.3 · p. 42
  • With matched gaps and masked timestamps, the reference raised pooled recall by 0.063, while its difference from the first design’s 0.081 gain was unresolved; the detector scored 0.536 accuracy, below every model arm.Clear-frame false positives rose by 0.016 with an interval spanning zero.Source: §I.3 · pp. 42–43

I.4 Extended Analysis: Aerial Question Answering

The extended analyses compare model performance with task-specific baselines across aerial question answering, fire-danger forecasting, database queries, and a broader model sweep. Results vary by task, and the analyses also identify label and prompt-sensitivity concerns.

I.4 Extended Analysis: Aerial Question Answering

  • All 16 models miss 14 aerial items whose references say no hotspot or no fire despite frame maxima of 272–567 °C; source and question type are collinear.The authors treat the post hoc exclusion as a pointer for label audit, not as a bound on label error.Source: §I.4 · pp. 43–44
  • The 0.664 hybrid rule exceeds every aerial model point estimate; five of six core models sit within paired error of the majority, while Llama 4 Maverick falls below it.The thermal block’s point estimates rise on 48 closed-form questions and fall on the other 360.Source: §I.4 · p. 43, Table A28 · p. 45
  • Accuracy rose from 0.646–0.792 bare to 0.771–0.938 grounded on 48 closed-form aerial items, but fell by 0.006–0.031 on the remaining 360.The closed-form gains offset losses elsewhere, with all reported intervals containing zero; paired tests also find declines for all six core models on 347 remaining items, with only the two Claude models reaching unadjusted p < 0.05.Source: §I.4 · pp. 45–46, Table A29 · p. 46

I.5 Extended Analysis: Fire Danger Forecasting

  • A boosted classifier scores 0.851, above the best prompted run; monthly climatology lowers all four proprietary models’ AUPRC by 0.005–0.032, with no interval excluding zero.Only gemini-3.1-pro and claude-opus-5 beat the 0.654 temperature rule.Source: §I.5 · p. 47
  • A single prompt paraphrase reverses the leading models’ order: claude-opus-5 moves from 0.688 to 0.731 AUPRC while gemini-3.1-pro moves from 0.697 to 0.648.The leader gap under the original prompt is comparable to the measured temperature-zero repeat changes.Source: §I.5 · p. 48

I.6 Extended Analysis: Fire Data Tool Use

  • The SQL tool raises database accuracy from 0.000–0.160 bare to 0.885–1.000 grounded, with paired gains of 0.808–0.994 and all intervals excluding zero.Open-weight models account for all 35 tool errors, chiefly through date-format mismatches or Python snippets on percentage items.Source: §I.6 · p. 48
  • Removing schema literals costs Qwen3-VL 0.175 and Llama 4 Maverick 0.126 on 143 clean database items; only Qwen3-VL excludes zero under sensitivity expansion.Twenty of 91 errors write full state names into two-letter code columns.Source: §I.6 · p. 48

I.7 Extended Analysis: Model Sweep

  • Among added full-capability models, none beats the aerial majority comparator; across text-only models, no bare-minus-persistence interval falls below zero and no fire-danger run beats the temperature rule.The aerial thermal block raises accuracy on 11 of 16 added models, while tool gains exclude zero upward for 15 of 17 text-only models.Source: §I.7 · p. 49

J Scoring Rules and Comparators, by Task

The appendix specifies each task’s scoring metric, task-specific rules, and non-LLM comparators, then describes paired interval estimation and its dependence units.

J Scoring Rules and Comparators, by Task

  • Smoke detection uses smoke recall, clear-frame false-positive rate, and offset-bucket accuracy; aerial QA uses accuracy against released answers and reports 48 thermal items separately.Comparators include two smoke constants, a reference-frame difference detector, the held-out aerial majority, a closed-form thermal rule, and a hybrid rule.Source: J Scoring Rules and Comparators, by Task · p. 49
  • Fire-danger forecasting reports AUPRC and fire-class F1, fills missing calls by thresholding stated probabilities at 0.5, and compares against a calendar-month prior and temperature rule..Source: J Scoring Rules and Comparators, by Task · p. 49
  • Personnel allocation uses normalized error with mean filed staffing as the fire-specific normalizer, compared with persistence and a trained regressor.Source: J Scoring Rules and Comparators, by Task · p. 49
  • Database tool-use accuracy is exact for counts, years, and names, uses a 0.5 percent tolerance with a one-acre floor for acres, and a 0.1-point tolerance for percentages.Its comparator is the best constant per question family.Source: J Scoring Rules and Comparators, by Task · p. 49
  • Paired contrasts use 20,000-resample 95 percent percentile bootstraps, clustered by task-specific dependence units ranging from fires to question families.The appendix also points to a small-cluster sensitivity expansion and describes what it can establish.Source: J Scoring Rules and Comparators, by Task · p. 49

K Reproducibility Details

The reproducibility appendix provides abridged task-prompt examples, describes item sampling and serving, records limits on verifying model settings, and discloses AI assistance in the research process.

K.1 Example Prompts

  • Example prompt pairs show how task-specific additions change inputs, including a same-camera earlier reference frame for smoke detection and a SQL tool for database questions.Prompt blocks are abridged; released item records and run scripts reconstruct full prompts.Source: §K.1 · pp. 49–50

K.2 Item Sampling, Climatology, and Serving

  • The fire-danger evaluation uses 386 items from a 4,120-cell pool, while personnel allocation uses 300 fire-days across 245 incidents; training-year climatology supplies the grounded forecast addition.The calendar-month prior is fit on 19,504 training samples and achieves 0.555 average precision on the 386 test items.Source: §K.2 · pp. 50–51

K.3 Reasoning Settings and Token Use

  • Proprietary endpoint defaults and versions are unpinned, and released code cannot inspect effective reasoning settings, limiting rerun reproducibility.Requests specify temperature zero and a 1,536-token cap, but these request settings do not establish effective backend settings.Source: §K.3 · p. 51

L AI Use Statement

  • The authors used AI coding and writing assistants for implementation, analysis tooling, figure preparation, and manuscript drafting and revision, while taking responsibility for the final content.Source: L AI Use Statement · p. 51
Loading 2610.10946v1…