PaperlayerView exact revision on arXiv

Source-linked AI summary

HazardWeaver: Scientific Route Selection for Hazard Analysis Agents

Wangshu Zhu, Xueqi Cheng, Liang Wu, Yushun Dong

arXiv:2610.03591v1cs.AI

TL;DR

Hazard analysis requires choosing among scientific routes that may be appropriate and executable under the current evidence and resources, with eligibility changing as analysis proceeds. HazardWeaver grounds applicability in literature evidence, checks capability compatibility, and revises route decisions; on its benchmark it outperforms the strongest baseline, especially on multiple-route tasks.

  • Problem

    Scientific route choice for a hazard task depends on both evidence-based applicability and current resource availability, and those conditions can change during analysis.

    Source: §1 · pp. 1–2

  • Method

    HazardWeaver combines evidence-linked conditions from HKC with typed capability reachability in HCG so HWA can select, execute, and revise routes as the analysis state evolves.

    Source: §1 · p. 2, §3.3 · p. 5

  • Results

    89.4% DCA versus 41.1% for the strongest baseline across 141 sealed instances, with the largest gains on multiple-route tasks.

    Source: §7 · p. 9

  • Takeaways & Limitations

    The benchmark evaluates artifact correctness together with route validity and recovery-aware abstention across hazard domains and interaction classes.

    Source: §1 · p. 2

    The system uses a frozen capability catalog and evidence index, and the 141 instances do not exhaust event, regional, sensor, or operational settings.

    Source: J LIMITATIONS · p. 24

Abstract

from arXiv · show

Understanding and assessing natural hazards is essential for disaster preparedness and risk reduction. Recent advances in large language models have spurred growing interest in AI agents for hazard analysis, particularly their ability to integrate scientific data, models, and tools into automated workflows. However, effective automation requires agents to determine which scientific methods are appropriate for a given event and executable with the available data and tools. As new evidence and execution results become available, these conditions can change, requiring agents to reconsider their choices. We formulate this problem as state-dependent scientific route selection and introduce HazardWeaver. Specifically, HazardWeaver first leverages the Hazard Knowledge Compiler to extract evidence-linked conditions governing scientific applicability, then its Hazard Capability Graph represents executable scientific capabilities and checks compatibility between their inputs and outputs. Using these complementary representations, the Hazard Weaver Agent component selects applicable and executable routes, carries out their workflows, and revises its decisions as the analysis state changes. To evaluate both the scientific outputs and the decisions that produce them, we introduce the Hazard Weaver Benchmark, comprising 141 instances across seven single-hazard domains and four multi-hazard interaction classes. The benchmark accommodates multiple valid scientific routes and evaluates output correctness, route validity, and justified abstention. Extensive experiments on this benchmark show that HazardWeaver outperforms existing agent systems, with the largest gains on tasks with multiple eligible scientific routes. Our code is publicly available at https://github.com/LabRAI/HazardWeaver.

1 INTRODUCTION

Hazard analysis can have several scientifically plausible routes, but the appropriate route depends on both event evidence and currently available resources. HazardWeaver formalizes this as state-dependent route selection and pairs an evidence-grounded, adaptive agent with a benchmark that evaluates artifacts and route decisions.

1 INTRODUCTION

  • Route choice must be grounded in scientific applicability and current executability, then reconsidered as evidence, tools, or downstream requirements change.Multiple routes may estimate the same target, and compound hazards can alter conditions for assessing another hazard.Source: §1 · pp. 1–2
  • HazardWeaver combines evidence-linked applicability conditions from HKC with typed capability compatibility in HCG, enabling HWA to select, execute, and revise routes as the analysis state evolves.The language model manages route decisions and execution flow; scientific models and solvers produce numerical, spatial, and probabilistic outputs.Source: §1 · p. 2
  • HWB evaluates artifact correctness, route validity, and recovery-aware abstention across 141 instances spanning seven single-hazard domains and four multi-hazard interaction classes.The benchmark is built from scientific datasets and benchmarks and assesses both artifact correctness and route validity.Source: §1 · p. 2
  • The paper formulates hazard analysis as state-dependent scientific route selection, requiring both evidence-based applicability and capability reachability for eligibility.Eligibility is updated throughout execution.Source: §1 · p. 2

2 PRELIMINARIES

The paper defines a route as usable only when it is scientifically applicable and executable with capabilities available in the current analysis state. Its central problem is to keep selecting eligible routes as that state changes, or abstain when none can be reached within the remaining budget.

2 PRELIMINARIES

  • A task instance includes its hazard setting, task, event and environmental context, requested output, and information cutoff; the analysis state tracks artifacts, capability availability, and remaining budget.Scientific routes and executable capabilities are represented separately in the problem setup.Source: §2 · p. 3
  • A route is eligible at state s_k only if it passes both evidence-grounded scientific applicability and capability-reachability checks.Scientific applicability is resolved from evidence attached to the route; capability reachability is resolved from executable assets available in the state.Source: §2 · p. 3
  • The problem is to execute an eligible route at each committed step, recomputing eligibility as new evidence, artifacts, or capability availability change the state.The returned output must belong to the task's requested output space.Source: §2 · p. 3

3 METHODOLOGY

HazardWeaver forms eligible routes by combining evidence-grounded scientific applicability with capability reachability. Its components then support workflow selection and revision, while HWB evaluates artifacts and decisions under route-aware criteria.

3 METHODOLOGY

  • The methodology presents the system as a sequence from scientific applicability through capability reachability and adaptive route execution to route-aware evaluation.Figure 2 depicts the intersection of HKC and HCG outputs as the eligible route set for HWA.Source: §3 · p. 3, Figure 2 · p. 4

3.1 HAZARD KNOWLEDGE COMPILER: SCIENTIFIC APPLICABILITY

  • HKC extracts route conditions from scientific evidence and resolves them against task evidence, recording each condition's role, scope, direction, and supporting source.The conditions cover assumptions, required observations, validity ranges, scale, uncertainty, and known failure conditions; condition status is recomputed from the current state.Source: §3.1 · p. 3

3.2 HAZARD CAPABILITY GRAPH: SCIENTIFIC CAPABILITY COMPOSITION

  • HCG represents capabilities and typed artifact transfers in a directed graph, checking whether available assets can realize a route with compatible adjacent inputs and outputs.Information-changing operations such as aggregation and interpolation remain explicit capabilities in the workflow provenance.Source: §3.2 · p. 4

3.3 HAZARD WEAVER AGENT: ADAPTIVE ROUTE SELECTION

  • HWA combines HKC applicability with HCG reachability to identify eligible routes, selects and executes a workflow, and reassesses eligibility as execution changes the analysis state.If no eligible route remains, it explores permitted recovery actions within the remaining budget; failure to find a route is not proof that none exists.Source: §3.3 · p. 5

3.4 HAZARD WEAVER BENCHMARK: ROUTE-AWARE EVALUATION

  • HWB evaluates final artifacts alongside route commitments and execution state, using hidden benchmark-side route criteria and provenance, temporal-cutoff, and replay checks.Its interventions vary evidence, capability availability or compatibility, post-start state, or an upstream hazard variable while holding other task factors fixed.Source: §3.4 · pp. 5–6

4 EXPERIMENTAL SETUP

The experiments compare systems on the same sealed benchmark, capability catalog, and evaluator, using Decision-Constrained Accuracy as the primary metric. The setup also reports artifact and trajectory validity and uses five backbones for ablations.

4 EXPERIMENTAL SETUP

  • The primary comparison uses 141 sealed HWB instances, with 90 single-hazard and 51 multi-hazard cases; 45 instances admit one route and 96 admit multiple routes.The benchmark draws on eleven published datasets, operational products, and scientific benchmarks.Source: §4 · p. 6
  • HazardWeaver is compared with adapted planning, self-consistency, tree-search, AutoML, reflection, and data-science agent baselines under the same inventory, policy backbone, route catalog, replay, and evaluator.The listed systems include Plan-and-Solve, DisasterBench-ToT, MLE-STAR, AutoML-Agent, Reflexion, AIDE, DS-Agent, R&D-Agent, and ReAct.Source: §4 · p. 6
  • Decision-Constrained Accuracy is the primary metric, alongside artifact validity, aggregate trajectory validity, and Macro DCA, which weights the 11 tracks equally.Binomial rates use Wilson 95% confidence intervals.Source: §4 · p. 6
  • The five ablation backbones are OLMo-3.1-32B, Mixtral-8x22B, Gemma-4-31B-it, Llama-3.3-70B, and DeepSeek-V4.1-Flash.The evaluated variants are listed in Appendix D.Source: §4 · p. 6

5 EXPERIMENTS

The experiments compare HazardWeaver with agent baselines, test route-eligibility mechanisms, and assess performance across backbones and task structures. HazardWeaver performs best overall, with its largest gains on multiple-route tasks.

5.1 SYSTEM PERFORMANCE

  • 89.4% overall DCA versus 41.1% for the strongest baseline; HazardWeaver reaches 93.8% on multiple-route tasks.Plan-and-Solve performs well with one route but declines sharply when multiple routes are available.Source: §5.1 · p. 7, Table 1 · p. 7

5.2 COMPONENT AND MECHANISM ANALYSIS

  • Across five backbones, removing either HKC conditions or HCG typing lowers DCA, with the largest reported drop on Llama from 93.8% to 45.8% or 47.9%.HKC filters routes whose assumptions do not match task context; HCG prevents incompatible input-output compositions.Source: §5.2 · p. 7, Figure 3 · p. 7
  • Updated eligibility raises pooled DCA from 16.0% to 41.5% across all five backbones on 40 instances where eligibility can change during execution.A fixed eligible set retains routes that later become invalid.Source: §5.2 · p. 8, Table 2 · p. 8

5.3 GENERALIZATION ACROSS BACKBONES AND TASK STRUCTURES

  • Overall DCA ranges from 60.3% to 89.4% across five backbones, while artifact validity broadly follows DCA and trajectory validity remains consistently high.The remaining separation is concentrated in producing the required scientific artifact and selecting a valid route.Source: §5.3 · p. 8, Table 3 · p. 8
  • Llama and DeepSeek remain above 80% on multi-hazard tasks, while the performance gap widens for Mixtral and Gemma.Composed analyses require carrying intermediate state across stages and selecting compatible downstream workflows.Source: §5.3 · p. 8, Figure 4 · p. 8
  • Backbone scores cluster on one-route tasks but separate more widely on multiple-route tasks, where Llama and DeepSeek perform better than weaker policies.Flood is the clearest shared bottleneck, while compound tracks such as multi-peril impact and earthquake-triggered landslide separate backbones more strongly.Source: §5.3 · pp. 8–9, Figure 4 · p. 8

6 RELATED WORK

Related work establishes hazard-analysis tools and executable workflow representations, while HazardWeaver connects those capabilities to scientific applicability and state-dependent route decisions.

6 RELATED WORK

  • Existing workflow representations capture executable dependencies, but task-conditioned scientific applicability remains largely external to route structures.Selecting among methods targeting the same output also depends on variables, units, spatial and temporal support, resolution, uncertainty, provenance, and operating scale.Source: §6 · p. 9
  • HazardWeaver grounds alternative route families in evidence-linked conditions and composes them through typed executable capabilities.This connects scientific applicability to executable workflow structure.Source: §6 · p. 9
  • HazardWeaver recomputes eligible scientific routes after each executed step and evaluates artifact correctness, committed-route validity, and justified abstention.This addresses route selection as evidence and execution outcomes change the current scientific state.Source: §6 · p. 9

7 CONCLUSION

HazardWeaver frames hazard analysis as state-dependent scientific route selection and combines literature-grounded applicability, typed capability composition, and workflow updates. Across 141 sealed instances, it reaches 89.4% DCA versus 41.1% for the strongest baseline, with the largest gains on multiple-route tasks.

7 CONCLUSION

  • 89.4% DCA versus 41.1% for the strongest baseline across 141 sealed instances, with the largest gains on multiple-route tasks.The instances span seven single-hazard domains and four multi-hazard classes.Source: §7 · p. 9
  • HazardWeaver grounds applicability in literature evidence, composes models and tools through typed capability relations, and updates eligible workflows as analysis evolves.Source: §7 · p. 9
  • Ablations and evaluation across five backbones confirm the contributions and transferability of the route-selection architecture.Source: §7 · p. 9

A HWB SCOPE AND BENCHMARK CONSTRUCTION

HWB is a sealed benchmark of hazard-analysis tasks and candidate scientific routes, constructed from scientific sources and explicit evaluation contracts. Its construction also specifies route applicability and executable capabilities, with typed inputs and outputs and auditable operations.

A.1 BENCHMARK SCOPE

  • HWB contains 141 sealed instances across seven single-hazard domains and four multi-hazard interaction classes, drawing on sources that cover eight hazard task areas.The source families provide observed events, forcing variables, operational products, and reference outputs.Source: §A.1 · p. 17

A.2 HAZARD AND ROUTE STRATA

  • At the initial state, 45 instances admit one registered route and 96 expose multiple candidates; hazard composition and route multiplicity are independent strata.Multiple-route cases occur in both single- and multi-hazard settings; Figure A.1 reports counts across tracks and Figure A.2 reports joint hazard-and-route composition.Source: §A.1 · p. 17, §A.2 · p. 17, Figure A.1 · p. 17

A.1 BENCHMARK SCOPE

  • The construction atlas contains 3,062 registered scenarios, of which 542 were materialized, but headline results use only the sealed 141-instance inventory.Selected instances bind tasks to output contracts, candidate routes, capability states, and task-native evaluators while preserving scientific-source diversity.Source: §A.1 · p. 17

A.2 HAZARD AND ROUTE STRATA

  • Each instance specifies a target, initial state, information cutoff, candidate routes, capability state, output contract, and task-native evaluator.The evaluator checks the committed route against the benchmark-side criterion bound to its scenario.Source: §A.2 · p. 17

A.3 CONSTRUCTION AND SEALING

  • Construction maps scientific data and operational products to explicit task outputs and evaluation criteria, instantiates candidate routes, runs schema, provenance, cutoff, and replay checks, then freezes the inventory and contracts.Table A.1 identifies the scientific data sources used in construction.Source: §A.3 · p. 17, Table A.1 · p. 18
  • Every sealed instance has a construction-time workflow that can produce its required artifact, establishing solvability without prescribing the evaluated agent’s execution trajectory.Source: §A.3 · p. 17

B.1 ROUTE CARD REPRESENTATION

  • Route Cards record the scientific and executable information needed to assess applicability and realizability; Full HazardWeaver resolves their frozen HKC-derived conditions and revisits them when permitted evidence changes.The HKC-off condition removes evidence-linked applicability while retaining the rest of the evaluation pipeline; Table B.1 summarizes principal Route Card fields.Source: §B.1 · p. 18, Table B.1 · p. 18

B.2 TYPED CAPABILITY REPRESENTATION

  • HCG represents 82 specifications for 79 executable capabilities with typed input and output ports; information-changing operations remain explicit, and each agent operation inspects state, executes a capability, or records an auditable transition.The ports record variables, units, representations, spatial and temporal support, resolution, uncertainty metadata, and provenance requirements; Tables B.2 summarizes grounded tools.Source: §B.2 · p. 19, §B.3 · p. 19, Table B.2 · p. 19

C EVALUATION AND REPORTING PROTOCOL

The protocol evaluates artifact quality jointly with route and execution validity, while retaining task-specific evaluation criteria and reporting uncertainty and paired comparisons. Its primary accuracy measure therefore requires all applicable decision and execution criteria to be met.

C.1 DECISION-CONSTRAINED ACCURACY

  • A DCA pass requires task-native artifact success plus route, provenance, typed and causal execution, and applicable resource validity; abstention counts only under the registered recovery criterion.Missing, incomplete, or unevaluated submissions remain in the denominator and count as failures.Source: §C.1 · p. 19

C.2 TASK-NATIVE EVALUATORS

  • HWB preserves the evaluator appropriate to each output rather than forcing heterogeneous artifacts into a common numerical error.Exact thresholds, output fields, and units are stored with sealed instance contracts; Table C.1 summarizes evaluator families.Source: §C.2 · p. 19, Table C.1 · p. 20

C.3 STATISTICAL REPORTING

  • Instance-level proportions use Wilson 95% confidence intervals, and baseline comparisons use exact paired McNemar tests over the same 141 instances.Macro DCA is the unweighted mean of 11 track-level rates, with its 95% interval estimated by 5,000 track-resampling replicates using seed 42.Source: §C.3 · p. 19

D IMPLEMENTATION DETAILS

The implementation records route and execution decisions alongside generated artifacts and provenance. Baselines are adapted to a shared route-selection interface and evaluated through the same replay and decision-accuracy stack.

D.1 LANGUAGE-MODEL BACKBONES

  • Table D.1 identifies the policy backbones used with Full HazardWeaver, while non-policy components remain fixed.Source: Table D.1 · p. 20

D.2 EXECUTION RECORDS

  • Each run records route choices, capability invocations, artifacts, provenance edges, state transitions, and terminal submission; registered capabilities generate outputs while the policy controls selection and execution flow.Source: §D.2 · p. 20

E BASELINE ADAPTATIONS

  • Baseline adaptations preserve each method’s control strategy while withholding method-specific tools, private reference artifacts, and evaluator feedback during route selection.Methods share the task description, route cards, capability identifiers, output requirements, and execution interface, and use the same sealed replay and DCA evaluation; Table E.1 summarizes adaptations and rounds.Source: E BASELINE ADAPTATIONS · pp. 20–21, Table E.1 · p. 21

F.1 EXACT RQ2 ABLATION RESULTS

The ablations and paired comparisons report stronger route-selection outcomes for HazardWeaver, while validation, review, and execution traces examine the evaluator and route decisions. The appendix also documents track-level patterns and a reproducible evaluation process.

F.1 EXACT RQ2 ABLATION RESULTS

  • Updated eligibility passes 83 of 200 evaluated backbone–cell combinations, versus 32 of 200 under fixed initial eligibility.Removing either evidence-linked applicability conditions or typed capability composition reduces DCA for every backbone across the 96 multiple-route instances.Source: §F.1 · p. 21, Table F.1 · p. 21

F.2 PAIRED SIGNIFICANCE TESTS

  • HazardWeaver uniquely solves 75 instances against Plan-and-Solve and 84 against ReAct, while those baselines solve seven and zero exclusively, respectively.The instance-paired comparisons control for task difficulty across the same 141 cases; all exact McNemar tests are significant by a wide margin.Source: §F.2 · p. 22, Figure F.1 · p. 22

F.3 JOINT HAZARD AND ROUTE STRATA

  • Figure F.2 reports DCA across strata defined by hazard composition and route multiplicity, with successful-instance counts over each stratum size.The caption identifies the dimensions and annotations but does not provide specific stratum values.Source: Figure F.2 · p. 22

F.4 TRACK-LEVEL CROSS-BACKBONE RESULTS

  • Performance spans all 11 tracks: every backbone solves tropical-cyclone trajectories, dynamic flood inundation remains a shared bottleneck, and multi-hazard tracks separate backbones more.Figure F.3 reports track-level DCA for Full HazardWeaver.Source: §F.4 · p. 22, Figure F.3 · p. 23

G.1–G.3 EVALUATOR VALIDATION, INFORMATION SEPARATION, AND ROUTE-REGISTRY REVIEW

  • The evaluator accepted or rejected all 160 controlled fixtures as expected, while information separation keeps references and route judgments outside solver context and frozen before policy runs.A review of 86 route–condition pairs across 44 cells found all sampled witness routes applicable and no contract-valid route omitted from the registry; it was quality control, not proof of an exhaustive catalog.Source: §G.1 · p. 23, §G.2 · p. 23, §G.3 · p. 23

H.1 ROUTE REVISION AFTER AN EXECUTION-STATE CHANGE

  • In a seismic trace, an execution-state change invalidates the current realization, prompting HWA to reconstruct a compatible capability path while retaining the scientific target.The revised route and phase-picking output satisfy the sealed evaluator.Source: §H.1 · pp. 23–24

H.2 FAILURE UNDER AN INCOMPLETE CASCADE

  • A debris-flow volume route fails a task requiring the complete burn–rainfall–debris-flow cascade because its meaningful intermediate output does not satisfy the terminal route and output contract.The evaluator rejects the submission; successful execution of one model alone does not establish a decision-valid multi-stage analysis.Source: §H.2 · p. 24

I REPRODUCIBILITY AND RELEASE CONTENTS

  • Reproduction materializes frozen task and capability versions, runs the selected configuration on all 141 instances, evaluates terminal records, and aggregates per-instance decisions.Released table inputs allow a direct check that manuscript values match sealed evaluation outputs.Source: I REPRODUCIBILITY AND RELEASE CONTENTS · p. 24, Table I.1 · p. 24

J LIMITATIONS

HazardWeaver’s evidence index, capability catalog, and route judgments are frozen for benchmark evaluation. The benchmark spans diverse settings but does not exhaust operational conditions, so deployment requires updated evidence, local calibration, and domain review.

J LIMITATIONS

  • HazardWeaver uses a frozen scientific capability catalog and evidence index, so new evidence, revised standards, or updated models may require recompilation and review.Headline scoring covers routes registered when the benchmark is sealed; newly established routes require catalog extension and resealing.Source: J LIMITATIONS · p. 24
  • The 141 instances span eleven hazard tracks and diverse geographic settings but do not exhaust events, regions, sensor configurations, or operational constraints.Reported validity denotes reproducible agreement with the benchmark’s frozen, documented criteria.Source: J LIMITATIONS · p. 24
  • Operational forecasting and emergency decision-making require updated evidence, local calibration, and domain review.Source: J LIMITATIONS · p. 24

Related papers

Loading 2610.03591v1…