PaperlayerView exact revision on arXiv

Source-linked AI summary

Towards Explainable Benchmarking for Data-driven Post-Wildfire Debris Flow Prediction

Zhisheng Qi, Li Zhu, Utkarsh Sahu, Douglas Tommey, Josh Roering, Yu Wang

arXiv:2610.07358v1cs.LG

TL;DR

Data-driven PFDF prediction remains fragmented across features, models, and evaluation protocols, with limited systematic analysis of heterogeneous drivers. The paper benchmarks diverse models and feature configurations and uses reinforcement learning-based feature selection; learned models outperform the fixed USGS baseline, while important factors vary across regions.

  • Problem

    PFDF prediction studies are fragmented across feature settings, model architectures, and evaluation protocols, and lack systematic analysis of heterogeneous drivers.

    Source: §1 · p. 2

  • Method

    The paper establishes a unified benchmark and uses reinforcement learning-based feature selection to identify factors that distinguish positive from negative PFDF events.

    Source: §1 · p. 2, §2.2 · p. 2

  • Results

    Learned models consistently outperform the fixed operational USGS logistic-regression baseline, augmented features further improve performance, and feature-importance patterns vary substantially across regions.

    Source: §4 · p. 5

  • Takeaways & Limitations

    Globally fixed coefficients are insufficient across diverse geographic zones, while adaptive learning models can better capture region-specific dynamics.

    Source: §4 · p. 5

Abstract

from arXiv · show

Post-wildfire debris flows (PFDFs) are destructive sediment-laden hazards triggered when intense rainfall strikes recently burned terrain, destabilizing hillslopes and threatening infrastructure, local economies, and community safety. Data-driven methods have been proposed to learn predictive patterns directly from historical PFDF observations. However, the current research landscape of data-driven PFDF prediction remains highly fragmented across feature spaces, model architectures, and evaluation protocols, making rigorous comparison and the derivation of scientific insights difficult. Moreover, existing studies lack a systematic investigation into the relative importance of heterogeneous factors (e.g., meteorological conditions, terrain characteristics, soil properties, and burn severity) in triggering PFDF. To address these limitations, we present a unified benchmark for data-driven PFDF prediction, enabling fair and comprehensive evaluation across diverse models and feature configurations. Furthermore, to better understand the underlying drivers of PFDF formation, we propose a reinforcement learning-based feature selection framework that identifies factors whose perturbations render positive and negative events indistinguishable, thereby discovering the regional underlying mechanisms of PFDF occurrence across regions. Our code and benchmark are publicly available at https://github.com/KINDLab-Fly/PFDF-Benchmark.

1 Introduction

Postfire debris flows are destructive rainfall-triggered hazards that threaten people, infrastructure, and local economies. The paper addresses fragmented prediction research and limited understanding of PFDF drivers with a unified benchmark and reinforcement learning-based feature selection.

1 Introduction

  • Intense rainfall over recently burned terrain can trigger destructive debris flows that threaten human life, critical infrastructure, and local economies.Fire-induced reduced infiltration, hillslope destabilization, and increased sediment availability can contribute to rapid, high-energy flows after heavy precipitation.Source: §1 · p. 1
  • PFDF research has progressed from empirical rainfall thresholds and fixed logistic regression toward nonlinear classifiers and process-linked learning, alongside flume studies [8].Related landslide-warning work has also applied deep learning to forecast hydrologic response from rainfall and soil moisture.Source: §1 · p. 1
  • Fragmented features, model architectures, and evaluation protocols impede fair comparison, while black-box prediction limits scientific understanding of PFDF formation.The paper proposes a unified benchmark, forecast-derived contextual features, and reinforcement learning-based feature selection to identify global and regional drivers.Source: §1 · p. 2

2 Benchmark Setup

The benchmark predicts binary PFDF occurrence for storm-watershed events using meteorological, wildfire, and geologic or geomorphic inputs. It evaluates fixed and trainable models, augmented features, and a reinforcement learning-based feature-removal approach for identifying factors that distinguish positive and negative events.

2 Benchmark Setup

  • Each sample is a storm-watershed event with a binary observed debris-flow outcome and meteorological, wildfire, and geologic or geomorphic inputs.Inputs include rainfall intensity, storm duration, burn severity, dNBR, terrain slope, and soil erodibility.Source: §2 · p. 2

2.1.1 Dataset.

  • The dataset contains 1,243 events from 33 fires across 10 western-US zones; the complete-case evaluation subset has 1,131 events and 292 positives.The full dataset has 316 positive cases (25.4%); the subset has 292 positives (25.8%).Source: §2.1.1 · p. 2

2.1.2 Feature Setup.

  • Augmented features add terrain, antecedent hydrologic conditions, and LLM-structured forecast indicators to baseline event attributes.The forecast indicators include precipitation threat, extreme rainfall potential, and overall PFDF risk; the schema specifies 18 numeric indicators and one concise textual summary.Source: §2.1.2 · p. 2, Table 1 · p. 3

2.1.3 Baseline and Model Families.

  • The benchmark compares frozen-coefficient USGS physics logistic regression with trainable LR, CART, SVM, NN, RF, and HGB baselines, including augmented-feature variants.The fixed operational model is not retrained.Source: §2.1.3 · p. 2

2.1.4 Evaluation Metric.

  • Threat Score evaluates correct debris-flow detections while penalizing missed events and false alarms; models use zone-stratified splits across four train/validation/test ratios and five seeds.The ratios are 50/25/25, 60/20/20, 70/15/15, and 80/10/10; hyperparameters are selected by validation Threat Score.Source: §2.1.4 · p. 2, §2.1.5 · p. 2

2.2 RL-based Feature Selection

  • The feature-selection framework learns a binary mask that suppresses selected features by replacing them with training-set reference values, to identify features separating positive and negative events.Standardized continuous features use zero as their reference value after normalization.Source: §2.2 · p. 2

3 Experimental Results

The benchmark finds that selected augmented features improve most model families, while global and regional feature-importance patterns point to distinct PFDF drivers across zones. Regional diagnostics show that the fixed USGS model fits some settings better than others, supporting region-aware prediction.

3.1 Benchmarking Performance

  • Selected augmented features improve most model families, particularly SVM, random forest, and neural networks, while modern learning methods outperform the fixed USGS reference.Table 2 reports mean test Threat Scores across five zone-stratified split seeds.Source: §3.1 · p. 3, Table 2 · p. 3

3.2 Global Feature Importance

  • Storm-related variables dominate global feature importance, alongside soil erodibility, burn severity, terrain slope, and geographic zone.The prominence of geographic zone highlights region-specific characteristics in PFDF formation.Source: §3.2 · pp. 3–4, Figure 2 · p. 3
  • OSD and SGSBSJ are dominated by storm and rainfall factors, whereas terrain and geographic features are most influential in SWCO and WMT.The reported feature-importance patterns vary substantially across regions.Source: §3.2 · p. 4

3.3.1 OSD/SGSBSJ.

  • In OSD, expanded storm descriptors separate PFDF from non-PFDF events more strongly than several original USGS inputs, while the fixed formula misses 9 of 25 PFDF events.The cited high-ratio descriptors include storm accumulation, 30/60-minute rain, 15-minute intensity, and average storm intensity.Source: §3.3.1 · p. 4

3.3.2 SWCO.

  • In SWCO, the fixed USGS inputs already align with observed outcomes and miss only 1 of 18 PFDF events; additional learned interactions can slightly reduce performance.The observed PFDF events have larger R, X1, and X2 values than non-PFDF events, while X3 is close to parity.Source: §3.3.3 · pp. 4–5

3.3 Further Analysis

  • Regional PFDF mechanisms are not uniform: some regions benefit from additional susceptibility and storm features, while others are already well described by the fixed USGS structure.The authors conclude that models should empirically learn region-dependent mechanisms.Source: §3.3.3 · p. 5

4 Conclusion and Future Work

The benchmark’s learned models outperform the fixed operational USGS logistic-regression baseline, and augmented physical and meteorological features further improve prediction. Reinforcement-learning feature selection reveals globally dominant storm variables but regionally varying importance patterns; future work will pursue regional physics-informed equations and study system trustworthiness.

  • Learned models consistently outperform the fixed operational USGS logistic-regression baseline, and augmented physical and meteorological features further improve predictive performance.Source: §4 · p. 5
  • Storm-related variables remain globally dominant, but feature-importance patterns vary substantially across regions, revealing distinct regional PFDF mechanisms.The authors conclude that globally fixed coefficients are insufficient across diverse geographic zones, while adaptive learning models can better capture region-specific dynamics and remain physically interpretable.Source: §4 · p. 5
  • Future work will develop physics-informed equations for regional PFDF prediction and investigate prediction-system vulnerabilities to knowledge poisoning and data extraction attacks.Source: §4 · p. 5
Loading 2610.07358v1…