PaperlayerView exact revision on arXiv

Source-linked AI summary

When Should a Satellite Estimate Be Changed? Stress-Testing Neural Corrections for Evapotranspiration

Marco Trotta

arXiv:2609.35314v1cs.LG

TL;DR

The paper asks whether a selector can tell when a neural correction improves an existing OpenET estimate, especially when weather inputs are unsupported. It evaluates residual corrections and support screening across clean records and input stress tests, finding mixed evidence and no established reliable clean-input advantage.

  • Problem

    A selector must predict whether a neural correction improves the available OpenET estimate, rather than merely whether its neural model is uncertain.

    Source: §1 · p. 1

  • Method

    The study evaluates neural residual corrections and a support rule on screened flux-tower records, including controlled weather transformations and cropland training comparisons.

    Source: §1 · p. 1, §5.3 · p. 3

  • Results

    The evidence is mixed: clean data do not establish a reliable advantage for support screening, while controlled weather changes show cases where it reduces error by rejecting corrections.

    Source: §1 · p. 1

  • Takeaways & Limitations

    The study does not identify a generally reliable rule for changing clean satellite estimates, and crop-only training had lower error on the tested cropland cohort with method-selection uncertainty.

    Source: §8 · p. 5

    The fixed weather transformations show selector responses to chosen input changes but do not estimate natural fault rates, and the study does not establish a reliable clean-input advantage for support screening.

    Source: §7 · p. 4

Abstract

from arXiv · show

Neural residuals can improve satellite evapotranspiration (ET) estimates, but selectors must predict when a correction helps and reject unsupported inputs. We evaluate ten-member models on 16,366 flux-tower observations from 151 stations paired with OpenET, across nine rolling years and five spatial folds. At one held-out station, Gain accepted corrections on all 32 physically invalid records: it predicted a mean benefit of 0.83 mm/day, but the corrections increased mean absolute error by 21.6 mm/day versus OpenET. On spatially held-out unit errors, SupportGain reduced station-macro MAE versus Gain by 0.148 mm/day under wind x3.6 (simultaneous 95% interval, 0.070 to 0.226), with 9.3% acceptance versus Gain's 51.8%; on clean inputs, its 0.006 mm/day advantage had an interval that includes zero. These fault analyses are exploratory; none of 40 preplanned temporal comparisons passed Holm correction, while a separate predeclared cropland contrast found 0.041 mm/day lower station-macro MAE with crop-only training (95% interval, 0.009 to 0.079).

1 Introduction

The paper asks whether a selector can predict when a neural correction improves an existing OpenET estimate, especially under unsupported weather inputs. It tests this decision with retrospective evaluations and reports mixed evidence across clean data, input stress tests, and cropland training.

1 Introduction

  • The evaluation uses 16,366 screened records from 151 stations in rolling tests from 2012 through 2020, which reuse stations and do not test unseen-site transfer.Rejected corrections return the original OpenET estimate, and every test record remains in the score.Source: §1 · p. 1
  • Clean data do not establish a reliable advantage for support screening over learned benefit prediction, while stress tests show cases where screening reduces error by rejecting corrections.The paper also reports a severe station-level failure and a predeclared cropland comparison favoring crop-only training, subject to method-selection uncertainty.Source: §1 · p. 1
  • The study evaluates corrections by error removed from OpenET using spatially cross-fitted selector data, separates observed failures from controlled weather changes, and compares all-station with crop-only training.These retrospective prediction results do not identify irrigation status or measure irrigation response.Source: §1 · p. 1

2 Related work

The paper applies selective regression to deciding whether to defer to the existing OpenET estimate, alongside environmental input-distance screening. It situates that evaluation among ET correction methods and spatial validation concerns while limiting the claims to retrospective actual-ET estimation.

2 Related work

  • Here, selective regression predicts whether a neural correction improves the existing OpenET estimate, with rejected records scored using that OpenET fallback.The decision rule is established; the paper presents an environmental stress test rather than a new deferral theory.Source: §2 · p. 2
  • A nearest-neighbor distance screen is tested beside learned selector scores as a simple check, not a calibrated guarantee of reliable predictions.Input distance and cross-fitting assess resemblance to training data, while ensemble disagreement can change outside that distribution and neural networks may extrapolate along unsupported inputs.Source: §2 · p. 2
  • Prior ET methods use water-balance supervision, physical covariates, and geographic transfer, while this study uses OpenET and flux archives that do not independently validate OpenET development.Land-cover-specific OpenET biases motivate testing a cropland training population.Source: §2 · p. 2
  • Because environmental records are spatially dependent, the study groups nearby stations for inner selection and reports a separate spatial holdout; rolling years test later dates at observed stations, not new locations..Source: §2 · p. 2
  • The target is measured actual ET, with reference ET used only as an input; the study makes no claim about crop water need, irrigation response, or reference-ET forecasting.Source: §2 · p. 2

3 Data and evaluation

The evaluation pairs OpenET estimates with measured ET from screened station-date observations, then tests rolling temporal transfer and scores complete-system predictions with fallbacks. Uncertainty procedures distinguish preplanned selector comparisons from a predeclared cropland contrast.

3.1 Observations and inputs

  • The physical screen retains 16,366 observations from 151 stations, grouped by proximity, and the cropland subset contains 5,203 observations from 58 stations.Observations are sparse validation dates rather than a daily time series; negative measured ET values are retained, and land-cover labels do not identify irrigation status.Source: §3.1 · p. 2
  • The model inputs are OpenET, reference ET, seasonal position, temperature, VPD, and wind, using target-date inputs in a retrospective estimate rather than a forecast.The seasonal encoding repeats every 365 days, and release times are not reconstructed.Source: §3.1 · p. 2

3.2 Rolling temporal evaluation

  • Rolling tests train on earlier years and test each year from 2012 through 2020, with proximity-group inner folds and a crop-only arm formed by filtering the same saved partitions.Stations may appear in both training and test years, so this design tests temporal transfer rather than transfer to unseen locations; outer test labels do not determine fits, scores, or thresholds.Source: §3.2 · p. 2

3.3 Scoring and uncertainty

  • The primary score averages within-station MAE across stations, giving each station equal weight, while the complete-system score uses OpenET whenever a selector rejects a correction.Coverage is the station-weighted share of observations receiving a correction.Source: §3.3 · p. 2
  • The 40- and 52-comparison selector families use Holm correction, while the predeclared cropland contrast uses a paired group bootstrap that keeps fitted models and selectors fixed.The intervals omit retraining uncertainty, and repeated split settings are stability checks rather than independent trials.Source: §3.3 · p. 2

4 Neural correction models

The correction model predicts the residual from OpenET to measured ET, and selectors use the ensemble mean correction and between-network spread. The comparison includes applying every correction and applying none.

4 Neural correction models

  • The ten-network model predicts the residual from OpenET to measured ET, and selectors use the ensemble mean correction and between-network spread.Each network has two 32-unit ReLU layers, one scalar output, and seven input features.Source: §4.1 · p. 2
  • Full applies every correction, whereas OpenET applies none.Source: §4.1 · p. 2
  • Training uses fixed seeds and does not use test-based stopping or architecture search.Each model has 1,345 trainable parameters; the passage reports Adam training for 120 iterations.Source: §4.1 · p. 2

5 When should the correction be applied?

The selector must predict whether a neural correction improves on OpenET, not merely whether the neural model is uncertain. The paper describes selector training and tests input-support screening under measured-weather probes.

5.1 Predicting benefit differs from predicting error

  • The selector targets expected correction benefit over OpenET, because low neural error or ensemble disagreement alone need not indicate improvement over the existing estimate.Positive expected benefit is the decision criterion for applying a correction.Source: §5.1 · p. 3

5.2 Learning and checking the selectors

  • SupportGain requires both positive predicted benefit and acceptance by Support95, while Gain accepts only when its predicted benefit is positive.The neural estimator is the same for every selector; Gain uses a fixed boosted-tree regressor with six inputs.Source: §5.2 · p. 3

5.3 Earlier measured-weather probes

  • Weather probes multiply VPD or wind by 10 or add 20 °C to temperature while holding labels and fitted parameters fixed.Only rows with nonnegative exported actual vapor pressure are eligible; the transformations are not weather interventions or estimates of fault prevalence.Source: §5.3 · p. 3

6 Results

The results show a severe correction failure at one held-out station and mixed evidence for support screening: it helps under selected altered inputs but accepts fewer records. A predeclared cropland comparison favors crop-only training on the tested cohort.

6.1 One station shows how a correction can fail

  • At held-out manilacotton, Gain accepted all 32 physically invalid records and raised station-macro MAE by 21.586 mm day−1 versus OpenET, despite predicting 0.83 mm day−1 benefit.This was one station and one group, with no multi-group interval.Source: §6.1 · p. 3
  • Outside manilacotton, SupportGain’s MAE was 0.015 mm day−1 lower than Gain’s on 133 rule-violating records, but OpenET remained lower at 0.755 mm day−1.These records do not establish a device fault or broad selector advantage.Source: §6.1 · p. 3

6.2 Support screening helps under altered inputs, but accepts fewer records

  • None of the 40 preplanned all-station comparisons passed Holm correction; the smallest adjusted one-sided p-value was 0.126.These tests do not directly compare SupportGain with Gain.Source: §6.2 · p. 4
  • Across 30 settings, SupportGain lowered clean MAE over Gain by a mean of 0.00438 mm day−1, but reused observations did not establish a stable clean advantage.Only one of 30 nominal group-bootstrap intervals excluded zero.Source: §6.2 · p. 4
  • SupportGain had lower point MAE than Gain in all 30 settings under each of 12 synthetic weather transforms; temperature increased by 32 °C yielded a mean improvement of 0.55623 mm day−1.The transforms keep labels and fitted models fixed and do not estimate natural fault prevalence.Source: §6.2 · p. 4
  • Under wind multiplied by 3.6, SupportGain lowered MAE by 0.1478 mm day−1 versus Gain in the post hoc spatial holdout (simultaneous 95% interval [0.0697, 0.2260]).This analysis used the same archive and conditioned on fitted models.Source: §6.2 · p. 4
  • SupportGain accepted 9.3% of records versus Gain’s 51.8%, with coverage below 0.3% under Fahrenheit-as-Celsius and VPD multiplied by 10.These are exploratory results from the post hoc spatial holdout.Source: §6.2 · p. 4

6.3 Crop-only training lowers error on the tested cohort

  • On the same 3,234 cropland test records, crop-only SupportGain lowered MAE by 0.0414 mm day−1 versus all-station SupportGain (95% interval [0.0085, 0.0788]).The interval conditions on the fitted models, selectors, and thresholds.Source: §6.3 · p. 4

7 Discussion and limits

Support screening has no established clean-input advantage, and exploratory fault analyses show a severe single-station failure without establishing a broad selector benefit. Crop-only training improves error on the tested cropland cohort, but selection uncertainty and study scope limit interpretation.

7 Discussion and limits

  • The study does not establish a reliable clean-input advantage for support screening, and the fixed weather transformations do not estimate natural fault rates.The 40-test family does not compare SupportGain directly with Gain; direct clean differences are small and uncertain.Source: §7 · p. 4
  • Gain accepted large corrections on physically invalid weather at one station, while SupportGain fell back to OpenET; other groups showed no broad benefit, and rule violations alone do not prove sensor faults.The outcomes were visible before the exploratory audit, so independent sites with documented faults are needed to test the pattern.Source: §7 · p. 4
  • Crop-only training lowers error on the tested cropland cohort, but the reported interval omits method selection and model refitting.The interval therefore does not account for SupportGain being selected using outcomes from the same archive, including test records.Source: §7 · p. 4
  • Gain predicted benefit for all 32 held-out-station records despite correction size and ensemble spread exceeding their inner-training 95th percentiles.Figure 2 shows the records individually.Source: Figure 2 · p. 5
  • The cropland result supports population matching within this archive, not general crop performance.The study does not identify irrigated fields or test irrigation interventions, and all 360 cropland-run neural fits reached the iteration limit with convergence warnings.Source: §7 · p. 5
  • Rolling-year tests reuse stations and do not measure unseen-site transfer; land-cover labels do not identify irrigated fields, and the study does not test irrigation interventions or forecast skill.The measured-weather analysis is exploratory, and available gridMET data provide no device-accuracy scale.Source: §7 · p. 5

8 Conclusion

The study does not identify a generally reliable rule for when a neural correction should change a clean satellite estimate. Its findings remain bounded by method-selection uncertainty and require confirmation at independent sites with documented input faults.

8 Conclusion

  • The study finds no generally reliable clean-estimate correction rule, while independent sites and documented input faults are needed to confirm its severe selector failure and cropland result.The cropland finding is limited to the tested cohort and carries method-selection uncertainty.Source: §8 · p. 5

Data, code, and AI assistance

The paper describes its reproducibility materials and presents figures on rule-violating records, weather transformations, clean-data screening, and cropland training. Eight of nine yearly estimates favor crop-only training, while the pooled contrast also favors that approach.

Data, code, and AI assistance

  • The reproducibility package includes checksums, cohort rows, split assignments, model predictions, protocols, tests, and figure-generation scripts.The source datasets are available through cited Zenodo records, and the repository contains model runs, selector protocols, and saved predictions.Source: Data, code, and AI assistance · p. 5
  • Among 133 rule-violating records outside the known station, only two of 17 groups show a Gain–SupportGain difference; OpenET has the lowest station-macro MAE.Rejected corrections return the OpenET estimate.Source: Figure 3 · p. 6
  • Figure 4’s top row shows MAE reduction over OpenET and its bottom row station-weighted acceptance across fixed weather changes.Wind labels are multipliers, temperature labels are Celsius offsets, and the VPD panel tests one kPa-to-hPa error.Source: Figure 4 · p. 7
  • Figure 5 shows clean SupportGain improvement over Gain across ten inner split seeds and three proximity thresholds, with bootstrap intervals and split-seed variability.Whiskers are 95% group-bootstrap intervals; shading is the mean plus or minus one standard deviation across split seeds.Source: Figure 5 · p. 7
  • Eight of nine yearly estimates favor crop-only training, but the years reuse stations and groups rather than providing independent replications.The pooled all-station minus crop-only SupportGain error difference is 0.0414 mm day−1 (95% group-bootstrap interval [0.0085, 0.0788]).Source: Data, code, and AI assistance · p. 8, Figure 6 · p. 8
  • Figure 6 plots yearly and pooled all-station minus crop-only SupportGain error, where positive values favor crop-only training.The pooled difference is 0.0414 mm day−1 (95% group-bootstrap interval [0.0085, 0.0788]).Source: Figure 6 · p. 8

A Exploratory measured-weather fault and repair analyses

The appendix reports exploratory corrected three-member analyses on an earlier measured-weather cohort, after the original results had already been inspected.

A Exploratory measured-weather fault and repair analyses

  • The exploratory appendix recomputes corrections, spread, and support distance for transformed inputs on 7,758 rows from 84 stations and 62 proximity groups.It uses three seeds and an earlier measured-weather cohort; the original results had been inspected before correction.Source: A Exploratory measured-weather fault… · p. 9

A.1 Weather fault types

The appendix probes weather-fault transformations and selector repairs, then compares Gain and SupportGain on spatially held-out groups. The fault tests keep labels and fitted model parameters fixed, and the bootstrap intervals omit retraining uncertainty.

A.1 Weather fault types

  • Weather-fault probes vary wind, temperature, and VPD while keeping labels and fitted parameters fixed; they do not estimate natural fault prevalence.Dose tests include wind multipliers from 0 to 10 and temperature offsets up to 32 °C; additional tests use Fahrenheit as Celsius or alter wind values in several ways.Source: §A.1 · p. 9, Figure 7 · p. 9

A.2 Selector repairs

  • LocalShrinkage has clean station-macro MAE 0.806 mm/day versus SupportGain's 0.825, with an improvement interval of [0.001, 0.037] across 62 groups.The Holm-adjusted p-value is 0.154 across eleven post hoc comparisons; LocalShrinkage and Uniform differ by 0.003 mm/day, with interval [-0.009, 0.019].Source: §A.2 · p. 9
  • Under tenfold wind, MonoGain has station-macro MAE 1.230 mm/day versus Gain's 1.242; AugmentedGain's comparisons with SupportGain do not establish a confirmed selector repair.Under tenfold wind, AugmentedGain has MAE 0.862 and SupportGain 0.853; its improvement interval is [-0.0231, 0.0037] mm/day, and none of four comparisons passes Holm correction.Source: §A.2 · p. 9

A.3 Spatial holdout sensitivity

  • The spatial analysis pools five folds on 16,366 clean-cohort rows from 151 stations and 102 groups, with each test-group row predicted by a model excluding its station.It uses all dates from other groups, so it does not test forecasting; intervals condition on fitted models and omit retraining uncertainty.Source: §A.3 · pp. 9–10
  • Table 1 reports Gain-minus-SupportGain station-macro MAE across five spatial folds, with positive differences favoring SupportGain.Its intervals are simultaneous across the five test conditions.Source: Table 1 · p. 10

Related papers

Loading 2609.35314v1…