Source-linked AI summary
AFDBench: A Reasoning-First AI Scientist for NationalWeather Service Forecast Discussions
Manmeet Singh, Somnath Luitel, Prabhjot Singh, Manraaj Banga, Naveen Sudharsan, Josh Durkee
TL;DR
LLMs can hallucinate numerical values and fail to produce professional, input-grounded weather discussions. AFDBench benchmarks this task and trains a 7B-parameter model with GRPO and domain-specific rewards. On held-out offices, GRPO nearly doubles Style-Align and improves Input-Grounding, while single-timestep input limits Met-Align.
Problem
AFDs carry life-safety consequences, while zero-shot open-source LLMs score ∼0.33 on Style-Align and ∼0.88 on Input-Grounding when generating them from structured weather data.
Method
AFDBench pairs 7,732 expert discussions from 13 NWS offices with real AI weather forecasts and applies GRPO using temperature, synoptic, and format rewards.
Results
GRPO nearly doubles Style-Align from 0.318 to 0.619 and improves Input-Grounding from 0.881 to 0.940 on 1,033 held-out samples from two unseen NWS offices.
Takeaways & Limitations
AFDBench provides standardized tools for evaluating generative meteorological reasoning, while reinforcement learning teaches a 7B-parameter model to write in the NWS register and interpret WeatherNext 2 data.
Takeaways & Limitations
Met-Align remains bounded at ∼14% because WeatherNext 2 supplies one 6-hour forecast timestep while human AFDs synthesize multiple forecast periods.
Abstract
from arXiv · showhide
Large language models (LLMs) hallucinate numerical values when generating high-stakes meteorological text, posing risks for weather communication. We present AFDBench, an AI meteorologist that generates professional Area Forecast Discussions (AFDs) by reasoning through structured AI weather forecast data from Google's WeatherNext 2. We introduce AFDBench, the first benchmark for evaluating generative meteorological reasoning, comprising 7,732 expert written discussions from 13 National Weather Service (NWS) offices paired with real AI weather forecast inputs, and three complementary metrics: Met-Align (numerical accuracy), Style-Align (professional dialect adherence), and Input-Grounding (fidelity to source weather data). Zero-shot evaluations reveal that open-source LLMs achieve low Style-Align (~0.33) and moderate Input-Grounding (~0.88), failing to write in the professional NWS register or faithfully use their input data. We apply Group Relative Policy Optimization (GRPO) with domain-specific rewards targeting temperature accuracy, synoptic correctness, and format compliance. On 1,033 held-out samples from two unseen NWS offices, GRPO nearly doubles Style-Align from 0.318 to 0.619 and improves Input-Grounding from 0.881 to 0.940, demonstrating that reinforcement learning teaches a 7B-parameter model to write like a professional meteorologist and faithfully interpret AI weather data.
1. Introduction
AFDBench addresses the unresolved task of translating AI weather predictions into professional, synoptically reasoned Area Forecast Discussions. It benchmarks this task and applies domain-specific reinforcement learning to improve meteorological style and fidelity to forecast data.
- AFDs convert complex atmospheric model data into actionable language, but errors can affect agricultural decisions and severe-weather evacuations.
- Numerical weather AI systems predict gridded atmospheric fields rather than expert-level text, leaving the translation from forecasts to reasoning-rich communication unresolved.
- Zero-shot open-source LLMs score ∼0.33 Style-Align and ∼0.88 Input-Grounding, indicating generic prose and incomplete use of supplied weather data.
- AFDBench pairs 7,732 expert discussions from 13 NWS offices with real AI weather forecasts and evaluates Met-Align, Style-Align, and Input-Grounding.
- Domain-specific GRPO nearly doubles Style-Align from 0.318 to 0.619 and improves Input-Grounding from 0.881 to 0.940 on a 7B-parameter model.
3. Geographic generalization: the trained model pro-
The trained model produces faithful Area Forecast Discussions for two offices excluded from training, supporting geographic generalization beyond station-specific memorization.
- AFDBench produces faithful AFDs for two forecast offices held out during training, demonstrating learned meteorological reasoning rather than station-specific memorization.
2. Related Work
Prior work supplies numerical weather predictions and general approaches for specialized text generation, hallucination mitigation, and output alignment. AFDBench targets the distinct challenge of producing meteorological discussions requiring both numerical precision and domain-specific reasoning.
- GraphCast, Pangu-Weather, GenCast, and WeatherNext predict atmospheric variables on grids, providing numerical inputs for AFDBench’s text-generation task.
- Prior clinical, legal, and scientific text-generation systems do not target professional meteorological discussions requiring numerical precision and domain-specific reasoning.
- Chain-of-thought prompting, self-consistency, and retrieval-augmented generation are established approaches for reducing LLM hallucinations.
- AFDBench adapts GRPO for scientific text generation by using verifiable numerical accuracy as a reward instead of subjective preference labels.
3. AFDBench: A Benchmark for Meteorological Reasoning
AFDBench frames AFD generation as producing professional, numerically faithful text from structured WeatherNext 2 forecasts. Its geographically diverse dataset, reasoning-first representation, and complementary metrics evaluate both meteorological content and professional language.
- 3.1. Task Definition: The task is to generate a professional AFD that remains numerically faithful to structured weather data and consistent with NWS standards.
- 3.1. Task Definition: Each input contains a single WeatherNext 2 forecast timestep with surface, comfort-index, upper-air, and ensemble-spread variables.
- 3.1. Task Definition: The station network spans 13 NWS offices across seven U.S. climate regions, with blue training offices and red held-out offices for geographic evaluation.
- 3.1. Task Definition: A single forecast timestep tests interpretation and composition but bounds Met-Align because human AFDs synthesize multiple forecast periods.
- 3.2. Dataset: The dataset contains 7,732 professional AFDs selected for geographic and climate diversity, while January–April 2026 coverage excludes peak tornado and wind seasons.
- 3.2. Dataset: Each example uses Instruction, Input, Thinking, and Output fields, with two offices held out to create a 6,701/1,033 train/test split.
- 3.3. Evaluation Metrics: Met-Align measures overlap between generated and reference multi-digit numerical tokens, ranging from 0% for complete hallucination to 100% for perfect agreement.
- 3.3. Evaluation Metrics: Style-Align measures overlap with NWS professional vocabulary, while Input-Grounding averages checks for temperature, wind direction, and pressure-regime fidelity.
4. Method: Reasoning-First Training Pipeline
AFDBench uses a reasoning-first pipeline that has the model perform synoptic analysis before drafting an Area Forecast Discussion. GRPO then optimizes verifiable weather-domain rewards for numerical grounding, synoptic correctness, and NWS format compliance.
- Reasoning-first training: The training data separates each human AFD’s synopsis/analysis into a dedicated “Thinking” field before the forecast discussion.This ordering mirrors meteorologists’ process of explaining why weather will evolve before stating what will occur.
- Training pipeline: SFT did not measurably improve evaluation metrics in this setting, but its checkpoint initializes GRPO, which produces substantial improvements in Style-Align and Input-Grounding.The reported SFT setup used one training epoch, while GRPO continued from the merged SFT checkpoint.
- Reinforcement learning: GRPO supplies verifiable supervision through domain-specific rewards rather than human annotation.The reward system targets temperature accuracy, synoptic accuracy, and format compliance.
- Reward design: Temperature reward scores outputs by whether generated values fall within 3°F or 10°F of the WeatherNext 2 surface forecast.The tiers assign rewards of 1.0, 0.5, or 0.1, grounding numerical outputs in the input data.
- Reward design: Synoptic reward checks wind direction, pressure regime, and thermal regime against WeatherNext 2 inputs.The maximum component weights are 0.4 for wind direction, 0.3 for pressure regime, and 0.3 for thermal regime.
- Reward design: Format reward scores section delimiters, required AFD headers, and NWS domain vocabulary.It awards points for &&, .SHORT TERM, .LONG TERM, AVIATION, and professional vocabulary usage.
5. Results
On geographically unseen NWS offices, GRPO substantially improves professional style and input fidelity while numerical alignment remains near its baseline; training dynamics and an example show the learned behavior.
- Main Results: 0.318→0.619 Style-Align and 0.881→0.940 Input-Grounding are achieved by GRPO on 1,033 held-out samples from two unseen NWS offices.Met-Align remains approximately constant at ∼14%.
- Training Dynamics: Rtemp saturates at 1.0 immediately, Rsyn stays near 0.8, and Rfmt rises from 0.12 to 0.30 during training.Generation entropy remains near 0.85, indicating stable output diversity without mode collapse.
- SFT vs. GRPO Ablation: SFT leaves Met-Align, Style-Align, and Input-Grounding identical to the zero-shot baseline: 14.05%, 0.318, and 0.881.The reported setting combines 4-bit quantized training, fp16 evaluation, and one training epoch.
- SFT vs. GRPO Ablation: GRPO improves Style-Align and Input-Grounding while Met-Align remains approximately flat across the evaluated models.The authors associate style gains with format and NWS vocabulary rewards, and grounding gains with temperature and synoptic rewards.
- Qualitative Example: A Louisville example shows the model reasoning in a Thinking block before producing a structurally correct AFD with appropriate values and the correct synoptic feature.The output reports “mid to upper 60s” and identifies a moisture-starved boundary, matching the human discussion.
- Geographic Held-Out Evaluation: Style-Align reaches 0.619 and Input-Grounding reaches 0.940 across Boston and Morristown, suggesting learned conventions transfer across distinct climates.The held-out offices represent maritime Northeast and Appalachian Southeast settings.
6. Discussion and Limitations
The discussion identifies single-timestep inputs as the main numerical-alignment boundary and notes limitations involving SFT, reward design, dataset coverage, human evaluation, and architecture.
- Single-Timestep Input Bounds Met-Align: Met-Align plateaus at ∼14% because human AFDs synthesize many forecast periods while WeatherNext 2 supplies one 6-hour timestep.The proposed next step is ingesting multiple forecast timesteps covering the full AFD horizon.
- SFT Ineffectiveness: SFT is ineffective in this setting, potentially because of quantized training, fp16 evaluation, or insufficient training duration.The authors call for deeper SFT with more epochs, higher rank, or full-precision training.
- Reward Signal Concerns: Temperature reward saturation raises a possible reward-hacking concern, although held-out Input-Grounding of 0.940 suggests genuine input interpretation.The concern is that outputs could include broad ranges of plausible temperatures to hit the reward threshold.
- Dataset Scope: The dataset covers 13 U.S. offices from January–April 2026, excluding peak tornado and peak wind seasons and international terminology.This constrains direct coverage of other seasonal and national forecasting settings.
- No Human Expert Evaluation: All reported metrics are automated, so operational evaluation by NWS forecasters remains absent.The paper identifies human expert evaluation as the gold standard for assessing AFD quality.
- Single Architecture: Only Qwen2.5-7B is evaluated, leaving generalization to other architectures and model scales untested.
7. AFDBench on the AI Scientist Spectrum
AFDBench is currently positioned as a human-supervised drafting tool because it lacks real-time observations, situational awareness, and life-safety prioritization. The paper describes a possible path toward co-authorship, while autonomous founder-level operation remains distant.
- AFDBench generates draft forecast discussions that human meteorologists must review, edit, and approve before operational use.The model cannot ingest real-time observations, lacks situational awareness, and cannot prioritize life-safety information.
- Multi-timestep NWP data, observational verification, and multi-objective rewards could enable forecasters to refine first drafts rather than write AFDs from scratch.The paper describes this as a trajectory toward co-author-level assistance.
- Autonomous founder-level operation remains distant but not inconceivable, involving identification of forecast challenges, analysis-strategy design, and warning issuance.The paper raises whether safety-critical weather communication should ever be fully automated or require permanent human oversight.
8. Conclusion
AFDBench combines a 7B-parameter AI meteorologist with standardized evaluation for generating professional NWS discussions from WeatherNext 2 data. GRPO improves stylistic alignment and input fidelity on held-out offices, while single-timestep input limits Met-Align and motivates multi-timestep integration.
- AFDBench is a 7B-parameter AI meteorologist that generates professional NWS Area Forecast Discussions from real AI weather forecast data.
- GRPO nearly doubles Style-Align from 0.318→0.619 and improves Input-Grounding from 0.881→0.940 on held-out offices.The reported improvements teach the model to write in the NWS professional register and interpret WeatherNext 2 data faithfully.
- Met-Align plateaus at ∼14% due to single-timestep input, identifying multi-timestep integration as the critical next step.
- AFDBench provides standardized evaluation tools for the emerging task of generating expert meteorological text from AI weather predictions.
- Future work includes full-horizon multi-timestep WeatherNext 2 input, operational NWS evaluation, deeper SFT, retrieval-augmented baselines, and broader service and model coverage.The listed extensions include international weather services and larger model architectures.
A. Training Hyperparameters
The domain-specific GRPO configuration uses a triple reward over temperature accuracy, synoptic accuracy, and format compliance for generated meteorological discussions.
- The training algorithm requires generated text ˆy and WeatherNext 2 input x as its principal inputs.
- Temperature accuracy rewards generated temperatures according to their minimum absolute error against the input forecast temperature.
- Synoptic accuracy combines wind-direction presence, pressure matching, and thickness matching into a domain-specific reward.
- Format compliance rewards required markers and NWS vocabulary, then combines the three capped reward components.