Source-linked AI summary
A joint error metric for scoring CFD against wind tunnel data, scaled by the robust spread of the participants (AutoCFD5 Case 2)
Hendrik Hetmann, Charles Mockett
TL;DR
Unlike pressure, velocity, and force errors cannot be combined directly because their scales differ. The note defines a joint error using robust participant spreads pooled into component scales, with the resulting scores intended for ranking runs on an error–cost diagram rather than pass-or-fail judgments.
Problem
The components differ in kind, magnitude, and number of points, so their deviations cannot be added directly to produce the single error value needed for an error–cost comparison.
Source: §1.1 · p. 1Method
The method scales deviations by robust participant spreads, pools point-wise spreads as a root mean square for each component, and combines component errors with fixed weights.
Source: §1.2 · p. 2, §2.3 · p. 4Results
A component error of 1 means the run’s RMS deviation from experiment equals the RMS participant spread for that component, while the joint error is intended for ranking rather than pass-or-fail verdicts.
Source: §4 · p. 6Takeaways & Limitations
The joint error provides a single value for ranking runs on the error–cost diagram, but its magnitude is not a pass-or-fail judgment.
Source: §1.1 · p. 1, §4 · p. 6The method treats wind-tunnel values as exact, so when measurement uncertainty is not small relative to participant spread, it can penalize differences the measurement cannot resolve.
Source: §5 · p. 7
Abstract
from arXiv · showhide
AutoCFD5 Case 2 compares the participants' CFD results with wind tunnel measurements of surface pressure, velocity profiles and force coefficients. To place each run on a diagram of error against computational cost, these quantities must be combined into one number. This note defines that number, the joint error. Following the practice of interlaboratory proficiency testing (ISO 13528), every deviation from the experiment is expressed in units of the spread of the participants' results at that point, estimated robustly with the standard's Algorithm A. The point-wise spreads are pooled into one scale per component, so that every measurement point carries equal weight within its component. The joint error is a fixed weighted sum of five component errors. The scales are computed once from all AutoCFD5 submissions received by 5 October 2026 and then frozen, so that scores are reproducible and comparable across workshops.
1 Motivation
AutoCFD5 Case 2 needs one error value per run and variant to compare CFD runs with wind-tunnel measurements on an error–cost diagram. The note uses participant spread to scale unlike physical quantities on a common basis, following proficiency-testing practice.
1.1 The task
- Each run is scored separately for configurations 2a and 2b and their difference DD, using pressure, velocity, and force or moment measurements.The DD variant tests whether a run captures the effect of the configuration change.Source: §1.1 · p. 1
- Because the components differ in units, magnitudes, and point counts, each needs a scale before its error can contribute to one value.For example, a 0.05 deviation in pressure, velocity, and drag cannot be added directly.Source: §1.1 · p. 1
1.2 Choosing the scale
- Participant-result spread is the proficiency-testing basis: CFD runs correspond to laboratories, and the wind-tunnel result to the assigned value.ISO 13528:2022 recognizes robust participant spread from the same round as one way to set σpt.Source: §1.2 · p. 2
- Using participant spread gives every component error the same interpretation: one means one typical participant spread from the experiment.The main remaining judgment in the metric is the choice of component weights.Source: §1.2 · p. 2
2 Method
The method computes deviations from wind-tunnel data, estimates robust participant spreads, and pools point-wise spreads into component scales before combining component errors. It requires complete coverage for a joint score and freezes the scales at the stated submission cutoff.
2.1 Deviations from the experiment
- Deviations are computed against experiment values, with pressure shifted to match probe 107, velocity interpolated onto experimental points, and DD formed from configuration differences.Force coefficients are taken as submitted; the pitching moment coefficient is derived from axle lift coefficients.Source: §2.1 · p. 2
2.2 Participant spread at each point
- ISO 13528:2022 Algorithm A estimates each point’s robust spread by iterative clipping and recomputation, limiting a distant result’s influence without discarding it.Every valid run contributes where it has data; points need results from at least five runs to be included.Source: §2.2 · p. 3
- 0.20 is the converged robust spread for the eight-run example, versus 1.77 for the ordinary standard deviation and 0.14 for the seven agreeing values alone.The distant eighth result remains included but no longer inflates the spread twelvefold.Source: §2.2 · p. 3
2.3 One scale per component
- Point-wise spreads are pooled as a root mean square into one scale per component; all velocity lines form one component, while each force coefficient is a single point.Each variant has its own five scales.Source: §2.3 · pp. 3–4
2.4 Component errors and joint error
- A component error is its RMS deviation divided by the pooled scale, and the joint error is a weighted sum with default weights assigning 75% to profiles and 25% to forces.The default weights follow the AutoCFD5 AI/ML technical focus group methodology; other weightings are also defensible.Source: §2.4 · p. 4, Table 1 · p. 4
2.5 Completeness
- A run receives a joint error only if it has values for every component, so runs with missing data are omitted from the error–cost diagram.Leaving data out cannot lower a run’s error; available component errors for incomplete runs are still reported.Source: §2.5 · p. 4
2.6 Freezing the scales
- Scales are computed once from submissions received by 5 October 2026 and then frozen; later or corrected runs are scored against them without changing them.The same frozen scales can score future workshops using the same measurement points.Source: §2.6 · pp. 4–5
3 Design choices
The design pools participant spread to give measurement points equal weight within each component, then combines component errors for ranking. Robust spreads and the error formulation address sensitivity to outliers and preserve a direct deviation-based measure.
3.1 Why pool the spread instead of using a z score at every point?
- Point-wise z scores would overweight low-scatter probes, while pooled spread gives every point equal weight and lets high-scatter regions count in proportion to their deviations.The Cp reference shift at probe 107 illustrates the issue: all runs agree there by construction, so point-wise z scores would give nearby probes undue weight.Source: §3.1 · p. 5
3.2 Why pool the variances rather than average the spreads?
- The pooled scale is the root mean square of point-wise spreads, matching the RMS deviations in the component error and combining spreads through their variances.A run whose deviation at every point equals the local spread scores exactly 1, even when spread varies across points; the arithmetic-mean alternative scores 1.12 in the stated half-at σ, half-at 3σ example.Source: §3.2 · p. 5
3.3 Why not take the spread of each run’s integrated error?
- Taking the spread across runs of each run’s integrated error measures differences in run quality, not how far predictions lie apart, and can rank uniformly poor runs better than mixed-quality ones.That alternative also loses the link to ISO 13528; pooling point-wise spread keeps one scale per component while measuring the spread of predictions.Source: §3.3 · p. 5
3.4 Does the reference probe need special treatment?
- Probe 107 needs no special treatment because its zero deviation and zero spread add zero to both numerator and denominator, leaving component error unchanged.Source: §3.4 · p. 6
3.5 Why a robust spread?
- Algorithm A avoids letting a single failed or misconfigured run inflate the scale for other runs, without requiring a judgment-based outlier-discarding rule.It is described as a standardized compromise between ignoring a distant result and letting it dominate.Source: §3.5 · p. 6
3.6 Why an error rather than a score?
- The joint error is zero at perfect agreement and grows in proportion to deviations; unlike an R2-derived score, it remains linear in deviations within each component.Any strictly monotonic transformation of an axis preserves the Pareto front, and the error is used rather than a score because it does not compress differences between good runs.Source: §3.6 · p. 6
4 Interpretation
Component errors provide a rough scale for interpreting deviations, but the joint error is designed for ranking and error–cost plots rather than pass/fail decisions. The aggregate can also conceal trade-offs and does not measure accuracy against the experiment by itself.
4 Interpretation
- A component error of 1 means the run’s RMS deviation equals the RMS participant spread; ISO 13528’s single-z thresholds are only a rough guide for the weighted joint error.The joint error is intended for ranking and the error–cost diagram, not pass/fail verdicts.Source: §4 · p. 6
- If all participants share a deviation from the experiment, such as from turbulence modeling or a setup difference, all errors can be large because the scale measures field reproducibility, not accuracy.Source: §4 · p. 6
- The joint number can hide opposing force and profile performance, so component errors are reported alongside it.Source: §4 · p. 6
5 Limitations of a spread taken from the participants
Participant-derived scales involve explicit choices about the field, the treatment of runs, and experimental uncertainty. The resulting scales are frozen, but they remain specific to AutoCFD5 and rely on treating wind-tunnel values as exact until their uncertainties are available.
5 Limitations of a spread taken from the participants
- No agreed accuracy requirement exists for these quantities on this geometry, so independently chosen tolerances could reintroduce hidden weights; frozen scales instead follow the previous-round approach for later workshops.ISO 13528 permits participant-derived spreads but notes advantages of criteria set independently of participants.Source: §5 · p. 6
- Scale uncertainty leaves rankings within a component unchanged but acts like uncertainty in the effective weights between components; freezing makes the scales fixed parts of the metric.Source: §5 · p. 7
- Each run counts as a result although ISO 13528 assumes one result per laboratory, so participants submitting many similar or diverse runs can respectively narrow or widen the scale.Correlated runs make the field’s effective size smaller than its run count.Source: §5 · p. 7
- The frozen scales reflect AutoCFD5’s codes, setups, and participants, so future workshops scored against them are compared with that field rather than a universal reference.Source: §5 · p. 7
- The method treats wind-tunnel values as exact, so when measurement uncertainty is not small relative to participant spread, runs can be penalized for unresolved differences.The pooled z′-score equivalent is planned once wind-tunnel uncertainties become available.Source: §5 · p. 7
6 Summary of definitions and parameters
The parameter definitions set robust point-wise spreads, a minimum run count, pooled component scales, and default weights. The scales are frozen from submissions received by the stated cutoff date.
6 Summary of definitions and parameters
- Point-wise spread uses ISO 13528 Algorithm A, with constants 1.483, 1.5, and 1.134; each point requires results from at least 5 runs.Source: §6 · p. 8
- The component scale is the RMS of point-wise spreads, and the velocity scale is one for all lines.Source: §6 · p. 8
- Default weights are Cp / V / CD / CL / CM = 0.30 / 0.45 / 0.15 / 0.05 / 0.05, adjustable in the dashboard; scales are frozen from runs received by 5 October 2026.Source: §6 · p. 8