Accountable and uncertainty-aware evaluation of sensor-based AI under distribution shift: devices, subjects, and nearly three years underground

Benny Platte; Rico Thomanek; Christian Roschke; Marc Ritter

Source-linked AI summary

Accountable and uncertainty-aware evaluation of sensor-based AI under distribution shift: devices, subjects, and nearly three years underground

Benny Platte, Rico Thomanek, Christian Roschke, Marc Ritter

arXiv:2609.09257v1Sep 8, 2026cs.LGstat.MEstat.ML

TL;DR

Sensor-based AI is often evaluated on data sharing the devices, people, and recording period used for training, leaving deployment shifts insufficiently measured. This paper introduces a staged measurement protocol with declared references, repeated-training uncertainty, and quantile-based decisions, then demonstrates it on smartphone geomagnetic localisation across two mines. Its conclusion emphasizes decomposing long-term degradation and selecting models under deployment-relevant shifts rather than trusting random-split rankings.

Problem
Random train-test splits do not reveal how sensor-based models perform when devices, subjects, sites, or recording epochs differ from training conditions.
Method
The paper evaluates deployed models as measurements through cumulative device, subject, and time holdouts, correct-class-count chance references, repeated-training quantiles, and an out-of-present-scope rate.
Results
The protocol demonstrates that long-term degradation should be decomposed before attribution, with class loss explaining 23 points of an apparent 47-point loss and a defensible 23-point upper bound for genuine signature change.
Takeaways & Limitations
Model selection should follow the shifts deployment will impose rather than rankings from random splits, whose agreement with harder-regime rankings the data rule out as strong.
Takeaways & Limitations
Subject effects are demonstrated using two persons per mine without population inference, while the two sites cover only two magnetic distortion regimes and campaign-level route or day resampling remains future work.

Abstract

from arXiv · show

Sensor-based AI systems are rarely operated under the conditions under which they were trained: devices, personnel and recording epochs change, and each change degrades performance in ways a random train-test split cannot reveal. We propose a staged, accountable evaluation protocol that treats the evaluation of a deployed model as a measurement with declared reference levels and a quantified uncertainty. Four cumulative generalisation stages hold out devices, subjects and time. Each stage is judged on quantiles of repeated trainings against chance references with the correct class count, an out-of-present-scope rate exposes silent misdirection towards classes that are no longer present in deployment relative to training, and an explicit decision rule ties roll-out decisions not to means but to 5% quantiles. We demonstrate the protocol on infrastructure-free geomagnetic localisation with smartphone-based recurrent classifiers in two real underground mines, including a replication of the scheme's training stages at the second site. Unchanged models are re-evaluated on data recorded 34 months after the training campaign, on a device generation unknown at training time and with a held-out surveyor. The 5% quantile of their present-conditioned precision there is 0.39 over 299 repeated trainings, 16.5 times the chance level; across the composition of the 42 reachable location classes the figure varies by +/-0.08, several times the spread between repeated runs. Repeated trainings of a single configuration show why means mislead: a bimodal configuration passes a mean-based test decisively while its 5% quantile lies more than an order of magnitude below chance.

1 Introduction

Sensor-based models face device, personnel, and temporal shifts that random splits do not reveal. The paper proposes an accountable protocol using staged holdouts, reference levels, uncertainty, and quantile-based deployment decisions.

1 Introduction: The protocol cumulatively holds out devices, subjects, and recording epochs, while declaring chance references and quantifying uncertainty through repeated trainings.It treats deployed-model evaluation as a measurement with a defined measurand and explicit reference levels.
1 Introduction: 34 months after training, unchanged models were tested on a later device generation and a surveyor absent from training, extending evaluation beyond random splits.The demonstration used infrastructure-free smartphone geomagnetic localisation in two underground mines.
1 Introduction: +0.14 and +0.41 quantify random-split optimism in matched cross-device and cross-subject scenarios, respectively.The paper attributes this optimism to window overlap and defines a corresponding reporting rule.
1 Introduction: A cross-time decomposition separates class loss, annotation-boundary ambiguity, and bounded residual signature change, guarded by an out-of-present-scope rate.This prevents apparent long-term degradation from being attributed to genuine signal change without accounting for changed class inventories and annotations.
1 Introduction: A bimodal repeated-training configuration passes a mean-based test while its 5% quantile falls more than an order of magnitude below chance.The result motivates rollout decisions based on lower quantiles rather than means.
1 Introduction: Under distribution shift, additional attitude information hurts and random-split representation rankings do not predict rankings under harder deployment stages.The paper treats this as protocol-relevant evidence about evaluation rather than a claim about magnetic physics.

2 Related work

Geomagnetic localisation uses smartphone-recorded field signatures where external infrastructure is unavailable, but prior evaluations mainly use within-campaign splits. The paper targets unreported device, person, and epoch shifts as explicit evaluation factors.

2 Related work: Geomagnetic fingerprinting localises users from position-dependent geological field distortions recorded by smartphone magnetometers without installed infrastructure.Earlier systems include magnetic fingerprinting with an augmented particle filter and LSTM-based approaches using magnetic data.
2 Related work: Single-campaign train-test splits evaluate devices, surveyors, sites, and periods already seen during training rather than deployment conditions.This common pattern limits what reported accuracy says about generalisation beyond the campaign.
2 Related work: Different smartphone magnetometers introduce systematic bias, sensitivity, and noise differences, while prior heterogeneity treatments rely on within-campaign calibration.Performance on entirely unseen devices, people, or epochs remained unreported in the cited localisation literature.
2 Related work: Long-term radio-localisation studies provide longitudinal resources, but the paper reports no comparable years-later re-evaluation resource for unchanged geomagnetic models.The cited WiFi dataset spans 15 months with monthly training and test sets.
2 Related work: Distribution shift is established as a failure mode because in-distribution accuracy can overstate performance under naturally occurring changes across sites, devices, and time.The paper places sensor-based localisation within broader shift taxonomies and benchmark evidence.

3 Experimental setting

The experiments span three campaigns in two underground mines with different geomagnetic structures, smartphone fleets, operators, routes, and recording times. The historic mine supplies the 34-month re-measurement used for cross-time evaluation.

3 Experimental setting: Three campaigns were conducted across two underground sites whose magnetic environments differ in origin and structure.The second site serves as a structural replication of the evaluation scheme under a different distortion regime.
3 Experimental setting: The historic-mine campaign used three smartphones, two operators, 95 direction-resolved location classes, and 110 cleaned sequences containing 804,543 samples.The target classes combine marker identity with movement direction, with dwell markers represented without direction.
3 Experimental setting: The uranium-mine campaign used five smartphone generations across two operators, but device and person effects were coupled because assignments remained fixed.Archived logs did not record which devices or persons the splits held out.
3 Experimental setting: The re-measurement occurred 34 months later with unchanged models, a later device generation, a held-out surveyor, and only 42 of the original 95 classes reachable.Access was restricted to the rear loop, and removed physical markers were reconstructed from planning documents and salient structures.
3 Experimental setting: A 700-sample window spans 14 seconds and roughly 15–20 metres, exceeding half a historic-mine segment; neighbour tolerance therefore corresponds to one marker spacing.Class-exact localisation includes both segment and walking direction.
3 Experimental setting: Geomagnetic anomalies range from −0.72 µT to 1.21 µT, comparable in scale to smartphone magnetometer errors of ±0.1 µT to ±0.45 µT.The task therefore operates close to the sensor noise floor.

4 A staged evaluation protocol under distribution shift

The protocol evaluates deployed sensor classifiers as measurements under cumulative distribution shifts, using macro-precision, correct-class-count chance references, repeated-training quantiles, and an out-of-present-scope safeguard. It escalates from random splits to cross-device, cross-subject, and cross-time conditions.

4 A staged evaluation protocol under distribution shift: Macro-averaged precision is the measurand, and four cumulative stages hold out devices, subjects, and recording epochs under declared conditions.Macro averaging weights locations equally, while uniform-random chance is 1/K despite class imbalance.
4 A staged evaluation protocol under distribution shift: The cross-time stage re-evaluates unchanged models years later on a new device generation and held-out person, accumulating earlier device and subject shifts.This stage addresses the operating-period question.
4 A staged evaluation protocol under distribution shift: Acceptance uses the 5% quantile of repeated trainings against stage-specific chance references, with an out-of-present-scope criterion for predictions assigned to absent classes.At cross-time, present-conditioned metrics use 42 reachable classes; the OOPS ceiling is half the chance predictor’s absent-class prevalence, 0.279.
4 A staged evaluation protocol under distribution shift: The LSTM comparison covers eight representations of the same measured field vector, including magnitude, device-frame components, and gravity-referenced features.When equal-information representations behave differently under shift, the finding concerns evaluation robustness rather than magnetic physics.
4 A staged evaluation protocol under distribution shift: The protocol’s uncertainty analysis uses repeat ensembles to expose training stochasticity, but archived logs omit the optimiser, input normalisation, and random seed needed for bit-exact reproduction.Archived FA+L evaluation columns were also rescaled in two notebooks and subsequently descaled from stored evidence.

5 Results: reference levels and the first three stages

Across the first three evaluation stages, random splits substantially overestimate deployment performance, while device and subject shifts expose degraded and representation-dependent generalisation. Quantile-based reporting and stage-specific comparisons show that the random split is useful as an upper bound, not as an operating estimate.

5.1 Reference levels: the random split as a declared upper bound: 0.992 versus 0.797 and 0.532: median macro precision fell from random split to cross-device and cross-subject evaluation across 112 matched configurations.The paired random-split excess was +0.143 over cross-device and +0.413 over cross-subject evaluation.
5.1 Reference levels: the random split as a declared upper bound: ρs = 0.33 and 0.30: random-split rankings weakly tracked cross-device and cross-subject rankings, while the structural protocols correlated at ρs = 0.89.The weak positive associations arose from the negative-control FA configurations; excluding them, the association vanished or reversed.
5.2 Leakage mechanism and reporting rule: 99.86% overlap at L = 700 and s = 1: near-duplicate windows enter both training and test sets, inflating random-split macro precision.Increasing the step size from 2 to 20 reduced random-split macro precision by 0.042, and increasing it to 100 reduced it by 0.256.
5.3 Stages 1–3: random split, device change, subject change: 0.9974 to 0.9544 to 0.6942: the historic-mine attainable optimum declined across random split, device change, and subject change; the uranium mine replicated the pattern at 0.9993, 0.8258, and 0.6111.These comparisons use attainable optima and per-stage distributions because the configuration distributions are skewed and contain divergent runs.
5.4 Representation findings: 0.0001: no historic-mine configuration exceeded this subject-change macro precision for the scalar total-field representation FA; the uranium-mine best reached 0.0167.FA also failed under device change, despite passing the random split when combined with attitude channels.
5.4 Representation findings: +0.010 to −0.202 to −0.174: in the uranium mine, adding attitude channels to FV helped under random split but hurt under device and subject change.Across representations, attitude-augmented sets ranked highly under random split, whereas attitude-free bases led under subject change; the stage-ranking tests could not establish strong agreement.

6 Stage 4: cross-time evaluation after 34 months

After 34 months, unchanged models were evaluated on an unseen device and subject with only 42 of 95 trained classes still accessible. The analysis separates class loss, neighbour tolerance, residual change, out-of-scope predictions, and run-to-run uncertainty.

0.422 median present-conditioned macro precision for FV was 17.7 times chance, rising to 0.636 with one-neighbour tolerance.The medians use 63 repeated runs; chance is 1/42.
23.5 percentage points of FV's strict long-term gap arise from the 53 absent classes, while neighbour tolerance recovers 21.4 points.The class-loss factor is exactly 42/95; tolerance treats adjacent-segment confusions as correct.
2.1 percentage points is FV's residual gap under neighbour-tolerant scoring versus 23.4 points under strict present-conditioned scoring, making the upper bound defensible.The lower bound relies on optimistic assumptions about annotation-boundary ambiguity and compares tolerant with strict scores.
FK+L combines 0.355 present-conditioned precision with a 0.415 OOPS rate, whereas FV+L reaches 0.340 precision with 0.095–0.184 OOPS rates among other learned sets.Its sample accuracy is 0.227 versus 0.312 for FV+L, showing why present-conditioned precision requires OOPS reporting.
FK is bimodal across 77 runs, spanning 0.001–0.434 with an IQR of 0.379, while stable configurations have present-conditioned IQRs of 0.012–0.015.A fraction of FK runs collapses to chance during training.
Misclassified windows include long-range confusions, although neighbour-tolerant recovery occurs for windows in adjacent segments.The error-distance distribution is based on repeat ensembles of two stage-4 candidate configurations.

7 Uncertainty of the deployed model: repeated trainings and verdicts

Repeated-training ensembles quantify deployment risk against class-count-matched chance references, while 5% quantiles expose failures that mean-based tests miss. Across deployment stages, stable configurations remain above chance, but rare modes and shared data splits limit interpretation.

7.1 One-shot risk assessment: The three-step assessment tests mean performance, the evidenced share of above-chance runs, and the conservative 5% quantile against 1/K chance.The quantile is reported as a factor over chance and is intended to represent conservative single-deployment performance.
7.1 One-shot risk assessment: 14.2 to 17.5 times chance: after 34 months, stable configurations retain this present-conditioned 5% quantile range.The corresponding structural-split factors are 62 to 90; uranium-mine factors are 24.6 to 25.6.
7.1 One-shot risk assessment: Mean-based testing accepts C5 despite a 5% quantile of 0.03 times chance, because 34 of 77 runs collapse to 3.24 × 10−4 while 43 runs perform much better.Its mean is 0.325 with p = 9.5 × 10−14, but the sign test gives p = 0.18 and the 99% lower success-share bound is 0.406.
7.1 One-shot risk assessment: The 5% quantile can miss rare failures: C2 has an 8-of-282 minority mode below the 5% threshold, yielding q0.05 = 0.932 despite precisions of 0.687 to 0.709.The minority mode accounts for 2.8% of runs, below the quantile level.
7.1 One-shot risk assessment: The assessment quantifies training stochasticity but makes no population inference because all runs share one data split.The protocol's compute-efficient adoption variant uses one candidate configuration, repeated cross-device training, and cross-time re-evaluation without new training.
7.2 Retrospective verdicts: The retrospective criteria pass all deployment stages at k = 5 and k = 10, with the hardest class-exact cross-time stage exceeding the k=10 threshold by 1.6.For that stage, q0.05 = 0.392 against 10/42 = 0.238.

8 Discussion

The discussion shows that random-split evaluation can misrepresent deployment performance and ranking, while the protocol separates measurable degradation from unresolved uncertainty. Its evidence remains bounded by limited subjects, sites, walks, calibration tracking, and annotation conditions.

8.1 What the protocol measures and what it changes: +0.14 under device change and +0.41 under subject change: random-split optimism is measurable in matched configurations.The random-split representation ranking also does not predict deployment ranking.
8.1 What the protocol measures and what it changes: A 47-point apparent long-term loss leaves a defensible upper bound of 23 points for genuine signature change after accounting for class inventory and annotation boundaries.The decomposition avoids attributing the full observed loss to changing magnetic signatures.
8.1 What the protocol measures and what it changes: Models trained under harder structural splits age no worse, but a genuine advantage over random-split training cannot be evidenced.Across 112 matched triples, the margin is +0.004 with a confidence interval including zero, below the +0.007 schedule difference.
8.2 Transfer beyond magnetometry: The protocol generalizes its inputs and instruments beyond magnetometry to changing devices, personnel, recording epochs, class inventories, present-conditioning, and repeat-ensemble verdicts.These are presented as standard industrial deployment conditions.
8.3 Uncertainty budget: Uncertainty remains unquantified for walk-level sampling with n = 2, inter-campaign sensor calibration, annotation transfer and marker conventions, and secular or diurnal field variation.The residual signature-change component cannot be separated from secular and diurnal geomagnetic variation here.
8.4 Limitations: Subject effects are demonstrated but not sampled as a population because the cross-subject axis uses two persons per mine.Further magnetic distortion regimes remain untested, and device and person are coupled in the uranium mine.

9 Conclusion

The conclusion recommends staged evaluation that reports random splits only as upper bounds, bases roll-out on repeat-training quantiles, and conditions long-term scores on present classes. The study also decomposes degradation and selects models under deployment-relevant shifts rather than trusting random-split rankings.

9 Conclusion: Random splits overstated deployment performance by up to +0.41 and failed to predict deployment ranking, so they should be reported only as declared upper bounds.The recommendation explicitly includes reporting their window overlap.
9 Conclusion: A mean-based test accepted a configuration whose 5% quantile was more than an order of magnitude below chance, supporting quantile-based roll-out decisions.The conclusion recommends judging repeat ensembles against chance references with the correct class count.
9 Conclusion: Present-conditioned long-term scores should be guarded by OOPS rates because one representation family combined competitive scores with three to four times the alternatives' misdirection rate.The OOPS guard addresses predictions toward classes absent from the present deployment scope.
9 Conclusion: Of the apparent 47-point degradation, class loss explained 23 points and a defensible upper bound of 23 points remained for genuine signature change.The conclusion cautions against attributing total long-term loss to the underlying phenomenon.
9 Conclusion: Models selected under harder deployment-relevant regimes aged no faster over 34 months, while the data rule out strong agreement between random-split and deployment rankings.The conclusion does not claim a demonstrated advantage for harder-split training.
9 Conclusion: The full study cost 3237 GPU-hours, whereas adopting the protocol for a new system costs a fraction of a percent of that.The proposed deliverable is the staged report, providing evidence rather than a single flattering number.

Data availability

The study makes its supporting training-log data and analysis scripts available from the corresponding author upon reasonable request, with re-measurement confusion matrices also available.

Data availability: Training-log data and analysis scripts reproducing every figure and table are available from the corresponding author upon reasonable request.Per-run confusion matrices are available for the re-measurement, whose reported metrics were recomputed from them.