PaperlayerView exact revision on arXiv

Source-linked AI summary

Selective Next-Location Prediction with MetaModelSelect: A Lightweight Post-Hoc Confidence Metamodel

Calvin Jary, Nafiseh Kahani

arXiv:2610.19394v1cs.LGcs.NI

TL;DR

Unreliable next-location predictions can misdirect mobile-network resources, while average accuracy does not identify reliable individual predictions. MetaModelSelect learns top-1 correctness from a frozen RNN’s outputs; on path-balanced Changchun trajectories, it improves confidence rankings relative to maximum softmax probability and achieves the lowest reported mean risk at evaluated partial coverages.

  • Problem

    Average accuracy does not show which individual next-location predictions are reliable enough to use, although wrong predictions can direct resources toward the wrong region.

    Source: §I · p. 1

  • Method

    MetaModelSelect learns top-1 correctness from a frozen RNN using a predicted-location embedding, four leading softmax probabilities, and Softmax Concentration.

    Source: §I · p. 1

  • Results

    Relative to maximum softmax probability, mean per-run AURC, E-AURC, and AUGRC reductions are 2.6%, 4.2%, and 3.0% for CE and 2.0%, 3.3%, and 1.9% for SAT+EM.

    Source: §I · p. 2

  • Takeaways & Limitations

    MetaModelSelect improves confidence ranking for frozen cellular predictors while leaving the predicted location unchanged.

    Source: C. Conclusion · p. 8

    The evaluation covers one city, one week, and hourly clustered regions, so it does not establish performance on future periods or new cities.

    Source: B. Scope and limitations · p. 8

Abstract

from arXiv · show

Next-location prediction can support proactive mobile-network decisions, but acting on unreliable predictions can waste resources. We apply MetaModelSelect, a lightweight post-hoc confidence metamodel, to selective cellular next-location prediction. The mobility-specific model combines a learned embedding of the predicted location with the four largest softmax probabilities and Softmax Concentration. It learns whether a frozen recurrent neural network's top-1 prediction is correct, using only its outputs and without retraining the backbone. On path-balanced Changchun cellular mobility trajectories, we evaluate backbones trained with cross-entropy (CE) and self-adaptive training plus entropy minimization (SAT+EM), comparing against eight scalar confidence scores, two location-aware lookup methods, and a raw-logit multilayer perceptron. Relative to maximum softmax probability, MetaModelSelect achieves mean per-run reductions in area under the risk--coverage curve (AURC), excess AURC (E-AURC), and area under the generalized risk--coverage curve (AUGRC) of 2.6%, 4.2%, and 3.0% for CE, and 2.0%, 3.3%, and 1.9% for SAT+EM. It also achieves the lowest reported mean risk at every evaluated partial coverage. The results show that MetaModelSelect improves confidence ranking in this mobility setting while leaving the underlying location predictor unchanged.

I. INTRODUCTION

The introduction frames selective next-location prediction as ranking predictions by confidence so a network can abstain when evidence is weak. It adapts output-only correctness modeling to frozen cellular RNNs and reports improved confidence rankings on the evaluated dataset.

I. INTRODUCTION

  • Unreliable next-location predictions can direct resources toward the wrong region, while average accuracy does not identify which individual predictions are reliable enough to use.Potential uses include proactive caching, resource allocation, and handoff preparation.Source: §I · p. 1
  • The study evaluates a post-hoc selector on frozen RNNs using logits and predicted locations, for next-hour predictions of clustered cellular regions.The testbed uses public Changchun cellular traces and a compact RNN.Source: §I · p. 1
  • The selector combines a learned embedding of the predicted location with the four largest softmax probabilities and Softmax Concentration to learn top-1 correctness.Its score controls acceptance without replacing the predicted location.Source: §I · p. 1
  • The evaluation compares CE and SAT+EM backbones against eight scalar confidence scores, two location-aware lookup methods, and a raw-logit MLP.The study uses risk–coverage evaluation on path-balanced cellular trajectories.Source: §I · p. 1
  • MetaModelSelect has the lowest reported mean integrated metrics and partial-coverage risks across both backbone objectives, with integrated-metric reductions versus maximum softmax probability.For CE, AURC, E-AURC, and AUGRC reductions are 2.6%, 4.2%, and 3.0%; for SAT+EM, they are 2.0%, 3.3%, and 1.9%.Source: §I · pp. 1–2

A. Mobility prediction and confidence

Prior mobility work considered confidence counters and place-conditioned trust, while selective prediction formalizes acceptance through confidence-ranked coverage and risk. This study tests learned output-only correctness estimation for frozen cellular predictors.

A. Mobility prediction and confidence

  • Earlier mobility methods used confidence counters and location-aware trust estimates, motivating location-conditioned confidence for a frozen deep cellular predictor.The study evaluates learned confidence with risk–coverage metrics and location-aware lookup controls.Source: A. Mobility prediction and confidence · p. 2

B. Selective classification and scalar scores

  • Selective prediction trades coverage—the fraction accepted—against selective risk, the error rate among accepted predictions,.For a fixed classifier under 0/1 loss, ranking by conditional correctness yields the ideal selective ordering.Source: B. Selective classification and scalar… · p. 2
  • Scalar post-hoc scores are inexpensive but prescribe how outputs map to confidence, whereas learned selectors can combine output characteristics and condition their interpretation on the predicted class.The compared score families include MSP, negative entropy, energy confidence, logit margin, concentration statistics, and centered p-norm-normalized logits.Source: B. Selective classification and scalar… · p. 2

C. Learned correctness estimation

  • MetaModelSelect adapts output-derived top-1 correctness modeling with a predicted-class embedding, combining location identity, the top four softmax probabilities, and Softmax Concentration.The study evaluates the feature choices and their measured effects in cellular mobility.Source: C. Learned correctness estimation · p. 2
  • The study tests output-only correctness modeling with both CE and SAT+EM backbones, but does not establish superiority over internal-feature heads, ensembles, or other access regimes.SAT+EM modifies the backbone objective.Source: C. Learned correctness estimation · p. 2

A. Prediction and abstention

  • A frozen RNN maps five hourly location observations to a next-region prediction, and the selector scores that existing decision without changing its predicted location.The model outputs logits over 2,134 location classes, which are converted to probabilities.Source: A. Prediction and abstention · p. 2
  • A confidence threshold selects accepted predictions, and a useful selector ranks correct predictions ahead of errors so risk can fall as coverage decreases.Selective risk is defined for nonempty accepted sets.Source: A. Prediction and abstention · p. 2

B. Output-derived representation

The output-derived representation combines predicted-location identity with leading softmax alternatives and a statistic of the full output distribution. The input has five numerical features in addition to its learned location embedding.

B. Output-derived representation

  • The selector input concatenates the predicted-location embedding with the four largest sorted softmax probabilities and Softmax Concentration.The embedding conditions the score on the predicted region; the top four probabilities describe the leading decision and strongest alternatives, while concentration summarizes the full output.Source: B. Output-derived representation · p. 2
  • Softmax Concentration is monotone-equivalent to negative Gini impurity and is distinct from the Lorenz-Gini coefficient used as a baseline.Gini impurity is 1 − sum_k p_k^2.Source: B. Output-derived representation · p. 2
  • The numerical features are standardized using training-derived means and population standard deviations, and the selected representation is fixed before test evaluation.Test comparisons and ablations assess the selected representation rather than selecting features on the test set.Source: B. Output-derived representation · p. 2
  • The total input dimension is de + 5: 48 for CE and 29 for SAT+EM.The five numerical features are appended to the learned embedding.Source: B. Output-derived representation · p. 2, Table I · p. 3

C. Correctness training and inference

MetaModelSelect learns a correctness score from a frozen backbone’s output-derived features, with correctness labels used for training but not as inference inputs. The evaluation uses held-out paths while selector fitting observes in-sample backbone behavior.

C. Correctness training and inference

  • The two-layer metamodel is trained with unweighted binary cross-entropy against whether the RNN’s top-1 prediction is correct, while the RNN remains frozen.At inference, a sigmoid-transformed scalar output supplies the correctness score; this does not establish calibration or a guaranteed risk bound.Source: C. Correctness training and inference · p. 3
  • Training uses the true next location to determine correctness, but inference uses only the predicted class and output-derived features to score acceptance or abstention.The inference path requires no true label, hidden RNN state, ensemble, or repeated stochastic prediction; the fallback after abstention is application-specific and unevaluated.Source: C. Correctness training and inference · p. 3

A. Data and evaluation protocol

  • The data comprise one week of hourly Changchun observations, reduced to 197,415 unique complete daily paths and split 80% for training and 20% for testing.Each distinct complete path contributes once regardless of how many users followed it.Source: A. Data and evaluation protocol · p. 3
  • The base RNN follows a compact two-layer vanilla architecture, and the study evaluates CE and SAT+EM objectives.The architecture and training settings are summarized in Table II.Source: Table II · p. 3
  • Selector fitting and validation use outputs from the same 80% partition used to train each RNN, while the test partition is excluded from model selection.Thus the protocol tests held-out paths but selector training retains the limitation of observing in-sample backbone behavior.Source: A. Data and evaluation protocol · p. 3

B. Backbones and selector settings

The study uses compact CE and SAT+EM recurrent backbones and fits a post-hoc selector to frozen-model outputs. Comparison methods include scalar confidence scores, location-conditioned lookups, and a learned raw-logit selector.

B. Backbones and selector settings

  • The compact vanilla RNN is trained with CE or SAT+EM, with post-hoc scoring normalized over the K location classes.SAT+EM adds an abstention output during backbone training.Source: B. Backbones and selector settings · p. 3
  • Embedding dimension and batch size are selected separately for each objective by training-only five-fold validation, while the head architecture, optimizer, and duration are fixed.The selected dimensions are de = 43 for CE and de = 24 for SAT+EM.Source: B. Backbones and selector settings · p. 3, Table III · p. 4

C. Comparison methods

  • The eight scalar confidence scores are oriented so larger values indicate greater confidence; energy temperature and centered pNorm order are tuned within the training partition.Both objectives select p = 9.Source: C. Comparison methods · p. 3, Table IV · p. 4
  • Location-Reliability uses a Laplace-smoothed lookup by predicted location, whereas Location-Calibrated MSP combines that location with one of 20 equal-width MSP bins.Sparse cells shrink toward class-level reliability, and empty cells fall back to that value.Source: C. Comparison methods · pp. 3–4
  • The selector uses a predicted-location embedding and five numerical features, while its two-layer head learns correctness directly from class logits in the RawLogit-MLP comparison.The embedding has K+1 rows, with one reserved for a missing-entry token; input dimensions are 48 for CE and 29 for SAT+EM.Source: B. Backbones and selector settings · p. 3, Figure 1 · p. 4, C. Comparison methods · pp. 4–5

D. Metrics and implementation

The evaluation ranks predictions at specified coverages and reports integrated risk metrics, with implementation costs measured incrementally to the backbone pass.

D. Metrics and implementation

  • At each target coverage q, the highest-scored ⌈qN⌉ predictions are accepted, so the evaluation compares rankings rather than deployable thresholds.A validation-chosen threshold may attain different coverage after distribution shift.Source: D. Metrics and implementation · p. 5
  • AURC integrates selective risk over coverage, E-AURC subtracts the same classifier’s oracle AURC, and AUGRC integrates accepted-error fraction; lower is better.Risk is reported at 100%, 95%, 90%, 80%, 70%, and 50% coverage.Source: D. Metrics and implementation · p. 5
  • Reported means and standard deviations cover three backbone runs, and selector latency includes feature computation but excludes the RNN forward pass.Timing uses an AMD Ryzen 7950X3D CPU and NVIDIA RTX 4090 GPU; standard deviations describe run-to-run variation, not confidence intervals.Source: D. Metrics and implementation · p. 5

V. RESULTS AND ANALYSIS

MSP and Softmax Concentration are the strongest overall scalar rankings, but their effectiveness varies by objective and some other scores perform substantially worse.

A. Scalar confidence scores

  • MSP and Softmax Concentration tie for the strongest scalar rankings, with CE AURC 0.0421 and AUGRC 0.0337, and SAT+EM AURC approximately 0.0387 and AUGRC approximately 0.0311.The values are reported at the table’s precision; full-coverage risk is unchanged by selector reordering: 17.46% for CE and 16.61% for SAT+EM.Source: A. Scalar confidence scores · p. 5
  • pNorm has AURC 0.0753 under CE and 0.0481 under SAT+EM despite training-side tuning, while energy confidence and maximum logit are particularly weak under CE.These results caution that confidence-score performance in other domains need not transfer to cellular predictors.Source: A. Scalar confidence scores · p. 5

B. Main selector comparison

MetaModelSelect ranks predictions better than the tested alternatives for both backbone objectives, while feature ablations identify the predicted-location embedding and largest softmax probability as its largest measured contributors.

C. Feature behavior and ablations

  • Among correct CE predictions, classwise mean softmax confidence varies substantially, motivating location-conditioned confidence without directly estimating each location’s error rate.The figures also show that Softmax Concentration complements Top-1 Softmax and that a larger second probability is associated with more incorrect predictions; the scatter plots alone do not establish conditional error probabilities or incremental feature value.Source: Figure 3 · p. 5, C. Feature behavior and ablations · pp. 5–6

B. Main selector comparison

  • At 90% coverage, risk falls from 12.72% to 12.42% for CE and from 11.89% to 11.73% for SAT+EM with MetaModelSelect versus MSP.At 50% coverage, the corresponding risks fall from 1.71% to 1.54% and from 1.51% to 1.40%; these comparisons hold the accepted fraction fixed.Source: B. Main selector comparison · p. 5
  • Location-Reliability underperforms, Location-Calibrated MSP worsens AURC and E-AURC versus MSP, and RawLogit-MLP underperforms the strongest scalar scores on integrated metrics.Location-Calibrated MSP improves several fixed-coverage risks, but that advantage does not extend uniformly across the full ranking.Source: B. Main selector comparison · p. 5
  • MetaModelSelect has the lowest mean AURC, E-AURC, AUGRC, and risk at every listed partial coverage for both objectives.Table VI reports AURC reductions from 0.04207 to 0.04099 for CE and from 0.03870 to 0.03792 for SAT+EM, relative to MSP.Source: B. Main selector comparison · p. 5, Table VI · p. 6
  • 2.55% for CE and 2.03% for SAT+EM are the reported AURC reductions from replacing MSP with MetaModelSelect on the corresponding frozen backbone.Table VII distinguishes these selector changes from the 8.00% AURC reduction associated with switching from CE to SAT+EM; the percentages use different reference configurations and should not be added.Source: B. Main selector comparison · p. 5, Table VII · p. 6

C. Feature behavior and ablations

  • Removing the location embedding causes the largest CE-backbone ablation deterioration: AURC rises from 0.04099 to 0.04197, E-AURC from 0.02478 to 0.02576, and AUGRC from 0.03267 to 0.03362.Removing the largest probability causes the next-largest degradation; removing each other tested probability or Softmax Concentration also increases every mean integrated metric.Source: C. Feature behavior and ablations · p. 6

D. Computational overhead

The metamodel adds a compact parameter count and sub-millisecond incremental latency in the measured setup, though its latency is substantial relative to the small backbone.

D. Computational overhead

  • For K = 2,134, the metamodel has 142,806 parameters with CE (de = 43) and 98,441 with SAT+EM (de = 24).At fixed embedding dimension, each additional location adds de parameters while the numerical feature block and hidden-layer widths stay constant.Source: D. Computational overhead · p. 7
  • The de = 24 metamodel uses approximately 4.9% as many parameters as the RNN and 79% fewer than RawLogit-MLP.Despite having more parameters, RawLogit-MLP is faster in this setup, so parameter count alone does not determine runtime.Source: D. Computational overhead · p. 7
  • With de = 24, measured incremental latency is 0.131 ms per example on CPU and 0.465 ms on GPU, including feature construction.These are selector times incremental to the RNN pass; they are approximately 29% and 80% of the respective backbone times.Source: D. Computational overhead · p. 7

VI. DISCUSSION AND CONCLUSION

MetaModelSelect improves confidence ranking for an existing cellular predictor under both tested objectives. The gains are modest, and coverage choices depend on the costs of accepted errors and fallback.

VI. DISCUSSION AND CONCLUSION

  • MetaModelSelect achieves the lowest reported mean integrated metrics and risk at every evaluated partial coverage under both CE and SAT+EM.The SAT+EM-selected configuration has 98,441 parameters and adds less than one millisecond in the measured setup.Source: A. What the results establish · p. 7
  • The metamodel conditions confidence on predicted location and output features, while location-only and binned location/MSP controls do not match its tested ranking performance.RawLogit-MLP also performs worse despite its larger parameter count; these comparisons support the representation within the tested configurations.Source: A. What the results establish · p. 7
  • At 80% coverage, CE risk falls from 8.76% with MSP to 8.42% with the metamodel, while SAT+EM changes backbone predictions and the metamodel further improves their ordering.Results summarize three backbone runs and are not a formal significance test; operational coverage depends on the cost of an incorrect accepted prediction and invoking the fallback.Source: A. What the results establish · p. 8

B. Scope and limitations

The study adapts output-only correctness modeling to selective cellular next-location prediction and reports improved confidence rankings for frozen CE and SAT+EM RNNs. Its conclusions remain bounded by the evaluated data, selector-training protocol, and lack of downstream network-control validation.

B. Scope and limitations

  • The metamodel adds approximately 29% of base CPU time and 80% of GPU time; no network-control experiment tests whether risk reductions offset this cost.Source: B. Scope and limitations · p. 8
  • The dataset covers one city, one week, and hourly clustered regions, so results describe path-balanced rather than user-volume-weighted network impact.Source: B. Scope and limitations · p. 8
  • Selector training and hyperparameter selection reuse backbone-training outputs, while separate calibration data or out-of-fold predictions would provide a stricter protocol.Source: B. Scope and limitations · p. 8

C. Conclusion

  • A predicted-location embedding and five softmax-derived inputs improve frozen CE and SAT+EM RNN confidence rankings against evaluated scalar, lookup, and raw-logit selectors.Source: C. Conclusion · p. 8
  • The method is an economical abstention mechanism, but validation remains needed under temporal shifts, different cities, and application-specific costs.Source: C. Conclusion · p. 8

Related papers

Loading 2610.19394v1…