PaperlayerView exact revision on arXiv

Source-linked AI summary

Metacognitive Selective Ensemble for Mobile Systems

Sungmin Lee, Kichang Lee, Joonhee Lee, JaeYeon Park, Songkuk Kim, JeongGil Ko

arXiv:2609.31031v1cs.LG

TL;DR

Conventional ensemble aggregation may leave useful model diversity unrealized, while obtaining evidence from inactive members can require additional execution. MetaSE maintains a small active set, rejecting unreliable members using post-execution evidence and routing to replacements when needed. Across HAR datasets and architectures it improves over a fixed three-model ensemble; on a Raspberry Pi 4B it is 2.7× faster and uses 69% less memory than full ten-model inference.

  • Problem

    Conventional aggregation may not realize the diversity of ensemble members because it cannot know which member is correct and correlated failures can make additional votes redundant.

    Source: §1 · p. 1

  • Method

    MetaSE maintains a small active set across windows, using post-execution evidence to reject unreliable members and lightweight routing to select replacements when needed.

    Source: §1 · pp. 1–2

  • Results

    Across four HAR datasets and four architectures, MetaSE improves over a fixed three-model ensemble across most configurations; on a Raspberry Pi 4B, it is 2.7× faster and uses 69% less memory than full ten-model inference.

    Source: §1 · p. 2, §7 · p. 13

  • Takeaways & Limitations

    MetaSE exploits diversity from a larger pool while executing only three models during regular inference.

    Source: §1 · p. 2, §7 · p. 13

    Reliability signals learned before deployment may become less informative under distribution shifts, and relying on historical information for inactive candidates may miss the best replacement for a given window.

    Source: §6 · pp. 12–13

Abstract

from arXiv · show

Deep ensembles improve robustness in mobile sensing, but repeatedly executing many models over continuous sensor streams is costly. Selecting only a few members reduces this cost, yet adaptive selection often requires additional model execution to obtain reliable evidence about inactive candidates. We present MetaSE, an active ensemble framework that exploits short-term persistence in per-model reliability. MetaSE maintains a small active set across windows, uses post-execution evidence to reject unreliable members, and invokes lightweight routing only when replacement is needed. This stateful design accesses the diversity of a larger pool without repeated full-pool evaluation. Across four HAR datasets and four model architectures, MetaSE consistently improves over a fixed three-model ensemble and achieves accuracy comparable to substantially more expensive adaptive and full-ensemble inference. On a Raspberry Pi 4B, MetaSE is 2.7x faster and uses 69% less memory than full ten-model inference.

1 Introduction

Mobile sensing benefits from diverse models, but conventional aggregation may not realize their full potential, while full-pool and adaptive inference can impose substantial runtime costs. MetaSE maintains a small active set, using evidence from executed members to reject unreliable ones and routing to replacements only when needed.

1 Introduction

  • Oracle coverage can exceed conventional aggregation because voting or averaging cannot know which member is correct, and correlated errors can make additional votes redundant.The gap between ensemble prediction and oracle coverage represents diversity conventional aggregation does not fully exploit.Source: §1 · p. 1
  • Executing more members raises mobile inference computation, energy use, or latency, while adaptive selection may require extra inference and frequent replacement.These costs accumulate in continuous sensing, motivating a small active set drawn from a larger pool.Source: §1 · p. 1
  • MetaSE uses post-execution evidence from active members to retain or reject them, then uses lightweight pre-execution information to select replacements while otherwise carrying the active set forward.Its reject-then-route procedure preserves unaffected members and invokes routing only when replacement is needed.Source: §1 · pp. 1–2
  • MetaSE combines per-member rejection using predictions, features, other active members, and short-term stability with class-conditional replacement routing.The framework is evaluated across mobile HAR datasets and deployment conditions against static, full-ensemble, and adaptive-selection baselines.Source: §1 · p. 2
  • On a Raspberry Pi 4B, MetaSE is 2.7× faster and uses 69% less memory than full ten-model inference, while achieving accuracy comparable to full ten-model inference.Across four HAR datasets and four architectures, it improves over a fixed three-model ensemble across most configurations and outperforms representative dynamic selection methods under the same three-model budget.Source: §1 · p. 2

2 Background and Related Work

Prior work addresses ensemble diversity and inference efficiency through ensemble selection, dynamic competence estimation, and adaptive computation. MetaSE instead manages which full-task ensemble members remain active over a continuous sensing stream.

2.1 Deep Ensembles and Ensemble Selection

  • Deep ensembles combine independently trained models at inference, while static pruning fixes a subset before deployment and cannot respond when member usefulness changes.Dynamic selection chooses classifiers or subsets per query according to estimated competence.Source: §2.1 · pp. 2–3, §2.2 · p. 3

2.2 Dynamic Classifier and Ensemble Selection

  • Dynamic selection estimates input-dependent competence, while confidence-based selection uses current outputs that are available only after a classifier executes.MetaSE uses output-side evidence only for active members and routes replacements only when needed, rather than obtaining current outputs from every candidate.Source: §2.2 · p. 3

2.3 Efficient Ensemble Inference on Resource-Constrained Devices

  • Adaptive computation and ensemble methods often vary inference effort per input, whereas MetaSE changes which independently trained members occupy a small active set.Unlike HAR methods that change the training architecture or retain one network, MetaSE assumes an available model pool; its target also includes deployment-time replacement and transfer costs.Source: §2.3 · p. 3

3 Motivating Analysis

The motivating analysis examines whether a larger model pool offers useful alternatives, whether reliability persists across windows, and whether active members can be replaced without executing candidates. Its findings motivate a small, persistent active set with evidence-based rejection and replacement routing.

3.1 Opportunity for Selective Ensemble Execution

  • Soft-voting accuracy saturates as more models are added, while oracle coverage continues to increase, indicating that a larger pool provides useful alternatives for a K-member ensemble.The motivating study uses ten Transformer models trained with the same configuration but different random initializations on HHAR.Source: §3.1 · p. 4, Figure 1 · p. 4

3.2 Short-Term Persistence Supports Active-Set Maintenance

  • Within the same activity, conditional next-window error is substantially higher than ordinary error across all six activities, but this persistence weakens across activity transitions.This supports preserving the active set across nearby windows and reconsidering members when evidence indicates unreliability, rather than reselecting at every window.Source: Figure 3 · p. 4, §3.2 · p. 5

3.3 Updating the Active Set Without Candidate Execution

  • Runtime signals from executed members provide complementary, imperfect evidence for rejection, while the best replacement varies with the rejected member and its predicted class.A precomputed class-conditional ranking can select replacements without candidate execution; this asymmetry motivates stateful reject-then-route updates that preserve unaffected active members.Source: §3.3 · pp. 5–6, Figure 5 · p. 5

4 Design of MetaSE

MetaSE keeps a small active ensemble across sensing windows, using evidence from executed members to decide whether they remain useful and lightweight routing to replace rejected members. This avoids repeated full-pool selection and candidate inference.

4 Design of MetaSE

  • MetaSE preserves a K-member active set by default and uses reject-then-route updates only when its composition needs to change.The design targets unnecessary replacement and extra candidate-model inference.Source: §4 · p. 6, §4.1 · p. 6

4.1 Overall Framework

  • At each window, active models produce the prediction and evidence for whether they should remain; a routed replacement joins from the following window.Accepted members use margin-weighted soft voting, and only a rejected position is filled.Source: §4.1 · p. 6

4.2 Maintaining the Active Set Across Windows

  • If active members remain accepted, MetaSE carries them forward; rejection excludes a member from the current vote and replaces its position for the next window.If all K members are rejected, it falls back to voting over the current set and leaves that set unchanged.Source: §4.2 · p. 6

4.3 Post-Execution Member Rejection

  • Each member’s rejecter combines output confidence, representation distance, peer agreement or dissent, and five-window prediction stability to assess reliability.Separate rejecters are trained as correctness detectors; deployed checkpoints are selected using validation system accuracy over sampled K-member ensembles.Source: §4.3 · p. 7

4.4 Forward-Free Replacement Routing

  • For a rejected member and predicted class, MetaSE ranks inactive candidates using offline class-conditional complementarity scores, without executing them.When several members are rejected, candidates are ranked across the vacated positions and replacements are made together.Source: §4.4 · p. 8

5 Evaluation

Across four HAR datasets and four model architectures, MetaSE adapts a three-member active set and generally improves on fixed or adaptive alternatives while approaching full-ensemble accuracy. Its Raspberry Pi evaluation also reports lower latency and memory than ten-model inference.

5.1 Evaluation Setup

  • Evaluation covers four subject-disjoint HAR datasets, four architectures, and pools of ten independently trained same-architecture models, with K=3 active by default.Comparisons include fixed and full-pool ensembles, adaptive-selection baselines, and HAR-specific ensemble methods.Source: §5.1 · pp. 8–9, §5 · p. 8

5.2 Overall Performance

  • MetaSE beats Static K=3 in 14 of 16 dataset-architecture configurations and is more accurate than full N=10 soft voting in 11 of 16.In the remaining full-voting comparisons, its largest deficit is 0.8 percentage points.Source: Table 1 · p. 9, §5.2 · pp. 9–10
  • At the same three-forward budget, MetaSE beats KNORA-U in 15 of 16 configurations and stays within 0.1 percentage points of full ten-model voting on HHAR and UCIHAR.Averaged across architectures, it also exceeds full-pool KNORA-U and META-DES on all four datasets.Source: Figure 7 · p. 9, §5.2 · p. 10

5.3 Selection Gain across Pool Correctness

  • On HHAR and MOTIONSENSE Transformer pools, n_c=5–7 windows account for 79% and 59% of gains over Static K=3, respectively.The analysis finds limited contribution from very difficult inputs and from cases where most pool members are already correct.Source: Figure 9 · p. 10, §5.3 · p. 11

5.4 Rejecter Behavior and Error Cost

  • Correcting missed rejections raises end-to-end accuracy by 2.84 percentage points on average across 16 configurations, while correcting false rejections changes it by 0.10 points.The oracle-label replay changes only one rejecter error type and leaves the rest of the streaming pipeline unchanged.Source: Figure 10 · p. 11, §5.4 · p. 11

5.5 Active-Set Update Dynamics

  • On HHAR/Transformer, unchanged intervals have a median of 3 windows and mean of 9.2; 57% of replacement bursts end after one window.A further 84% of bursts end within two windows.Source: Figure 11 · p. 12, §5.5 · p. 12

5.6 On-Device Efficiency

  • On Raspberry Pi 4B, MetaSE is 2.7× faster and uses 69% less memory than full ten-model inference.MetaSE takes 25.5 ms and 625 KB, versus 68.5 ms and 2,046 KB for ten models.Source: Figure 12 · p. 12, §5.6 · p. 12

6 Discussion

MetaSE’s reliability signals may become less informative under distribution shifts, and inactive candidates are assessed using historical rather than current-input evidence.

6 Discussion

  • Distribution shifts may weaken predeployment reliability signals, while missed rejections remain a major source of error.Improving robustness to changing conditions without pool-wide inference is identified as future work.Source: §6 · p. 12
  • Because inactive candidates are not executed, historical information may miss the best replacement for a given window.Selective probing is proposed when its expected benefit outweighs its additional cost.Source: §6 · p. 13

7 Conclusion

MetaSE maintains a small model subset for continuous mobile sensing and achieves comparable or better accuracy than more expensive selection approaches while executing three models during regular inference.

7 Conclusion

  • On a Raspberry Pi 4B, MetaSE is 2.7× faster and uses 69% less memory than full ten-model inference.It achieves accuracy comparable to or better than more expensive selection approaches while executing only three models during regular inference.Source: §7 · p. 13

A Example Streaming Behavior

A 500-window HHAR/Transformer trace illustrates that MetaSE updates its active set according to observed member reliability, with localized changes and intervals without updates.

A Example Streaming Behavior

  • The 500-window HHAR/Transformer trace is presented as a qualitative view of runtime behavior.It complements aggregate update statistics.Source: A Example Streaming Behavior · p. 16
  • Updates can occur within an activity segment, and selected members can persist across different portions of the stream as reliability changes.The trace does not show a fixed mapping from activity labels to model subsets.Source: A Example Streaming Behavior · p. 16
  • When one member is rejected, unaffected members continue participating rather than triggering complete active-set reselection.This allows previous selection decisions to be reused when still valid.Source: A Example Streaming Behavior · p. 16
Loading 2609.31031v1…