PaperlayerView exact revision on arXiv

Source-linked AI summary

HB-PVI: A Hierarchical Bayesian Personalization and Value-of-Information Framework for Complex Activity Recognition

Hammed A. Olayinka

arXiv:2609.05582v1cs.LG

TL;DR

Population models can generalize poorly to new users, while personalization gains vary and calibration labels cost time. HB-PVI models participant-level benefit and harm and uses value-of-information to decide whether to acquire labels; under primary conditions, it bought none while preserving mean F1 within tolerance.

  • Problem

    Population models can generalize poorly to new users, while participant-specific personalization can harm performance and each calibration label carries an acquisition cost.

    Source: §I · p. 1

  • Method

    HB-PVI combines a hierarchical participant-gain model for four personalization mechanisms with a one-step EVSI rule for label acquisition.

    Source: D. Personalization actions, gain, and… · p. 3, E. Utility, Bayes action, and EVSI · p. 3

  • Results

    Under primary conditions, HB-PVI purchased no labels and preserved mean F1 within tolerance; relative to fixed 10-shot adapter personalization, it reduced labels by 100% with posterior probability 0.9992 that mean F1 loss stayed below 0.005.

    Source: §I · p. 6, §V · p. 8

  • Takeaways & Limitations

    The population-first recommendation is tied to the assumed cost structure, and decision surfaces across costs are informative when costs are uncertain.

    Source: §IV · p. 7

    The 47-participant cohort limits power for rare subgroups and higher-order interactions, and the primary EVSI analysis considers only the next chronological window.

    Source: B. Limitations · p. 7

Abstract

from arXiv · show

Personalization can improve activity-recognition performance, but participant-specific gains are heterogeneous, and every additional calibration label has an acquisition cost. This study presents HB-PVI, a hierarchical Bayesian personalization and value-of-information framework jointly modeling participant heterogeneity, the benefit and harm of four personalization mechanisms, and the economic value of an additional label, for the 47-participant MUSIC-CAR complex-activity cohort. A leakage-safe, leave-one-participant-out evaluation combines a sequential-Monte-Carlo participant-effect updater with a Student-$t$ hierarchical gain model and a one-step expected-value-of-sample-information (EVSI) stopping rule. Adapter personalization produced small positive mean F1 gains, growing from 0.00099 at one label to 0.00198 at ten, while adapter-plus-head and prototype-residual personalization were negative on average. Under the primary practical-benefit threshold ($\Delta_{\min}=0.01$) and cost setting, one-step EVSI was zero at every decision state, so the policy purchased no labels and retained population inference for all 47 participants, matching always-stop exactly (region-of-practical-equivalence probability $=1$). Relative to fixed ten-shot adapter personalization, this reduced labeling by 100\% while keeping the posterior mean F1 loss at 0.00217 (95\% credible interval, 0.00048 to 0.00389), with posterior probability 0.9992 of remaining below the 0.005 tolerance. HB-PVI was utility-optimal in 199 of 216 cost-threshold settings and in every setting at or above the primary label cost. These results argue for a population-first deployment policy whenever personalization gains are small relative to labeling, computation, and harm costs, and show that value-of-information reasoning, not raw predictive accuracy, should drive personalization decisions in health-sensing applications.

I. INTRODUCTION

Complex activities vary across people, and personalization can help but also harm while each calibration label costs time. HB-PVI treats both the choice of personalization and whether to acquire another label as sequential decisions.

I. INTRODUCTION

  • Prior cohort work found fixed 6-shot adapters raised mean F1 from 88.0% to 92.6% on average, but performance slightly declined for six of 47 participants.The prior studies did not model participant-level uncertainty in the personalization decision or price the value of another calibration label.Source: §I · p. 1
  • Unlike few-shot and meta-learning approaches that optimize accuracy without pricing adaptation data, HB-PVI addresses whether participant-specific context is worth purchasing, unlike mobile-health methods that treat context as free,.Source: §I · pp. 1–2
  • HB-PVI models participant-specific benefit and harm for four personalization mechanisms and uses one-step EVSI to decide whether to personalize and purchase another label.The framework frames personalization as a sequential label-purchase decision rather than a one-shot model-selection problem.Source: §I · pp. 1–2

A. Cohort, design, and leakage control

  • The MUSIC-CAR evaluation used leave-one-participant-out logic, excluding each held-out participant from representation training, normalization, and hyperparameter selection.Windows were prospectively ordered into adaptation, guard-gap, calibration, and test blocks; only the first n adaptation labels informed a decision at shot n.Source: A. Cohort, design, and leakage control · p. 2

B. Deterministic representation and cross-fitting

  • One deterministic ensemble member supplied shared pooled representations, while other members supplied uncertainty summaries; downstream Bayesian quantities were inferred with HMC/NUTS.This kept the representation coordinate system fixed across analyses and avoided treating the ensemble as an approximate posterior.Source: B. Deterministic representation and… · p. 2

C. Model I: hierarchical multi-label prediction and sequential updating

The models connect leakage-safe representations and sequential participant updates to a hierarchical estimate of gains across personalization actions. They represent both shared population structure and participant-specific uncertainty.

C. Model I: hierarchical multi-label prediction and sequential updating

  • Activity labels are modeled conditionally on representations with activity-specific coefficients and correlated participant intercepts, using regularized horseshoe shrinkage for high-dimensional coefficients.The participant intercepts are correlated across activities, while the primary specification applies the horseshoe prior to the activity coefficients alone.Source: C. Model I: hierarchical multi-label… · p. 2
  • 64,000 joint particles update participant effects one purchased label at a time, with resampling and rejuvenation governed by effective sample size.All 47 folds passed the required population-ancestor and final-particle ESS validation gates.Source: C. Model I: hierarchical multi-label… · pp. 2–3

D. Personalization actions, gain, and Model II

  • The four actions are adapter, head, adapter-plus-head, and prototype-residual; gain is each action’s sample-level F1 minus the same participant’s population-only baseline.Adapter tunes a low-rank bottleneck, head tunes the final layer, adapter-plus-head tunes both, and prototype-residual adjusts predictions without gradient fine-tuning.Source: D. Personalization actions, gain, and… · p. 3
  • A Student-t hierarchical gain model uses action-specific coefficients, participant-by-action random intercepts, and 19 sequential-state predictors spanning label history, uncertainty, participant effects, and representation shift.The Student-t likelihood downweights outlying participant-fold gains relative to a Gaussian likelihood; all 47 folds passed the stated convergence gates.Source: D. Personalization actions, gain, and… · p. 3

E. Utility, Bayes action, and EVSI

HB-PVI selects actions by posterior expected utility subject to benefit and harm eligibility constraints, then compares the expected value of another label with its cost. Its evaluation also assesses comparator utility and label efficiency.

E. Utility, Bayes action, and EVSI

  • Realized utility subtracts per-label acquisition cost, action-specific computation cost, and a harm penalty when gain is negative.The Bayes action maximizes posterior expected utility subject to minimum-benefit and maximum-harm probability constraints.Source: E. Utility, Bayes action, and EVSI · p. 3
  • Acquisition continues only while the expected value of sample information from the next label exceeds its cost, evaluated along the fixed chronological window order.The analysis used 256 hypothetical next-label draws per state and deferred active, nonchronological window selection.Source: E. Utility, Bayes action, and EVSI · p. 3
  • The sensitivity analysis crossed three practical-benefit thresholds with 72 cost combinations, yielding 216 settings.Primary costs were cL = 0.001, cC = 0.0005, and cR = 0.01.Source: E. Utility, Bayes action, and EVSI · p. 3

F. Statistical analysis and study objectives

  • Utility comparisons used participant-level paired Student-t posteriors, while label efficiency required posterior probability above 0.90 that mean F1 loss stayed below 0.005.A participant bootstrap provided an independent sensitivity analysis of the label-efficiency result.Source: F. Statistical analysis and study… · p. 3

A. Action-specific personalization outcomes

Across the 47-participant cohort, adapter gains were small on average but varied widely by participant, and the held-out gain model did not discriminate personalization harm beyond base rates.

A. Action-specific personalization outcomes

  • Adapter mean F1 gain rose from +0.00099 at one label to +0.00198 at ten, while adapter-plus-head and prototype-residual means were negative at every shot.Head personalization was near zero on average; adapter-plus-head had the largest benefit subgroup (up to 21.3%) and harm subgroup (up to 31.9%).Source: A. Action-specific personalization… · pp. 3–4

B. Population baseline strength and the ceiling on personalization value

  • At ten labels, adapter gains ranged from −0.0265 to +0.0168 despite an average gain no higher than +0.00198.The population-only ensemble averaged F1 0.9042; when the population baseline captures most recoverable signal, a calibration label carries little marginal information about who will benefit.Source: B. Population baseline strength and… · p. 4

C. Held-out gain prediction and calibration

  • Prototype-residual gain was easiest to predict (MAE 0.00325, 95.3% coverage), while adapter-plus-head was hardest (MAE 0.01396, 88.9% coverage).Harm prediction was less accurate than an empirical-rate baseline for all four mechanisms; population-only F1 was also weakly related to ten-shot adapter F1 (r = −0.06).Source: C. Held-out gain prediction and… · p. 4, Figure 2 · p. 5

D. EVSI, stopping, and the primary deployment decision

HB-PVI was utility-optimal in 199 of 216 cost-threshold settings, including every setting at or above the primary label cost. Under the primary setting it bought no labels and retained population inference for all participants.

D. EVSI, stopping, and the primary deployment decision

  • At the primary setting, one-step EVSI was zero at every state, so HB-PVI bought zero labels and retained population inference for all 47 participants.At Δmin = 0.005 or 0.01, realized mean gain and cumulative utility were zero, with no participant exposed to personalization harm.Source: D. EVSI, stopping, and the primary… · p. 5, Figure 3 · p. 6

E. One-step EVSI ceiling across the sensitivity grid

  • Positive one-step EVSI occurred in only 72 of 10,152 decisions (0.7%), all involving the single participant selected for head personalization at Δmin = 0.At label costs cL ≤ 0.001, the policy continued through ten shots; at cL ≥ 0.0025, it bought exactly one label before stopping.Source: E. One-step EVSI ceiling across the… · p. 5

F. Utility comparisons, paired differences, and cost sensitivity

G. Label efficiency and F1 preservation

Compared with fixed 10-shot adapter personalization, the policy eliminated calibration labels while keeping mean F1 loss within the stated tolerance. Under the primary costs it selected population inference for all participants, although the model did not prospectively identify who might be harmed.

H. Harm avoidance without demonstrated harm identification

  • At the primary setting, the decision rule retained population inference for all 47 participants, purchasing no labels under the specified costs.This reflected the conservative decision rule, not demonstrated prospective identification of participants at risk of harm.Source: H. Harm avoidance without demonstrated… · p. 6, §IV · p. 7
  • The fitted gain model did not prospectively discriminate harmful personalization: its harm-specific Brier score was worse than a trivial empirical-rate baseline for all four mechanisms.No participant experienced realized personalization harm in this study because the primary-setting policy declined personalization.Source: H. Harm avoidance without demonstrated… · p. 6

I. Summary of study objectives

  • HB-PVI was practically equivalent to always-stop, but the study did not show that it was better.The summary attributes the population-first recommendation to participant-level heterogeneity.Source: §I · p. 6

IV. DISCUSSION

  • Across the 47-participant evaluation, personalization gains were heterogeneous but generally small, and adapter personalization had the most favorable risk-benefit profile of the four mechanisms tested.Adapter-plus-head amplified both benefit and harm, while prototype-residual was reliably unfavorable.Source: §IV · pp. 6–7

G. Label efficiency and F1 preservation

IV. DISCUSSION

  • Under the primary assumptions, one additional label never had expected value above its cost; across the cost analysis, HB-PVI was utility-optimal in every setting at or above the primary label cost.Fixed personalization was preferable only in a minority of settings with very low label cost and low harm penalty, so the recommendation is conditional on the assumed costs.Source: §IV · p. 7

A. Implications for deployment

The paper recommends population-first deployment under its primary cost assumptions while specifying boundaries on that recommendation. Its supplement details the sequential updater, its validation, and the information used to predict personalization gains.

B. Limitations

  • The 47-participant cohort limits power for rare subgroups and higher-order interactions, and the tested adaptations used a frozen backbone rather than full retraining.The label-efficiency result concerns mean F1 difference, not every individual’s loss; the primary EVSI evaluated only the next chronological window and used shared rather than mechanism-specific runtime costs.Source: B. Limitations · pp. 7–8

V. CONCLUSION

  • Under the primary conditions, a population-first policy eliminated calibration labels while preserving mean F1 within tolerance; this recommendation is conditional on the cohort, pipeline, and cost weights.The conclusion reports a 0.9992 posterior probability that mean F1 loss versus fixed 10-shot adapter personalization remained below the 0.005 tolerance.Source: A. Implications for deployment · p. 7, §V · p. 8

S1.1 Population posterior used for new-participant updating

  • Each outer-fold population model used four NUTS chains and excluded the held-out and validation participants from population-model fitting.Four thousand draws per chain were retained after 2,000 warmup iterations, for 16,000 population-posterior draws.Source: S1.1 Population posterior used for… · p. 9

S1.2 Particle initialization and target distribution

  • Each retained population draw seeded four conditionally independent new-participant effect particles, and the target used only the first n chronological purchased adaptation windows.Each fold began with 64,000 joint particles retaining 16,000 distinct population-posterior ancestors.Source: S1.2 Particle initialization and… · p. 9

S1.3 Adaptive tempered resample-move algorithm

  • The updater tempered likelihood incorporation when a full update would reduce particle ESS below 25%, then used resampling and Metropolis-Hastings rejuvenation.Population-level posterior parameters were never rejuvenated, and states had to retain at least 400 distinct population-draw ancestors and final particle ESS of at least 400.Source: S1.3 Adaptive tempered resample-move… · pp. 9–10

S1.4 Prior specifications for Model I and Model II

  • Model I used a regularized horseshoe for high-dimensional representation coefficients, whereas Model II used a fixed-scale Gaussian prior for its 19 predictors.The stated rationale was to shrink Model I coefficients against overfitting, while the predictor-to-observation ratio was judged modest enough for Model II’s weakly informative prior.Source: S1.4 Prior specifications for Model I… · p. 10

S1.5 Validation gates

  • All sequential-updater validation gates passed on every outer fold, with no fold excluded or given a configuration override.The derived Model II gain-model inputs were therefore not confounded by degenerate particle sets in any participant.Source: S1.5 Validation gates · pp. 10–11

S1.6 Enriched sequential-state dataset

  • The all-fold sequential-state dataset contained 517 unique states, without calibration labels, outer-test arrays, or utility outcomes joined into the state.Adaptation loss was excluded because no complete, consistent, leakage-safe action-by-shot definition was identified for all mechanisms and shots.Source: S1.6 Enriched sequential-state dataset · p. 11

S2 Sequential-state predictor dictionary

The Model II state uses predictor information from five domains, not shot count alone, to condition sequential predictions of personalization benefit.

S2 Sequential-state predictor dictionary

  • The 19 Model II predictors span five information domains and condition benefit predictions on participants’ evolving uncertainty, calibration, and representation-shift trajectory.Predictors were standardized using training-fold-only moments; population sample F1, action sample F1, and gain were retained as outcomes rather than predictors.Source: S2 Sequential-state predictor dictionary · pp. 11–12

S3 EVSI cost specification

The supplement fixes EVSI costs before results are inspected and reports ten-shot outcomes and participant-level adapter heterogeneity. Adapter’s advantage remains small at the group level, while individual gains vary substantially.

S3 EVSI cost specification

  • The sensitivity grid was deliberately broad, and primary cost values were fixed before EVSI results were computed.This design shows how far the population-first recommendation extends beyond one selected cost combination.Source: S3 EVSI cost specification · p. 12, Table S4 · p. 12

S4 Full ten-shot action-specific outcomes, including mean F1

  • Across all ten shots, adapter’s advantage over the other three mechanisms remained small, while adapter-plus-head’s disadvantage widened through roughly shot 8 before leveling off.Group-level mean F1 for every mechanism stayed within about 1.5 percentage points of the 0.9042 population baseline.Source: S4 Full ten-shot action-specific… · p. 12, Table S5 · p. 13

S5 Participant-level heterogeneity in realized adapter gain and exact F1

S6 Held-out gain-model calibration diagnostics

Held-out diagnostics show that gain predictions can be miscalibrated, including a reversal in adapter-plus-head harm predictions. The paired-utility posterior comparisons passed their reported convergence checks.

S6 Held-out gain-model calibration diagnostics

  • Adapter’s meaningful-benefit probabilities were generally overpredicted, and adapter-plus-head’s highest predicted-harm bin had the lowest observed harm frequency.For adapter-plus-head, the highest predicted-harm bin was approximately 0.88, while its observed harm frequency was approximately 0.13.Source: S6 Held-out gain-model calibration… · p. 15, Figure S1 · p. 16

S7 Full paired-comparison posterior diagnostics

S8 Decision-region breakdown by threshold and label cost

Across the decision grid, HB-PVI was optimal at all tested settings once label cost reached the primary value, while modest label or harm costs narrowed the reversal region. The label-efficiency model’s diagnostics support its reported posterior estimates.

S8 Decision-region breakdown by threshold and label cost

  • HB-PVI won in every tested computation-cost and harm-penalty combination once label cost reached 0.001.Fixed-shot comparators won only at label costs of 0 or 0.0005; the main grid results are detailed by threshold and label cost, and Fig. S2 further breaks down label and harm costs.Source: S8 Decision-region breakdown by… · p. 17, Table S9 · p. 17, Figure S2 · p. 18
  • HB-PVI lost to a fixed-shot comparator in only three of 24 cells per Fig. S2 panel, all with cL ≤0.0005 and cR ≤0.005.Increasing either label cost or harm penalty to its next tested level restored the population-first recommendation in every setting.Source: S8 Decision-region breakdown by… · p. 17, Figure S2 · p. 18

S9 Label-efficiency Bayesian and bootstrap diagnostics

  • The label-efficiency model had clean convergence, and its low degrees-of-freedom estimates (ν ≈3.7–5.8) indicate heavy-tailed participant-level F1 losses.This supports the Student-t rather than Gaussian likelihood used for the comparison.Source: S9 Label-efficiency Bayesian and… · p. 18, Table S10 · p. 18

S10 Stopping-decision and action-selection frequencies

Most stopping decisions reflected one-step EVSI below label cost, and population inference was selected almost universally. Positive EVSI occurred only for one participant; the study also reports mixed results across its objectives, including failure to establish harm-risk discrimination.

S10 Stopping-decision and action-selection frequencies

  • One-step EVSI did not exceed label cost in 10,116 of 10,152 decisions (99.6%); population inference was selected in 10,080 (99.3%).The remaining 36 decisions stopped only at the ten-label horizon; head personalization was selected in 72 decisions (0.7%), all at Δmin = 0.Source: S10 Stopping-decision and… · p. 18
  • Every positive-EVSI decision in the 10,152-decision grid came from the same participant.At low label cost the model supported purchases through the ten-label horizon; above about 0.002, it supported exactly one purchase because EVSI dropped sharply afterward.Source: S10 Stopping-decision and… · p. 19, Table S11 · p. 19

S11 Study objectives and outcomes

  • The study did not establish that harm-probability predictions discriminated risk better than a trivial baseline.Although the primary decision rule avoided realized harm by selecting population inference for everyone, participant-level heterogeneity remained substantial.Source: S11 Study objectives and outcomes · p. 19, Table S12 · p. 19

Related papers

Loading 2609.05582v1…