PaperlayerView exact revision on arXiv

Source-linked AI summary

MPT: Missing Prototype Tracking via Barycentric Reconstruction in Vehicular Federated Learning

Hanju Jang, Gyeongmin Han, Sungmin Lee, Kichang Lee, Chunghan Lee, JeongGil Ko

arXiv:2609.12771v1cs.AIcs.CV

TL;DR

When a vehicle carrying most samples of a class departs, the class becomes rare as the shared feature space continues to change. MPT reconstructs its prototype from privacy-preserving class-level statistics, combining barycentric tracking with covariance-based residual prediction and reliability-aware calibration. Across the evaluated tasks, it outperforms representative baselines, including with only 1% of rare-class samples remaining on nuImages.

  • Problem

    When a vehicle holding most samples of a target class departs with its local data, the class becomes rare and its prototype can drift as the shared backbone evolves.

    Source: §I · pp. 1–2

  • Method

    MPT reconstructs the departed rare-class prototype from class-level statistics, tracking shared drift barycentrically and adapting the out-of-span residual using covariance changes and remaining rare-class samples.

    Source: §I · p. 2

  • Results

    Across three vehicle-classification tasks and four backbones, MPT consistently outperforms five representative baselines; it achieves 51.6% rare-class F1 on nuImages with 1% of rare-class samples remaining.

    Source: §I · p. 2, §VII · p. 9

  • Takeaways & Limitations

    MPT preserves rare-class recognition after vehicle departure under FL privacy constraints and without extra retraining.

    Source: §VII · p. 9

    The evaluation focuses on image classification; extending MPT to object detection is identified as future work.

    Source: §VI · p. 9

Abstract

from arXiv · show

Cross-vehicle federated learning enables vehicles to collaboratively improve perception models while keeping locally collected driving data private. However, vehicle participation is transient, and a vehicle may depart before training converges while permanently taking its local data. When this departing vehicle holds most samples of a target class, the class becomes rare in the remaining FL network, and its recognition can silently degrade as the shared backbone continues to evolve. Recovering the class is difficult since the few remaining samples provide a noisy prototype estimate, while FL privacy constraints prevent centralized access to raw data or per-sample features. This paper presents MPT, a cross-vehicle FL framework that maintains rare-class recognition by reconstructing its prototype at every round from privacy-preserving class-level statistics. MPT combines a barycentric decomposition that tracks drift shared with remaining-class prototypes, a covariance-based residual prediction that estimates out-of-span drift, and an adaptive calibration that weighs the remaining rare-class samples according to their reliability. We evaluate MPT on three vehicle classification tasks and four backbones against representative calibration and drift-compensation baselines. MPT outperforms all baselines in rare class F1, reaching 0.516 on the nuImages dataset with only 1\% of rare-class samples remaining, without raw data, per-sample features, or retraining.

I. INTRODUCTION

In cross-vehicle federated learning, vehicles can depart with their local data, leaving a target class rare while the shared model continues to evolve. MPT reconstructs that class’s prototype from privacy-preserving class-level statistics and outperforms baselines in the reported evaluations.

I. INTRODUCTION

  • A departing vehicle may hold most samples of a target class, leaving few samples in the remaining network while its local data is permanently lost.Vehicles may leave at any training round according to their charging schedules.Source: §I · p. 1
  • After departure, continued backbone updates misalign the stored rare-class prototype, while the few remaining samples provide only a noisy estimate.Existing approaches either depend on unstable class-level references or require sample-level features that cannot be centrally collected under FL privacy goals.Source: §I · pp. 1–2
  • MPT reconstructs a departed rare-class prototype using its barycentric relationship to remaining classes, covariance-based residual correction, and aggregate statistics from remaining rare-class samples.It updates the prototype after each FL round using privacy-preserving class-level statistics.Source: §I · p. 2
  • The paper characterizes dynamic participation and heterogeneous data as requiring rapid learning before departure and rare-class prototype tracking afterward.This need arises because continued backbone updates cause representation drift after a vehicle holding most samples of a class departs.Source: §I · p. 2
  • 51.6% rare-class F1 on nuImages with 1% of rare-class samples remaining; MPT exceeds the strongest baseline by 8.1%p and the 50.5% oracle.The evaluation spans three vehicle-classification tasks and four backbones.Source: §I · p. 2

II. RELATED WORK

Prior prototype and drift-compensation approaches face trade-offs in storage, backbone plasticity, or access to sample-level features. The preliminary study motivates tracking shared in-span drift and separately updating the out-of-span residual.

II. RELATED WORK

  • Prototype drift methods recompute from stored exemplars or freeze the backbone, while SDC and LDC depend on sample-level features that conflict with FL privacy constraints.Classifier-head forgetting and post-hoc calibration also require class-specific samples, gradients, or reliable statistics.Source: §II · p. 2

III. PRELIMINARY STUDY

  • After departure, NCM’s early rare-class F1 advantage gives way to a persistent gap from the no-departure curve, even as overall accuracy rises.The study attributes the gap to representation drift and noisy estimates from the few remaining rare-class samples.Source: §III · p. 3
  • Barycentric estimates stay closer to the oracle in-span component than direct estimates from the few remaining rare-class samples, but the out-of-span residual mismatch grows over time.The study therefore motivates updating both the shared in-span drift and the residual.Source: §III · p. 3
  • Post-departure rare-class F1 loss persists across nuImages classes and remaining-sample fractions, including when samples are relatively available.The study reports average per-round F1 decrease relative to each class’s non-departure curve over 25 post-departure rounds.Source: §III · p. 3
  • MPT’s design requirements are NCM before departure, shared-drift tracking from updated remaining-class prototypes, and reliability-aware updating of the out-of-span residual.The remaining rare-class samples alone cannot reliably track the changing prototype.Source: §III · p. 3

IV. MPT DESIGN

After a vehicle holding most samples of a class departs, remaining prototypes track shared drift but miss changing out-of-span drift. MPT models this as estimating the rare-class prototype from an affine combination of remaining-class prototypes and a residual.

IV. MPT DESIGN

  • After departure, remaining-class prototypes track shared drift but cannot capture the rare class’s changing out-of-span residual.The resulting task is to estimate the rare-class prototype as the shared backbone continues to evolve.Source: §IV · p. 4

A. Problem Formulation

  • MPT represents the rare-class prototype as an affine combination of remaining-class prototypes plus a residual.After departure, the in-span component is updated from remaining-class prototype statistics, while covariance changes predict the residual.Source: A. Problem Formulation · p. 4
  • MPT uses penultimate-layer mean feature vectors as class prototypes.For an input x, its feature is extracted immediately before the classifier.Source: A. Problem Formulation · p. 4

B. Overall Workflow

MPT maintains globally aligned class-level statistics during federated training and stores a departure-time relationship between the rare-class prototype and remaining-class prototypes. After departure, it applies that relationship to updated prototypes and adapts the residual to current feature-space geometry.

B. Overall Workflow

  • Before departure, clients train the shared model and report class-wise feature means, second moments, and sample counts for global prototype and covariance statistics.The workflow repeats model aggregation and re-embedding until departure, using an NCM readout in place of the classifier head.Source: B. Overall Workflow · p. 4
  • After departure, the server applies the stored barycentric relationship to updated remaining-class prototypes and predicts residual changes from covariance differences.Aggregate statistics from any remaining rare-class samples can further calibrate the reconstruction.Source: B. Overall Workflow · p. 4

C. Rare-Class-Prototype Estimation

1) Departure-time prototype decomposition:

  • The affine weights sum to one, making the relationship invariant to common prototype translation, while an L2 penalty discourages large coefficients and stabilizes the fit.The unexplained component is stored as a residual alongside the affine combination.Source: 1) Departure-time prototype… · p. 4
  • MPT stores pooled within-class covariance and applies trace-scaled ridge regularization to reduce sensitivity to poorly estimated low-variance directions.Covariance changes then adapt the stored residual before it is combined with the updated in-span component.Source: 1) Departure-time prototype… · pp. 4–5

2) Post-departure calibration:

After departure, MPT reconstructs the rare-class prototype from shared drift in remaining-class prototypes and a residual that is measured from available rare-class samples or predicted from covariance changes. It weights those residual estimates according to sample reliability.

2) Post-departure calibration:

  • The reconstruction combines shared drift from updated remaining-class prototypes with a residual that is measured or predicted after departure.The shared-drift term applies the fixed departure-time affine weights to current prototypes.Source: 2) Post-departure calibration · p. 5
  • MPT measures rare-class-specific residual drift by projecting the mean feature of remaining rare-class samples onto the subspace complementary to current remaining-class prototypes.These samples are held out rather than used for backbone training.Source: 2) Post-departure calibration · p. 5
  • The measured residual captures rare-class-specific drift not explained by remaining-class prototypes, but its reliability can decrease when few rare-class samples remain.Thus, scarce samples constrain the reliability of the sample-based calibration.Source: 2) Post-departure calibration · p. 5
  • When rare-class samples are insufficient, MPT predicts the residual by transporting the stored residual according to changes in remaining-class covariance.The transport adapts the residual’s scale and orientation and can operate without rare-class samples.Source: 2) Post-departure calibration · p. 5
  • MPT balances measured and predicted residuals according to the reliability of available rare-class samples, setting the measurement weight to zero when none remain.Larger, reliable sample sets increase the weight assigned to measurements.Source: 2) Post-departure calibration · p. 5

D. Mahalanobis Nearest-Class-Mean Readout

MPT uses a Mahalanobis nearest-class-mean readout with the reconstructed rare-class prototype and current remaining-class prototypes, avoiding the original classifier without retraining.

D. Mahalanobis Nearest-Class-Mean Readout

  • The NCM readout predicts the class whose prototype is closest under a regularized Mahalanobis distance, using covariance estimated from remaining-class features.The prototype set combines the reconstructed rare-class prototype with current prototypes for remaining classes.Source: D. Mahalanobis Nearest-Class-Mean… · p. 5
  • The readout incorporates the estimated rare-class representation without retraining and avoids the original classifier, which becomes increasingly biased after departure.It uses the covariance statistics already maintained by MPT.Source: D. Mahalanobis Nearest-Class-Mean… · p. 5
  • The Mahalanobis metric accounts for direction-dependent feature variation to help distinguish the rare class from nearby confusable classes.Source: D. Mahalanobis Nearest-Class-Mean… · p. 5

V. EVALUATION

The evaluation uses extensive experiments across three datasets and multiple backbones in a vehicular federated-learning setting.

V. EVALUATION

  • Experiments evaluate MPT on three datasets and multiple backbones in a vehicular federated-learning setting.The passage does not specify further setup or results.Source: §V · p. 5

A. Evaluation Setup

The evaluation covers road-object and vehicle-classification tasks, several backbone architectures, and a simulated vehicle-departure scenario. Rare-class F1 is the primary metric, and comparisons include calibration, re-estimation, drift-compensation, and reference methods.

Backbones and scenario

  • The default setting uses a three-layer CNN over 100 rounds, with departure at round 15 after the dominant vehicle holds 98% of rare-class samples.The remaining 2% of rare-class samples are distributed across active vehicles; ResNet-18, MobileNetV3, and ViT-Tiny are also tested.Source: A. Evaluation Setup · p. 6

Datasets and tasks

  • nuImages, coarse-grained Car-1000, and fine-grained Car-1000 tasks test road objects, vehicle categories, and visually similar sedan subcategories.Truck is the nuImages rare class, van the coarse-grained Car-1000 rare class, and micro sedan the fine-grained rare class.Source: A. Evaluation Setup · pp. 5–6

Baselines

  • Baselines cover standard FedAvg, calibration, direct prototype re-estimation, explicit drift compensation, and frozen-prototype and oracle references.The capability comparison defines PD as post-departure drift compensation, HFL as heterogeneous-FL support, PF as no per-sample feature storage, and TF as no additional training.Source: A. Evaluation Setup · p. 6

Evaluation metric

  • Rare-class F1 is the primary metric because overall accuracy can remain high even when rare-class performance drops.The metric accounts for both recall and precision.Source: A. Evaluation Setup · p. 6

B. Overall Performance

MPT remains closest to the oracle after vehicle departure and outperforms the compared calibration, prototype re-estimation, and drift-compensation baselines. Its system-cost trade-offs differ between the lightweight and full variants.

Communication overhead

  • MPT-lite gains F1 with only a small communication-cost increase, while full MPT adds a covariance summary per client.MPT uses one covariance summary to adapt the out-of-span residual; CCVR exchanges separate class covariances and also generates ghost features and retrains the classifier head.Source: Figure 5 · p. 7, B. Overall Performance · p. 7

Overall rare-class performance

  • MPT stays closest to the oracle in post-departure rounds, while CCVR, iCaRL, SDC, and LDC remain below it.MPT-lite also stays near the oracle, but MPT nearly overlaps it throughout the post-departure rounds.Source: B. Overall Performance · pp. 6–7

Training-time overhead

  • MPT achieves the strongest rare-class F1 at nearly the same wall-clock overhead as MPT-lite.LDC and CCVR incur substantially larger overhead, while MPT's adaptive residual weighting maintains stable F1 across remaining-sample fractions.Source: B. Overall Performance · p. 7

C. Generalization under Various Configurations

Across classification tasks and backbone architectures, MPT variants achieve the highest rare-class F1 in nearly all reported configurations. MPT's advantage over MPT-lite is more pronounced for fine-grained classes and ViT-Tiny.

Evaluation scope

  • The evaluation compares MPT with five representative baselines across three classification tasks and four backbone architectures, varying the remaining rare-class sample fraction from 1% to 10%.Source: C. Generalization under Various… · p. 7

Across backbones

Across classification tasks

  • MPT leads in 11 of 12 task settings, with margins over the strongest non-MPT baseline of 8.1%p on nuImages, 3.2%p on coarse-grained Car-1000, and 4.9%p on fine-grained Car-1000 at 1%.MPT-lite is 0.3%p higher on nuImages at 10% remaining samples, within reported variation.Source: C. Generalization under Various… · p. 8

Representation geometry

  • MPT's gap over MPT-lite grows on ViT-Tiny, which the authors conjecture reflects the limits of scalar correction in a more curved feature space.The paper presents MPT-lite as a lower-communication option for backbones with near-linear feature drift.Source: C. Generalization under Various… · p. 8

D. Method Analysis

The analysis isolates the contributions of in-span tracking and out-of-span correction to prototype reconstruction. It finds that barycentric reconstruction tracks shared in-span drift, while continued alignment requires updating the out-of-span residual and adapting to the reliability of remaining samples.

D. Method Analysis

  • Barycentric tracking stabilizes in-span reconstruction, but the fixed departure-time residual becomes outdated; sustained alignment requires residual updates.The analysis uses Mahalanobis distance to the current oracle prototype as its reconstruction-error measure and reports that closer prototypes achieve higher rare-class F1.Source: D. Method Analysis · pp. 8–9
  • With 2% of rare-class samples remaining, MPT weights prediction more heavily; with 20%, combining prediction and measurement yields lower error than either source alone.The comparison measures reconstructed against current oracle out-of-span residuals.Source: Figure 7 · p. 9, D. Method Analysis · p. 9
  • Reserving remaining rare-class samples for prototype measurement avoids the gradual rare-class F1 decline seen when those samples also train the backbone.The two approaches perform similarly near departure; the training variant loses F1 as FL continues.Source: Figure 7 · p. 9, D. Method Analysis · p. 9

VI. DISCUSSIONS

The discussion identifies classification-only evaluation and controlled parking-lot mobility as scope boundaries, while describing a feasible pre-departure statistics exchange. The conclusion reports that MPT outperforms five baselines across three vehicle-classification tasks and four backbones without extra retraining.

VI. DISCUSSIONS

  • Evaluation covers image classification; extending MPT to detection could replace the detection head’s classification branch to help preserve rare-class recognition.The discussion leaves evaluation on other perception tasks as future work.Source: §VI · p. 9
  • MPT requires current class-level statistics before departure; charging scenarios can trigger a final upload, after which the server stores reconstruction statistics if a class becomes rare.Vehicles already report class-wise feature statistics and sample counts each round.Source: §VI · p. 9
  • Evaluation isolates permanent departure in a controlled parking-lot scenario; dynamic arrivals, departures, and route-dependent data distributions remain unevaluated.The authors leave evaluation under more realistic mobility and data-distribution dynamics to future work.Source: §VI · p. 9

VII. CONCLUSION

  • Across three vehicle-classification tasks and four backbones, MPT outperforms five representative baselines while closely tracking the oracle prototype, without extra retraining.The conclusion frames this result as preserving rare-class recognition after departure under FL privacy constraints.Source: §VII · p. 9

Related papers

Loading 2609.12771v1…