An Efficient Federated Distillation Learning System for Multi-task Time Series Classification

• Earlier TSC research introduced approaches including fully convolutional networks, TapNet, FCN-LSTM, RTFN, shapelet-neural networks, and dynamic temporal pooling.Source: §1 · p. 1
• Federated learning lets users benefit from shared models without sending their original data, and FedAvg, FTL, and FKD are identified as its three mainstream directions.Source: §1 · p. 1
• Multi-task TSC requires secure sharing of expertise across tasks, but varied dataset characteristics make this difficult and privacy and knowledge sharing both need attention.The introduction notes that datasets can differ in length and variance and be strongly Non-I.I.D.Source: §1 · p. 2
• EFDLS uses feature-based knowledge distillation within each user and matches each user with the most similar expertise by least square distance for cross-user sharing.Unlike traditional FKD, which uses average user weights, EFDLS selects a partner for each user.Source: §1 · p. 2
• EFDLS outperforms six state-of-the-art FL algorithms across 44 UCR 2018 datasets on mean accuracy, win/tie/lose, and AVG rank, all based on top-1 accuracy.Source: §1 · p. 2

2 RELATED WORK

• The section broadly notes that many traditional and deep-learning algorithms have been developed for TSC.Source: §2 · p. 2

2.1 Traditional Algorithms

• Traditional TSC methods include distance-based DTW-nearest-neighbor ensembles and feature-based approaches such as shapelet transformation, bag-of-features, and convolutional-kernel features.Source: §2.1 · pp. 2–3

2.2 Deep Learning Algorithms

• Deep TSC models are described as single-network or dual-network architectures, with examples including FCN, ResNet, FCN-LSTM, RTFN, and TapNet.Source: §2.2 · p. 3
• Most existing algorithms emphasize single-task TSC, while secure and efficient sharing of similar expertise across different tasks remains challenging.Source: §2.2 · p. 3

3.1 System Overview

• Users run one time-series classification task each, while the system supports different tasks or data sources across users through a central server and mobile clients.Users train locally and share model weights through the server.Source: §3.1 · pp. 3–4

3.3 Distance-based Weights Matching

• DBWM matches each connected user with the user whose model weights are most similar, using least square distance to measure weight similarity.The server launches matching after receiving all connected users’ uploaded weights.Source: §3.1 · p. 4, §3.3 · p. 4

3.2 Feature-based Student-Teacher Framework

• FBST uses teacher and student models with identical network structures, transferring hidden-layer knowledge from teacher to student to support richer feature representations.The student captures representations from its input data under teacher supervision.Source: §3.2 · p. 4

3.2.1 Feature Extractor

• The feature extractor uses three 1D convolutional blocks, average pooling, and a dense layer for local-feature extraction, followed by a dense-layer and Softmax classifier.Each convolutional block contains a convolution module, batch normalization, and ReLU.Source: §3.2.1 · p. 4

3.2.2 Knowledge Distillation

• Knowledge distillation defines student training loss using teacher and student hidden-layer outputs, alongside supervised cross-entropy loss.The paper sets ε = 0.9.Source: §3.2.2 · p. 4

3.3.1 Least Square Distance

• The server computes pairwise squared distances between uploaded student hidden-layer weights and uses each distance row’s minimum to identify a matching user.Matched weights are sent to users, who load them into their teachers’ hidden layers at the next federated learning epoch.Source: §3.3.1 · p. 5

3.4 Communication Overhead

• EFDLS waits until all connected users’ weights arrive before matching, and its total communication overhead is proportional to 2BW · FLEs · Nconn.Uploading and downloading each require BW per user per epoch because each user has the same model structure.Source: §3.4 · p. 5

• The evaluation section introduces its experimental setup and performance metrics, presents an ablation study, and then evaluates EFDLS.Source: §4 · p. 6

4.1.1 Data Description

• 44 datasets were selected from the UCR 2018 archive’s 128 because available computing resources could not support evaluating the full archive.The selection comprised 11 datasets from each of four length categories.Source: §4.1.1 · p. 6

4.1.2 Implementation Details

• Training used batch-normalization decay 0.9, L2 regularization, and Adam with an initial learning rate of 0.0001.Source: §4.1.2 · p. 6
• Table 2 reports results for different algorithms on 44 datasets with Nconn = 44 and Ntot = 44.Source: Table 2 · p. 7
• Experiments ran on an Nvidia GTX 1080Ti GPU with 11GB memory and an AMD R5 1400 CPU with 16G RAM under Ubuntu 18.04.Source: §4.1.2 · p. 7

• ‘Win’/‘tie’/‘lose’ count datasets where an algorithm performs better than, equal to, or worse than the others; ‘best’ sums wins and ties.Source: §4.2 · p. 7
• AVG rank measures the average difference between a model’s accuracy and the best accuracy among all models.Source: §4.2 · p. 7

4.3 Ablation Study

• The ablation uses 44 users across the 44 datasets, with each user assigned data from one dataset; users may share a task but use different data sources.Source: §4.3 · pp. 7–8

4.3.1 Impact of Nconn

• A larger Nconn tends to produce higher MeanACC across 44 datasets, with more connected users making more time-series data available to the system.Source: §4.3.1 · p. 8, Figure 2 · p. 8

4.3.2 Impact of ϵ

• With Nconn = 44 and Ntot = 44, ϵ = 0.90 yields the highest MeanACC, 0.7014.The paper describes ϵ as balancing each connected user’s supervised and KD losses.Source: §4.3.2 · p. 8, Figure 3 · p. 8

4.4 Experimental Analysis

• EFDLS wins 18 of 44 datasets versus 4 for Baseline, which uses standalone feature extractors without knowledge sharing.Source: §4.4 · p. 8, Figure 4 · p. 9
• EFDLS outperforms FTLS on ‘best’, MeanACC, and AVG rank; the paper attributes the contrast to FBST’s teacher-to-student knowledge transfer.Source: §4.4 · p. 8, Figure 4 · p. 9
• EFDLS outperforms FKD on ‘best’, MeanACC, and AVG rank; DBWM pairs users with similar expertise, unlike FKD’s use of average weights.The paper notes that averaging may lead to catastrophic forgetting when a user’s weights differ substantially from the average.Source: §4.4 · pp. 8–9, Figure 4 · p. 9
• EFDLS leads all seven comparison algorithms with MeanACC 0.7014, ‘best’ 20, and AVG rank 2.1478; FKD is second on these metrics.FedAvg and FedAvgM are reported as the two worst algorithms.Source: §4.4 · pp. 8–9, Figure 4 · p. 9

• FBST transfers knowledge from a teacher’s hidden layers to its student’s, while DBWM pairs users with the most similar uploaded weights.Together, the components support secure sharing of similar expertise across different tasks.Source: §5 · p. 9
• EFDLS is a winner over six benchmark FL algorithms on 44 datasets by MeanACC and AVG rank, and on 20 datasets by the ‘best’ measure.Source: §5 · p. 9
• Against single-task Baseline, EFDLS records 32/4/8 wins/ties/losses.Source: §5 · p. 9