PaperlayerView exact revision on arXiv

Source-linked AI summary

Unknown-Traffic Detection, Calibration and Shortcut Reliance in Distilled Encrypted-Traffic Classifiers over One Year

Mahmoud Abbasi

arXiv:2609.31141v1cs.NIcs.LG

TL;DR

This paper asks whether encrypted-traffic classifiers distilled for edge deployment inherit properties beyond accuracy, and whether those properties reflect a specific teacher rather than soft-target regularisation. It uses a pre-registered teacher-swap design on year-long TLS data to test inheritance, calibration, shortcuts, and drift. The reported findings are selective: score-pattern inheritance and transmitted over-confidence appear under specified conditions, while other expected benefits are absent or reversed.

  • Problem

    Distillation work on encrypted-traffic classifiers has largely evaluated accuracy, leaving limited evidence about inherited unknown detection, calibration, shortcut reliance, and persistence under drift.

    Source: §I · p. 1

  • Method

    The study distils a 101k-parameter student from two equally accurate, differently constructed teachers and uses a direct student baseline in a pre-registered evaluation across 18 test windows.

    Source: §I · p. 1, Contributions · p. 2, E. Hypotheses and Analysis · p. 5

  • Results

    The student’s per-flow unknown-score pattern shifts toward its own teacher only at the conventional temperature, and a shortcut-reliant teacher passes over-confidence to a student that never sees the feature.

    Source: Contributions · p. 2

  • Takeaways & Limitations

    Under the registered logit-based scores, distillation adds no measured unknown-detection benefit over a temperature-scaled direct student, while teacher-specific habits can still transfer.

    Source: Contributions · p. 2, §VII · p. 14

    The study covers one dataset, one network, and one year; natural shortcut fields such as server IP, ASN, and JA3 were excluded rather than tested.

    Source: Limitations · pp. 13–14

Abstract

from arXiv · show

Knowledge distillation is the standard way to compress encrypted-traffic classifiers for the edge, and almost all such work judges students by accuracy alone. We ask what else a student inherits: unknown-traffic detection, calibration, shortcut reliance, and whether any survives a year of drift. Resemblance proves little on its own, since soft targets also regularise. We therefore distil one 101k-parameter student from two teachers of equal accuracy but different construction, a five-member ensemble and a single wider model, so that following one rather than the other is attributable to it. The design was pre-registered before any test result was seen. We tested ten hypotheses on CESNET-TLS-Year22, a year of real TLS traffic, across 18 test windows over 35 weeks. Two are supported: a student's per-flow unknown-scores shift toward its own teacher, but only at a conventional temperature, not the accuracy-optimal one; and a shortcut-reliant teacher passes its over-confidence to a student that never sees the feature. The drift prediction is reversed under both scores, the gap narrowing rather than widening and the student overtaking under the energy score in two of three replicates, as is the prediction that such a teacher harms its student's detection, which improves slightly. Shortcut reliance is set by model size, not distillation. Under the logit-based scores nothing else transfers: distillation beats neither a temperature-scaled direct student nor label smoothing. Exploratory analysis shows this turns on the scoring rule: with a feature-space detector the teacher detects unknown traffic 0.073 AUROC better than the direct student, where the energy score sees 0.000, and the conventional-temperature student inherits most of it. Label smoothing, with no teacher, recovers more. Distillation transfers the teacher's habits; what looks like an inherited ability is available without one.

I. INTRODUCTION

This study asks whether distilled encrypted-traffic classifiers inherit deployability-relevant properties beyond accuracy, and whether any such properties come from a particular teacher rather than soft-target regularisation. It introduces a teacher-swap control and examines the question against prior work on distillation, open-set detection, drift, and shortcuts.

II. RELATED WORK

I. INTRODUCTION

  • The paper tests inheritance across a year because traffic drifts, unknown services appear, and shortcuts can predict labels without representing intended signals.The study asks whether a small student inherits a large model’s handling of these deployment concerns.Source: §I · p. 1
  • Soft targets can regularise without transmitting teacher-specific behaviour, so resemblance between a student and teacher alone cannot establish inheritance.The proposed control changes teacher identity while holding the student design fixed.Source: §I · p. 1
  • A teacher-swap control tests whether a student’s per-flow unknown-score pattern shifts toward its specific teacher beyond the preference of a directly trained student.The study uses two equally accurate teachers built differently and distils each into the same student.Source: §I · p. 1
  • The confirmatory grid uses teachers at the small end of the model range, so it cannot settle whether the findings hold for hundred-million-parameter foundation-model teachers.The tested ensemble has 11.3M parameters and is distilled into a 101k-parameter student.Source: §I · pp. 1–2
  • The pre-registered evaluation covers 18 test windows over 35 weeks and tests ten hypotheses with Holm correction.Two hypotheses were supported, three reversed, and five were not supported.Source: §I · p. 2, Contributions · p. 2

II. RELATED WORK

  • Prior transfer research reports mixed evidence on inherited outputs, invariances, biases, and calibration, but does not use teacher identity to test open-set detection over time.The paper frames its teacher-identity test as distinct from forensic tracing of student models.Source: What distillation transfers · p. 2, Whose student is it? · p. 2
  • Work on shortcut mitigation and compressed-model brittleness motivates measuring whether shortcuts pass through distillation, including when students never see the shortcut feature.The paper measures reliance rather than proposing a mitigation method.Source: Shortcuts and compression · p. 3

III. EXPERIMENTAL DESIGN

The experimental design uses a pre-registered teacher-swap comparison across three time-based replicates. CESNET-TLS-Year22 supplies TLS flows, while held-out services and consecutive test windows operationalise unknown detection and temporal evaluation.

III. EXPERIMENTAL DESIGN

  • Across three start dates, two accuracy-matched but differently constructed teachers are distilled into the same student, and inheritance is measured against a teacher-free student baseline.Subtracting the direct student’s teacher preference removes preference not attributable to distillation.Source: §III · p. 3

A. Data

  • CESNET-TLS-Year22 contains one year of TLS flows labelled with 180 web services; the study uses its 25M-flow S subset after excluding weeks affected by an exporter update or low volume.No service first appears mid-year, so unknown services are simulated by holding them out.Source: Dataset · p. 3
  • Each flow is represented by its first 30 packets, flow statistics, and a packet-size histogram, with server name, IP addresses, ports, autonomous system, and JA3 excluded.Input scaling is fitted on each training window only.Source: Inputs · p. 3
  • A fixed split designates 102 services as known, with separate validation and test unknowns divided into near services and whole held-out categories.The remaining 46 services are excluded, mostly for insufficient or inconsistent volume.Source: Known and unknown services · p. 3
  • Three start dates use four training weeks, one validation week, and consecutive four-week test windows, yielding 18 windows spanning 3.5 to 35.3 weeks after training.Each window samples 100k known and 100k test-unknown flows, except the final, year-truncated window.Source: Time structure · p. 3

B. Models

The model comparison distils a shared small student from two teachers of different construction, with training conditions varying the teacher and distillation approach. Temperature tuning separates an accuracy-optimal arm from the conventional-temperature arm used to test inheritance.

B. Models

  • Teacher A is a five-member ensemble and Teacher B is a single wider network from the same family; both are distilled into the same 101k-parameter student.Teacher B was within 0.2 macro-F1 points of Teacher A on all three validation weeks.Source: B. Models · pp. 3–4
  • The ensemble prediction averages member softmax outputs and member energies, keeping its energy score on the scale of a single member’s score.The aggregation choice was fixed in the pre-registration.Source: B. Models · p. 3

C. Training Conditions

  • Students share the same architecture and training recipe, while Hinton distillation varies the teacher and temperature and uses α = 0.9.Students train for 20 epochs; the retained checkpoint minimises validation known-flow cross-entropy.Source: C. Training Conditions · p. 4
  • Validation macro-F1 selected T = 1 as the accuracy-optimal temperature, while T = 4 was added before pre-registration freeze for inheritance and shortcut tests.At T = 1, nearly one-hot teacher targets made distilled students’ per-flow scores indistinguishable from direct training in the tuning grid.Source: Tuning · p. 4
  • The pre-registered training conditions are distinguished from the lower, exploratory conditions added after confirmatory results were known.Table I’s caption identifies the upper block as pre-registered and run at all three start dates.Source: Table I · p. 4

D. Outcomes

The analysis evaluates classification, unknown detection, calibration, and risk across models and test windows, with hypotheses specifying teacher inheritance, gains beyond regularisation, drift, shortcuts, and distribution distillation. Uncertainty is estimated by resampling day-by-service clusters and student seeds.

D. Outcomes

  • Outcomes include macro-F1, energy- and MSP-based unknown-detection AUROC and FPR at 95% TPR, OSCR, calibration metrics, and risk–coverage, across flow groups and unknown-service sets.Per-flow inheritance is measured by Spearman correlation between student and teacher energy scores.Source: D. Outcomes · p. 4

E. Hypotheses and Analysis

  • The analysis unit is one start date × test window, with 18 units; bootstrap resampling redraws day × service clusters and student seeds.Intervals are 95% percentile intervals, and Holm correction is applied across ten hypotheses.Source: E. Hypotheses and Analysis · p. 5
  • H1 tests whether T = 4 students shift their per-flow score correlations toward their own teacher beyond the direct student’s baseline preference.Both teacher directions must show a positive difference because all students correlate more with the ensemble.Source: E. Hypotheses and Analysis · p. 5
  • H2 requires distilled students to exceed label smoothing and direct temperature scaling on unknown-detection AUROC and post-hoc NLL while remaining matched in macro-F1.H3 predicts that the Teacher A–student AUROC gap grows with weeks since training.Source: E. Hypotheses and Analysis · p. 5
  • The shortcut experiment plants a class-correlated feature and tests both visible-feature reliance and changes in ECE or energy AUROC when a distilled student never sees it.Reliance is the macro-F1 difference between aligned and shifted feature values.Source: E. Hypotheses and Analysis · p. 5
  • H5 tests whether ensemble distribution distillation exceeds logit distillation in unknown-detection AUROC.Sensitivity analyses remove exact training-flow duplicates or flows with fewer than five packets, outside the Holm family.Source: E. Hypotheses and Analysis · p. 5

IV. CONFIRMATORY RESULTS

Across 18 test windows, the teachers were more accurate than the students, while the accuracy-tuned distillation arm matched the directly trained student across measured outcomes. The teachers were closely matched to support the teacher-swap comparison, and sensitivity analyses agreed with the primary decisions on the hypotheses they covered.

A. Setting and Headline Numbers

  • Teachers exceeded every student by 4.8 to 7.1 macro-F1 points, but led kdA by only 0.001 energy AUROC (CI −0.003 to 0.005).With MSP, their lead was 0.025 AUROC (CI 0.023 to 0.026).Source: A. Setting and Headline Numbers · p. 5
  • At T = 1, the accuracy-tuned distillation arm was indistinguishable from the directly trained student on every reported column.Source: A. Setting and Headline Numbers · p. 5
  • Teacher B trailed Teacher A by 0.55 macro-F1 points on average across windows, within the pre-registered tolerance for the teacher-swap comparison.Source: A. Setting and Headline Numbers · p. 5
  • Both sensitivity analyses reached the same decision on each of the seven covered hypotheses, with every p-value within 0.1 of the primary analysis.They removed exact duplicate training flows or restricted testing to flows with at least five packets; shortcut hypotheses were not covered.Source: A. Setting and Headline Numbers · p. 5

B. RQ1: What the Student Inherits

Distillation at a conventional temperature shifts students’ per-flow energy-score patterns toward their own teacher, but this resemblance does not translate into better measured detection or calibration than the controls. The transfer depends on temperature and includes the teacher’s strengths and weaknesses across unknown-service types.

C. RQ2: The Gap over Time (H3)

The teacher–student unknown-detection gap narrowed over time rather than widening, while shortcut results distinguished feature-visible reliance from confidence transferred through soft targets. Exploratory deployment comparisons found the student substantially faster and smaller than the re-tuned XGBoost alternative.

C. RQ2: The Gap over Time (H3)

  • The teacher–student energy-AUROC gap shrank by −0.0004 per week (CI −0.0007 to −0.0001), opposite the preregistered prediction, while the teacher remained nominally ahead by +0.001 across pooled windows.Every student’s macro-F1 fell from about 0.90 to about 0.69 over 35 weeks; energy-score crossings occurred at start dates 11 and 37, whereas MSP’s gap narrowed but never crossed.Source: C. RQ2: The Gap over Time (H3) · p. 7

D. RQ3: Shortcuts (H4a, H4b1, H4b2)

  • Feature-visible students relied on the shortcut about twice as much as the teacher below ρ = 1, regardless of training, while teacher-only shortcut exposure increased student ECE by +0.0086.The teacher-only student’s energy AUROC instead increased by +0.020, reversing the predicted detection harm.Source: D. RQ3: Shortcuts (H4a, H4b1, H4b2) · p. 7

E. Baselines and Deployment Cost

  • In a separate timing session, re-tuned XGBoost was 200× the student’s latency and 180× its weight footprint.The re-tuned model lost 2.9 macro-F1 points (0.854 against 0.883) and 0.029 energy-margin AUROC (0.826 against 0.855); this exploratory comparison was outside the Holm family.Source: E. Baselines and Deployment Cost · pp. 7–8

V. EXPLORATORY ANALYSES

Exploratory re-scoring finds that the apparent teacher advantage depends on the detector: feature-space scores reveal an advantage that the energy score does not. The conventional-temperature student recovers most of that feature-space advantage, while label smoothing recovers more.

V. EXPLORATORY ANALYSES

  • The exploratory analysis re-scores existing checkpoints with Mahalanobis and feature k-NN detectors; it was added after confirmatory results and was not pre-registered.No models were retrained for this comparison.Source: A. A Dedicated Open-Set Detector · p. 8, §V · p. 8

A. A Dedicated Open-Set Detector

  • Teacher A exceeds the direct student by 0.073 Mahalanobis AUROC and 0.034 feature k-NN AUROC, versus 0.000 energy AUROC; Fig. 5 compares the scoring rules.Teacher A’s advantage under MSP is 0.024.Source: A. A Dedicated Open-Set Detector · p. 9, Figure 5 · p. 10
  • kdA4 recovers +0.076 Mahalanobis AUROC over direct, near the +0.073 available, while accuracy-optimal kdA recovers −0.003 with an interval spanning zero.The comparison makes the temperature split visible in detection outcomes, not only teacher-preference rank correlation.Source: A. A Dedicated Open-Set Detector · p. 10
  • kdA4 is worse than direct under energy (−0.026) and MSP (−0.020), but better under both feature-space scores.The detector’s choice changes which model appears stronger.Source: A. A Dedicated Open-Set Detector · p. 10
  • Label smoothing exceeds kdA4 by +0.011 Mahalanobis AUROC and +0.017 feature k-NN AUROC; its +0.004 energy-score difference is indistinguishable from zero.This teacher-free control matches or beats distillation where the teacher has a feature-space advantage.Source: Table VIII · p. 9, A. A Dedicated Open-Set Detector · p. 10
  • MSP intervals are indicative because float16 storage collapses highly confident scores to 1.0; teacher–student Mahalanobis comparisons are also indicative across differing feature dimensions.The energy score is unaffected by the MSP storage issue; identical-architecture student comparisons are not affected by the dimensionality caveat.Source: A. A Dedicated Open-Set Detector · p. 10
  • Ensemble-distribution distillation is worst under energy (−0.092 against kdA) but among the best under feature-space scores (+0.077 and +0.043 against kdA).Both feature-space intervals clear zero; the passage links this contrast to the proxy-Dirichlet objective’s effect on logit scale and representation.Source: A. A Dedicated Open-Set Detector · pp. 10–11

B. Separating Teacher Identity from Ensembling, Width and Family

Exploratory teacher swaps indicate that the inherited per-flow score preference is tied to teacher identity, not merely ensembling, width, or matching teacher labels. The largest measured shift is across model families, though that result is reported as a probe.

B. Separating Teacher Identity from Ensembling, Width and Family

  • Students distilled from two same-architecture ensemble members follow their own member by +0.022 and +0.019, exceeding the registered kdA4 arm’s +0.007.The paired teachers differ only by random seed, so ensembling and capacity do not distinguish them.Source: B. Separating Teacher Identity from… · p. 11
  • HardA’s teacher-preference shift is −0.0000, while its validation macro-F1 and accuracy match the direct student at 0.913 and 0.941, respectively.The passage attributes the inherited pattern to soft targets rather than copied top-1 decisions.Source: B. Separating Teacher Identity from… · p. 11
  • A student distilled from transformer Teacher C follows it over Teacher A by +0.043, the largest measured shift; this probe uses nine units from one start date.Teacher C is within 0.6 macro-F1 points of Teacher A on its validation week.Source: B. Separating Teacher Identity from… · p. 11

C. A Null Scale for the Teacher Preference

The null comparison estimates how much teacher preference varies between equally good members of the same family when no teacher-specific effect is expected. Against that scale, some measured shifts are clearly larger, while the registered kdA4 shift is small.

C. A Null Scale for the Teacher Preference

  • Across 540 direct-student comparisons between Teacher A members, the null preference variation has median 0.006, 95th percentile 0.016 and maximum 0.025.The members are equally good and share architecture and training data, so the passage treats variation as noise.Source: C. A Null Scale for the Teacher… · p. 11
  • Identity-only shifts (+0.019, +0.022), kdB4 (+0.026) and the family swap (+0.043) exceed the null 95th percentile; kdA4’s +0.007 does not.The passage says kdA4 remains supported because its interval excludes zero, but is small relative to arbitrary same-family variation.Source: C. A Null Scale for the Teacher… · p. 11
  • The null variation and H1’s difference-in-differences are not strictly commensurable because H1 already removes the baseline preference.The comparison is reported without resolving whether +0.007 is large.Source: C. A Null Scale for the Teacher… · p. 11

D. Aggregation as an Explanation for the Ageing

The proposed ageing explanation is not aggregation under the energy score: the ensemble and its members age at the same rate, leaving slower student ageing as the remaining account. The section also reports overlapping test windows and a student-width sweep.

D. Aggregation as an Explanation for the Ageing

  • The ensemble and its members both age at −0.00285 energy AUROC per week; aggregation explains none of the reversed energy-score result.For MSP, the full-precision ensemble ages at −0.00316 per week versus −0.00307 for its members, accounting for about a quarter of the narrowing.Source: D. Aggregation as an Explanation for… · p. 11
  • The remaining energy-score interpretation is that the 101k student ages more slowly than the teacher and its members.Under MSP, the teacher does not lose its lead in any replicate, so the recommendation does not depend on which mechanism operates.Source: D. Aggregation as an Explanation for… · p. 11

E. Dependence Between Start Dates

  • Fifteen of 18 units share at least one test week, and the bootstrap handles within-window but not between-series dependence, making confirmatory intervals narrower than independent replication would.The 55 shared unit-weeks limit how independently the three start-date series can be interpreted.Source: E. Dependence Between Start Dates · p. 12
  • The H3 gap narrowed at every start date for kdA4; for tuned kdA, two slopes were negative and the middle series was flat.No start date showed the predicted widening gap.Source: E. Dependence Between Start Dates · p. 12

F. Student Capacity

  • Unknown detection does not improve beyond student width 48: width 16 is −0.024 energy AUROC versus width 48, while width 96 is no better (+0.002).The width-96 interval spans zero.Source: F. Student Capacity · p. 12
  • At widths 16, 48 and 96, kdA4 trails the directly trained student of equal size by 0.030, 0.026 and 0.017 energy AUROC, respectively.The passage says the small-student detection result holds across a sixfold range of student width.Source: F. Student Capacity · p. 12

G. Distilling the Representation Instead of the Logits

Representation matching preserved closed-set classification but did not transfer the teacher’s detection advantage. More broadly, teacher-specific score patterns appeared only at the conventional temperature, while shortcut reliance tracked model size rather than distillation.

G. Distilling the Representation Instead of the Logits

  • kdF gained +0.014 Mahalanobis AUROC over direct (−0.010 to +0.033) and +0.018 under feature k-NN (+0.001 to +0.028), versus the teacher’s +0.073.Against kdM0, kdF was 0.064 lower on Mahalanobis AUROC and 0.009 lower on feature k-NN, but +0.033 on energy and +0.030 on MSP.Source: G. Distilling the Representation… · p. 12
  • Similarity-preserving distillation matches batch-wise pairwise feature similarities, but leaves rotations and rescalings invariant while the Mahalanobis detector reads absolute class arrangement.Source: G. Distilling the Representation… · p. 12
  • The feature-distillation probe fixed β = 100 without tuning, so its null result cannot distinguish an ineffective objective from an unsuitable weight.The authors recommend testing a tuned weight across more than one start date.Source: G. Distilling the Representation… · p. 12

VI. DISCUSSION

  • Against a directly trained, temperature-scaled student, distillation improved no measured unknown-detection outcome; against label smoothing, it traded detection for calibration.Source: What transfers, and what does not · p. 12
  • At T = 1, teacher-specific signal did not survive; at T = 4, it did, at a cost of 1.6 macro-F1 points for Teacher A and 1.7 for Teacher B.Source: What transfers, and what does not · pp. 12–13
  • Over 35 weeks, the 11.3M-parameter ensemble lost unknown-detection AUROC faster than the 101k student, which led under energy in two of three replicates.The ensemble aged at its members’ rate under energy; the authors suggest that higher capacity may fit transient training-period structure that later decays, but do not establish the cause.Source: The gap closes instead of opening · p. 13
  • Below shortcut reliability 1, both 101k students relied on the planted feature about twice as much as the 2.3M teacher, regardless of distillation.The shortcut-reliant teacher also passed over-confidence to a student that never saw the feature, while that student’s unknown detection improved slightly.Source: Shortcuts follow capacity, not training · p. 13
  • Label smoothing led the grid under Mahalanobis and came within 0.001 of EnDD under feature k-NN, a result the authors distinguish from prior settings.Their setting used 102 fine-grained services, flow-level features, tuned smoothing strength 0.05, and TLS-service unknowns; they leave geometric analysis for future work.Source: Where we disagree with the literature · p. 13

VII. CONCLUSION AND FUTURE WORK

The conclusion is that logit distillation transfers teacher-specific habits, but does not improve registered logit-score detection over direct training. The authors identify feature-based distillation, teacher scale, and unexplained ageing as open questions.

VII. CONCLUSION AND FUTURE WORK

  • Under the registered logit-based scores, a directly trained small model matched the distilled models on detection, calibration, and shortcut robustness, and kept those properties at least as well over a year of drift.Source: §VII · p. 14
  • Future work should tune feature-distillation weights across start dates, test objectives that constrain absolute feature arrangement, examine larger teachers, and explain ageing.The current feature-based attempt used one untuned weight on one start date; the study also does not explain why the student aged more slowly.Source: §VII · p. 14
  • The authors released code, the frozen service split, per-window metric tables, and analysis; trained weights and derived scores are deposited separately.Source: REPRODUCIBILITY · p. 14

Related papers

Loading 2609.31141v1…