PaperlayerView exact revision on arXiv

Source-linked AI summary

SoK: Where Do Flow Labels Come From? Auditing Label Provenance in Encrypted Traffic Benchmarks

Sizhe Huang, Shujie Yang

arXiv:2609.02140v1cs.NIcs.LG

TL;DR

Encrypted-traffic benchmarks depend on labels that must hold for individual flows, yet label provenance between endpoint evidence and published records is rarely exposed. The paper systematizes that provenance across 14 benchmark entries and measures the consequences of coarse inheritance and overstrict filtering. It finds opposing label-side risks, with strict side-channel ceilings of 0.56 to 0.76 and discarded context improving macro accuracy from 0.44 to 0.65.

  • Problem

    Benchmark labels rarely record their evidence source, evidence unit, assignment operator, or intended task, leaving the relationship between endpoint evidence and published flows unclear.

  • Method

    The paper introduces the ten-field Label Provenance Record and audits 14 public benchmark entries using artifact-computable measures of label conflict and representation-relative performance limits.

  • Results

    Across audited benchmarks, coarse inheritance and overstrict filtering fail in opposite directions; strict side-channel ceilings range from 0.56 to 0.76, while discarded context raises macro accuracy from 0.44 to 0.65.

  • Takeaways & Limitations

    Benchmark builders and users should align published units and claimed tasks, report boundary shares, and compute ceilings using each evaluation’s declared feature representation.

  • Takeaways & Limitations

    The ceiling is representation-relative and bounded by what the artifact retains; payload-truncated artifacts may not support finer-key reconstruction.

Abstract

from arXiv · show

Encrypted traffic classification infers semantics beyond the flow record from transport-layer observables, and supervised training rests on labels that hold for the individual flow they are attached to. Recent systematizations scrutinize model in- puts and data splits; we systematize the complementary label side. Across 14 audited benchmark entries, we identify two recurring label-side strategies: coarse inheritance, which risks labelling flows the evidence does not cover, and overstrict filtering, which keeps only self-attesting flows and risks dis- carding relevant ones. No audited entry exposes a countable pre-selection population, and the task objects downstream papers attach to the same labels disagree with the recovered record in 8 of 23 referenced cells. Under strict side-channel features we derive a representation-relative ceiling on bal- anced accuracy for any classifier restricted to those features: on the public benchmarks that inherit, it ranges from 0.56 to 0.76. On the filtering side, only 24.95% of connections in our fully captured corpus carry an observable SNI of their own; yet the discarded connections raise macro accuracy from 0.44 to 0.65 through same-run co-occurrence features. We end with recommendations for benchmark builders and users.

1 Introduction

This paper audits how encrypted-traffic benchmarks derive labels, focusing on the gap between endpoint evidence and published flow records. Across 14 entries, it finds opposing label-side risks and quantifies their consequences before model training.

  • Endpoint evidence lives at the application or endpoint, while benchmark artifacts usually publish transport-layer flow records; this vantage-point gap is the paper’s object.
  • Recent systematizations examine model inputs and data splits, but labels can still limit classifiers when indistinguishable observations carry different labels.
  • Across 14 audited benchmark entries, coarse inheritance risks labeling flows beyond their evidence, whereas overstrict filtering risks discarding flows carrying label information.
  • No audited entry exposes a countable pre-selection population, and downstream task interpretations disagree with the recovered record in 8 of 23 referenced cells.
  • On inheriting public benchmarks, the representation-relative balanced-accuracy ceiling ranges from 0.56 to 0.76 under strict side-channel features.
  • Only 24.95% of paired-corpus connections carry their own observable SNI, while adding discarded context raises macro accuracy from 0.44 to 0.65.

2 Taxonomy and Classification

The paper frames label provenance as the path from evidence to published samples, formalized through evidence layers, units, and assignment operators. It distinguishes connection alignment from coarse inheritance and overstrict filtering, which respectively risk over-inclusion and over-exclusion.

  • Label provenance and task meaning: CSTNET-TLS1.3 labels are server domains derived from SNI, although downstream papers interpret them as applications, websites, or peer server domains.These interpretations imply different tasks for models trained on the same released labels.
  • Label Provenance Record: The Label Provenance Record represents provenance with ten fields covering evidence, layer, observation surface, units, assignment, lineage, and source tier.Its pipeline view records how evidence at U_source reaches the published unit U_sample through operator A.
  • Assignment operators: Connection alignment assigns evidence to samples at the same unit, whereas coarse inheritance copies a session-, run-, or window-level label onto contained flows.Derived samples and parent-label reuse propagate upstream choices rather than re-deriving evidence.
  • Assignment operators: The evidence layer and source unit constrain the assignment operator: connection-level evidence can align flow-by-flow, while coarser evidence requires inheritance.Overstrict filtering instead requires path-visible evidence and keeps only flows carrying an observable identifier of their own.
  • Opposite failure modes: Coarse inheritance risks labelling flows unsupported at the flow unit, while overstrict filtering risks discarding relevant flows that lack self-attesting identifiers.In the paired corpus, 87.0% of boundary flows were 1–4 packet connections, and only 24.95% of connections carried an observable SNI of their own.
  • Evidence availability: Endpoint-privilege evidence for generating applications is usually unavailable per flow, whereas path-visible evidence such as SNI can support filtering but not necessarily complete coverage.Publication processing can further remove payloads, identity fields, and raw bytes containing label evidence.

3 Systematization

The audit applies the provenance framework to 14 versioned benchmark entries and compares recovered semantic objects with downstream claimed tasks. It finds recurrent documentation, task-alignment, and surviving-channel problems, while showing that assignment operators separate reachable label ceilings.

  • Corpus and records: The audit covers 14 versioned entries, comprising 13 unique source papers, and classifies them into six inheriting, six aligned, one reusing, and one derived entry.The Nbrhd. annotation records whether released rows support rebuilding a flow’s neighbourhood.
  • Task–record alignment: The claimed object matches the recovered semantic object in 6 of 23 referenced cells, matches only at a coarser unit in 9, and differs in 8.The P marker appears on nine of the 14 datasets and denotes undeclared filtering in the evaluated population.
  • Surviving channels: Surviving label channels can act as shortcuts: SNI nearly saturates CipherSpectrum, while MAC reaches 0.99996 on USTC-TFC2016 and 1.0 on MIRAGE-2019.MIRAGE’s 0.0659 balanced accuracy is already the field’s full capacity because only two MAC addresses exist.
  • Cross-cutting patterns: For the public benchmarks with computable ceilings, aligned entries reach at least 0.8956 balanced accuracy while inheriting entries reach at most 0.7589.The cardinality-controlled comparison remains separated between MIRAGE-2019 at 0.8956 and USTC-TFC2016 at 0.7589.
  • Auditability gaps: Provenance is verifiable from released artifacts for 11 of 14 entries, from official material or the original paper for two, and not confirmable for one.The audit therefore concludes that documentation alone does not suffice for provenance recovery.
  • Auditability gaps: No audited entry exposes a countable pre-selection population, preventing recomputation of exclusion rates from the audited materials.The paired corpus is the only audited corpus retaining a countable pre-selection population.
  • Security consequence: D12’s T-MAL cell uses benign CipherSpectrum browsing captures as the unknown class in an open-set malicious-traffic detector.The paper identifies the resulting failure mode as directly deployable.

4 Measuring Label Provenance

The paper measures label provenance through representation-relative conflict, paired relabelling, and analysis of flows excluded by self-attesting identifiers. These instruments expose opposite risks: inheritance labels uncovered flows, while filtering discards flows carrying recoverable information.

  • 4.1 Measuring Label Provenance: A primary key groups flows by transport protocol, true packet count, signed L4 payload lengths, and a mask, making equivalent flows indistinguishable to the declared representation.The key excludes timing and identity fields; its equivalence classes define the observation relation.
  • 4.1 Measuring Label Provenance: The representation-relative ceiling bounds balanced accuracy before training, and any score above it cannot come from the declared input alone.The ceiling is diagnostic rather than accusatory and is computed on the declared key and evaluated rows.
  • 4.2 Fixed-row relabelling and the paired corpus: On CipherSpectrum’s fixed 123,000 sessions, access-level inheritance and per-connection SNI evidence agree for first-party sessions but diverge when the peer is not the accessed site.For non-first-party sessions, normalized conflict rate moves from ρ = 0.4197 to 0.0169, with ∆ρ = 0.4028.
  • 4.2 Where inherited labels land: Under run-target inheritance, 87.0% of all conflict occurs among 1–4-packet boundary flows, where labels are attached without per-connection evidence.The in-bin conflict rate falls from 0.8889 for 1–4 packets to 0.0009 for 100+ packets.
  • 4.3 Information in the excluded neighbourhood: Only 24.95% of paired-corpus connections carry their own SNI, yet excluded flows reach 0.1424 balanced accuracy and rise to 0.4508 when with-SNI neighbours are added.The paired corpus retains the original observed superset, enabling direct inspection of the excluded neighbourhood.
  • 4.4 Representation ladder: Refining side-channel features removes only 2.40 percentage points of boundary share, while high-cardinality fields can raise the ceiling without demonstrating generalizable signal.Probe gains also show that the ceiling is not a performance predictor.

5 Discussion and applicability

The paper recommends documenting label provenance, observation populations, exclusions, retained identifier channels, and claimed-task semantics. It also emphasizes release-time diagnostics and limitations on generalizing filtering-side results across traffic compositions and collection settings.

  • Scope and limitations: The filtering-side results come from one corpus and establish a mechanism rather than universal constants.The paired corpus also uses one browser, one network vantage, and a fixed site list; self-attesting shares vary with ECH deployment.
  • Guidelines for benchmark builders: Benchmark builders should publish the Label Provenance Record, including evidence source, unit, assignment operator, and semantic object.The LPR is intended to make label provenance explicit alongside the released artifact.
  • Guidelines for benchmark builders: Benchmark builders should declare the observation population and exclusion count before filtering, because no audited entry permits recomputing either.Publishing an unlabelled superset or verifiable labelled subset can preserve this auditability.
  • Guidelines for benchmark builders: When labels are inherited from a coarser context, report the coarse unit or the boundary share at the published unit.87.0% of the boundary mass lies in 1–4-packet connections, where inheritance is most likely to exceed evidence coverage.
  • Guidelines for benchmark builders: Identifier channels such as SNI and MAC should be documented and treated as label evidence because deterministic rules can capture most of their capacity when retained.Retention and predictiveness vary independently across artifacts.
  • Guidelines for benchmark users: Users should compute the representation-relative ceiling on the same evaluation rows and verify the claimed-task cell before citing a benchmark.A score above the ceiling indicates information outside the declared channel, while claimed objects disagree with the recovered record in 8 of 23 referenced cells.
  • Future directions: Release-time checks can flag inheritance-heavy or shortcut-heavy artifacts before publication, while semi-automated LPR extraction could extend audits beyond 14 entries.The paper also predicts that standardized padded encrypted DNS will reduce length-channel and co-occurrence predictiveness.

6 Related work

The paper complements input-side and evaluation-focused systematizations by auditing how labels are constructed and propagated. It distinguishes evidence coverage from label correctness and positions the LPR as provenance documentation specialized to traffic benchmarks.

  • Position relative to prior systematizations: Prior systematizations close strong input channels and correct evaluation artifacts, whereas this paper studies the label-side residue.The related work includes analyses of identifiers, representation learning, redundant samples, and split leakage.
  • Ground truth and provenance: Endpoint instrumentation can obtain per-flow ground truth, but prior ground-truth work does not track the evidence unit or assignment operator reaching published samples.The paper focuses on what evidence is exercised at the per-flow level in released artifacts.
  • Dataset documentation: The LPR specializes dataset provenance to record the unit where label evidence holds and the operator carrying it to each published sample.This addresses a question generic benchmark schemas do not ask, while Gap 1 concerns unrecoverable excluded populations.
  • Label conflict versus label noise: Unlike label-noise measures, the paper’s conflict mass can arise with perfectly correct labels when evidence is attached at a coarse representation.The distinction is evidence coverage rather than an error rate.

7 Conclusion

The paper concludes that benchmark labels derive from an underreported pipeline of evidence, unit, and assignment, whose two mainstream constructions fail in opposite directions. Both failure modes are measurable before model training, and scores above a declared side-channel ceiling indicate information outside that channel.

  • Conclusion: Benchmark labels come from a pipeline of evidence, unit, and assignment that is almost never reported.The paper identifies two mainstream constructions that fail in opposite directions.
  • Conclusion: Both label-construction failures are measurable before any model is trained.The paper frames the resulting diagnostics as checks on benchmark artifacts rather than model quality alone.
  • Conclusion: Scores are incomparable across benchmarks when their label provenance and representations differ.A score above a declared side-channel ceiling signals information from outside that channel, not a better model.

Ethical Considerations

The study audits public benchmark artifacts and a paired browsing corpus collected through scripted Chromium runs over 31 websites. No human participants were involved, and the released corpus is de-identified and redistributable in full.

  • Study conduct: The paired browsing corpus was collected through scripted Chromium runs over 31 websites without human participants.The broader work also audits publicly released benchmark artifacts and their source papers.
  • Data protection: The released corpus strips session cookies and credential-bearing or identifying material before release.The de-identified raw capture is described as redistributable in full.

Open Science

The paper releases the paired browsing corpus, its manifests, and the frozen evidence worksheet and scripts used for the audit computations.

  • The paired browsing corpus and audit materials are released after de-identification.The release includes captured connections, the evidence worksheet, and deterministic scripts for computing ceilings, chain decomposition, and channel attribution.

Appendix A – Dataset selection

Appendix A documents the audit corpus, its inclusion rules, and the recoverability of pre-selection populations across benchmark entries. No audited entry exposes a full countable population before filtering or processing.

  • Inclusion rules: The audit includes dataset versions with at least three confirmed using papers, a public labelled artifact, and papers within the screening window.D13 is retained as a flagged exception because it is a technical report without a published artifact.
  • Pre-selection population protocol: No audited entry exposes a countable pre-selection population.Collection-side entries omit captured-population counts, derived entries inherit upstream uncountability, and D13 has no published artifact.
  • Paired corpus: 888 scripted Chromium runs over 31 public targets form the paired corpus, combining raw traffic, browser request logs, and per-connection SNI.The corpus is the only one in the audit containing the original observed superset and both access-level and connection-level evidence.
  • Per-entry evidence: D1 and D2 have flow counts only after processing or post-labelling, while D3 reports 3.71 GB of pcaps and 752,040 generated records.These quantities do not constitute capture-population counts.
  • Per-entry evidence: D4 reports 276,871 released biflow records, but unattributable captured biflows are never counted.D7 reports 46,372 released flows against about 15 GB of pre-training traffic, and D9 documents exclusions without stating the count entering the chain.
  • Per-entry evidence: D12 narrows 2,000 Cloudflare Radar domains to 132 domains and 660 URLs, then releases exactly 120,000 sessions without counting the pre-trimming population.D10 likewise cannot recover its pre-sampling population, while D14 states exclusions in bytes and app counts rather than flows.

Appendix B – Claimed-task matrix and quoting protocol

Appendix B specifies a reproducible codebook for comparing claimed task objects with label semantics and sample units, supported by quoted downstream-paper evidence.

  • Quoting protocol: The claimtrace codebook orders each audited claim from dataset version and label semantics through evaluation scope, result, claim, and contract status.This makes the classification traceable rather than a verdict reverse-engineered from outcomes.
  • Claimed-task matrix: T-APP, T-SITE, T-SERVER, T-ACT, T-ENCAP, and T-MAL denote application, site, server, activity, encapsulation, and benign-versus-malicious task objects.These object types provide the vocabulary for the claimed-task matrix.
  • Cell markers: The matrix marks semantic and unit agreement as ✓, coarser-unit agreement as ◦, mismatch as ×, filtered populations as P, and unobserved claims as −.P can be combined with the other markers because downstream filtering may be introduced by using papers.
  • Evidence sources: Evidence is tiered from released artifacts and official sources through downstream papers, audit inference, and unrecoverable public evidence.The tiers are A through E, respectively.
  • Worked examples: Worked examples show application, website, server-domain, service, and malicious-detection readings can diverge for the same released labels.CipherSpectrum includes a usage-level substitution where benign browsing is injected as the unknown class despite no malicious samples.
  • Results: Across 23 referenced cells, 6 match, 9 are defined only at a coarser unit, and 8 differ from the recovered semantic object.The P marker attaches to nine datasets, while one AN-Net cell is unspecified rather than mismatched.

Appendix C – Key-length convergence, vocabulary control, and the worked example

Appendix C tests key-length stability, controls vocabulary differences, and applies the ceiling diagnostic to published evaluations. The analyses separate representation effects from row-set and feature choices.

  • Vocabulary control: The vocabulary control maps 85 access-level classes to registrable-domain labels while avoiding session-level SNI/access co-occurrence statistics.This isolates vocabulary differences between the inheritance and SNI-derived label arms.
  • Mixture-weight decomposition: Table 18 decomposes CipherSpectrum accuracy into per-probe components behind the mixture-weight scan.The decomposition uses N=123,000, with POS=122,359 and BND=641 at the key-mode baseline.
  • Key-length convergence: The key-length sweep runs from 5 to 50 on nine computable rows, with the k=20 slice reproducing Table 17 and curves generally flat by k=30.CESNET-QUIC22 and MIRAGE-2019 plateau at source packet limits rather than at the key length.
  • Worked example: The ceiling diagnostic compares three ISCXVPN2016 evaluations at matching metrics and the coarsest rung containing each declared input.Ceilings use the full released population, whereas the papers evaluate preprocessed subsets, so row-set mismatch cannot be excluded.
  • Declared inputs: ET-BERT removes Ethernet and IP headers and protocol ports while retaining datagram bytes; Yu et al. retain header fields and sequence/acknowledgement tokens.Sugar uses protocol headers only, with its ablation identifying IP addresses as the largest contribution.
  • Worked example: On Yu et al.’s rebuilt 15-class row set, the R3-key accuracy ceiling is 0.9038 versus reported accuracy 0.9874, while the full-population flow-level ceiling is 0.4638.The row set explains most of the excess, and the residual 0.084 is consistent with sequence/acknowledgement tokens acting as implicit flow identifiers under per-packet random splits.

Appendix D – Notation

Appendix D introduces the symbols used in Section 4.1 and the result tables, alongside tables documenting quoted task statements and their page references.

  • Section 4.1 and the result tables use the symbols introduced in the appendix.
  • Table 19 presents the paper’s table of symbols.
  • The ledger identifies quoted task statements and their page references for each cell.
  • Tables 15 and 16 provide a per-cell quote ledger for the claimed-task matrix in Table 2.The ledger continues across two parts and records verbatim quotations with PDF page numbers.
Loading 2609.02140v1…