Source-linked AI summary
Beyond Measurement Metrics: A Human-Centered Framework for Semantic Validation of Network Traffic Classification
Igor Cherepanov, David Sessler, Alex Ulmer, Thorsten May, Jörn Kohlhammer
TL;DR
Network traffic classifiers can achieve high predictive performance without learning semantically meaningful patterns, because conventional evaluation may miss shortcuts and unintended correlations. This paper adapts a human-centered knowledge-generation framework that combines data, models, explainability, visualization, and expert reasoning for iterative semantic validation. The resulting framework offers structured guidance for developing models that are accurate, robust, and trustworthy within the supported evaluation scope.
Problem
Conventional performance metrics cannot determine whether traffic-classification models rely on meaningful traffic patterns or dataset-specific shortcut features.
Method
The paper adapts the knowledge generation model for visual analytics by integrating data, ML models, explainability, visualization, stakeholder objectives, and expert reasoning into an iterative workflow.
Results
The resulting framework connects explanation-based analysis with model development and semantic validation to support traffic-classification models that are accurate, transparent, robust, and correct for the right reasons.
Takeaways & Limitations
Explanation findings can be fed back into data, feature, model, preprocessing, and evaluation decisions, while accumulated knowledge can improve dataset transparency and reproducibility.
Takeaways & Limitations
The framework's validation scope is constrained by dataset artifacts and sampling-induced shortcuts, including information leakage from packet-level splits that place packets from the same flow in training and test sets.
Abstract
from arXiv · showhide
Machine learning (ML) has become the dominant approach for network traffic classification, achieving very high predictive performance. However, a model is only valuable if it learns semantically meaningful and trustworthy patterns rather than exploiting spurious correlations. Conventional evaluation practices predominantly assess predictive performance. Consequently, whether the model relies on semantically meaningful patterns remains unknown. To address these challenges, we adapt the knowledge generation framework for network traffic classification. The adapted framework combines data, ML models, explainability, visualization, and expert reasoning to support the iterative exploration, verification, and refinement of model behavior and data preprocessing. The framework is grounded in findings from the literature, benchmark dataset analyses, practical experience with XAI-based traffic classification, and expert feedback, providing practical guidance for semantic model validation. By complementing predictive performance with semantic validation and human expertise, the proposed framework supports the development of network traffic classification models that are not only accurate but also robust and trustworthy.
1 Introduction
Network traffic classification has achieved strong predictive performance, but conventional evaluation does not reveal whether models learn meaningful traffic characteristics or exploit unintended correlations. The proposed human-centered framework integrates explainability and expert reasoning into iterative model development and semantic validation.
- 1 Introduction: High predictive accuracy can conceal reliance on dataset-specific shortcuts, spurious correlations, protocol artifacts, or biases that degrade performance in evolving network environments.These unintended dependencies may remain hidden during development and cause substantial deployment degradation as traffic characteristics and data distributions change.
- 1 Introduction: Semantic validation evaluates whether a model's decision strategy is based on meaningful traffic characteristics rather than only measuring predictive performance.This addresses the limited insight conventional traffic-classification pipelines provide into what models have actually learned.
- 1 Introduction: Explainability is positioned as a mechanism for continuously improving data, feature representations, models, and evaluation strategies.This broadens explainability beyond post hoc interpretation toward iterative refinement of the classification workflow.
- 1 Introduction: The framework adapts the knowledge generation model for visual analytics by incorporating explainability, stakeholder-specific objectives, and explanation-based evaluation.It extends performance-centric evaluation toward models that are accurate, transparent, robust, and correct for the right reasons.
- 1 Introduction: The framework organizes users, tasks, goals, explanation requirements, evaluation strategies, and semantic validation criteria into a structured design space for trustworthy XAI.It also derives design recommendations that incorporate domain experts into XAI evaluation.
2 Related Work
Related work shows that increasingly complex traffic-classification models have made explainability important, while existing approaches still need more human-centered strategies and stronger integration with expert reasoning. Visual analytics provides a foundation for iterative exploration, verification, and knowledge generation around model behavior and data.
- 2 Related Work: Earlier traffic-classification approaches relied on ports and deep packet inspection, but dynamic or shared ports and encryption reduced the reliability of these methods.These limitations contributed to the shift toward statistical and learned approaches.
- 2 Related Work: Deep learning shifted traffic classification toward representation learning, with architectures including CNNs, RNNs, LSTMs, autoencoders, GNNs, and transformers.As model complexity increased, XAI emerged to investigate and interpret their decision-making processes.
- 2 Related Work: Existing XAI traffic-analysis research uses explanations to understand predictions, debug classifiers, assess trustworthiness, and investigate behavior under evolving traffic scenarios.These applications demonstrate value beyond simply increasing black-box transparency.
- 2 Related Work: Encrypted traffic classification has used one-dimensional CNNs that learn representations directly from raw packet bytes, with CAM-based extensions visualizing packet regions contributing to predictions.This line of work connects high-performing traffic classifiers with explanation methods adapted from computer vision.
- 2 Related Work: Interactive visual analysis combined with XAI can help domain experts detect, interpret, and validate distributional shifts in deployed traffic-classification settings.This prior work addresses data drift and out-of-distribution detection as sources of degraded predictive performance.
- 2 Related Work: Explainability alone is insufficient for informed decision-making because domain experts must validate, contextualize, and reason about model and data insights.The literature therefore emphasizes sense-making and active expert involvement.
- 2 Related Work: Visual analytics frameworks combine automated analysis with human reasoning through iterative exploration, hypothesis generation, verification, and knowledge acquisition.The knowledge generation model is presented as a suitable foundation because it explicitly describes interaction between computational analysis and human reasoning.
- 2 Related Work: XAI surveys identify a need for more human-centered explanation strategies, support for different stakeholder groups, and tighter integration with the ML lifecycle.This motivates connecting explanation use to broader analytical and development processes.
3 Limitations of Traffic Classification Workflows
Traffic classification workflows are limited by dataset artifacts, information leakage, and performance-centric evaluation that can conceal shortcut learning. The evidence supports combining structural dataset analysis, masking experiments, explainability, visualization, and expert reasoning to validate whether models learn meaningful traffic characteristics.
- 3.1 Dataset-Related Limitations: 87% of all flow identifiers are unique to a single application on average, giving transport-layer identifiers substantial discriminative information before payload inspection.Unique flows and unambiguous packets quantify how readily application classes can be separated from (source port, destination port, protocol) alone.
- 3.1 Dataset-Related Limitations: Transport-layer shortcuts can yield high predictive performance without learning meaningful characteristics from encrypted application-layer data.The dataset analysis indicates that application-specific port and protocol combinations can encode the target class.
- 3.1 Dataset-Related Limitations: Flow-level and packet-level separability differ across applications, with some applications having mostly unique flow identifiers but many packets in a few shared high-volume flows.These structural differences affect how training and test sets are composed.
- 3.1 Dataset-Related Limitations: Random packet-level train-test splits can place packets from the same flow in both sets, causing information leakage and overly optimistic performance estimates.Packets within a flow share communication characteristics, so packet-level randomization may fail to separate independent examples.
- 3.1 Dataset-Related Limitations: The Sankey visualization exposes how unique flow signatures connect source-port bands, application classes, and destination ports across the dataset.Ribbon width represents distinct flow signatures, while source ports are grouped into IANA ranges and less frequent destination ports are aggregated.
- 3.1 Dataset-Related Limitations: Data-collection conditions can introduce temporal, environmental, and configuration-dependent shortcuts that conventional preprocessing may not reveal.Identified sources include collection order, network configuration, operating-system state, and background traffic; systematic analysis or explanation-based inspection is typically required.
- 3.2 Limitations of Performance-Centric Evaluation: Performance metrics alone cannot establish whether predictions rely on semantically meaningful traffic patterns or dataset-specific shortcut features.Explainability reveals that only the initial encrypted-payload bytes consistently contribute to predictions, while most payload bytes receive negligible attribution.
4 Specific Workflow for Network Traffic Classification
The workflow adapts the knowledge generation model into a human-centered process for semantically validating network traffic classification. It connects data, models, explanations, visual analytics, stakeholder objectives, and expert reasoning through iterative exploration and evaluation.
- Framework foundation: The adapted framework incorporates XAI into the computational pipeline and assigns multiple expert roles with stakeholder-specific objectives.It supports explanation-based exploration, verification, and knowledge generation.
- Data: The data workflow removes or masks dataset artifacts and preserves flow-level independence across training, validation, and test partitions.These measures target shortcut learning, information leakage, overly optimistic estimates, and unreliable generalization.
- Visualization: Visualizations connect automated analysis with human reasoning by supporting exploration, interpretation, validation, and interactive comparison of local and aggregated explanations.They should present compact, contextual representations aligned with analytical tasks and networking semantics.
- Stakeholders: Stakeholder objectives shape explanation requirements: practitioners use explanations to debug and assess models, while domain experts use them to validate predictions in application context.The framework therefore treats user goals as part of explanation design and evaluation.
- Model and explanation evaluation: Model assessment combines appropriate class-sensitive metrics with cross-validation, repeated training, robustness testing, and explanation-quality evaluation.Macro F1-score, precision, and recall are emphasized for imbalanced datasets, while confusion matrices reveal frequently confused classes.
- Knowledge generation: Knowledge generated through the workflow can guide future dataset construction, feature engineering, explanation design, visualization, evaluation, and model development.The resulting loop supports continuous improvement of explainable traffic-classification workflows.
5 Discussion and Future Work
The framework supports reuse of validated knowledge across studies while enabling consistent comparison of explanation methods and design choices. Its main practical limitation is the effort required for full workflow integration, motivating incremental adoption.
- Knowledge reuse: Accumulated findings can improve benchmark transparency, document information leakage, motivate standardized preprocessing and partitioning, and make design rationales reproducible.The framework feeds knowledge from explanation-based analysis back into earlier development stages.
- Future work: Separating explanation generation from analysis enables consistent evaluation of different XAI methods according to technical characteristics and practical utility.It also supports investigation of aggregation strategies for local explanations across application classes or similar samples.
- Limitation: Integrating explanation generation, interactive analysis, and expert verification may be infeasible in some settings because of time, computational-resource, or domain-expertise constraints.The framework therefore supports incremental adoption, with individual components still enabling documented feedback into later iterations.
6 Conclusion
The conclusion positions semantic validation as a complement to performance-centric evaluation and presents a human-centered framework for trustworthy traffic-classification development. The framework integrates technical and human elements into an iterative workflow with practical design guidance.
- Conclusion: The framework integrates data, ML, explainability, visualization, and expert reasoning into an iterative workflow for semantic model validation.It was motivated by literature, empirical analyses, practical XAI experience, and expert feedback.
- Conclusion: The resulting design recommendations provide practical guidance for trustworthy development and evaluation of network traffic classification models.They translate the framework into actionable support for model development and assessment.
7 Declarations
The declarations report no competing interests, no animal involvement, participant willingness to share information, and a Creative Commons Attribution 4.0 license.
- Declarations: The authors report no relevant financial or non-financial competing interests.
- Declarations: The research involved no animals.
- Declarations: Participants took part freely and willingly shared information relevant to the paper’s purpose and objectives.
- Declarations: The article is licensed under Creative Commons Attribution 4.0, permitting reuse with appropriate attribution and indication of changes.