PaperlayerView exact revision on arXiv

Source-linked AI summary

Do Flow-Generation Tools Matter? Unsupervised Characterization of MQTT Traffic Across Public Datasets

Mohammed Rashed, Carlos García-Rubio, Celeste Campo

arXiv:2610.29774v1cs.NIcs.CR

TL;DR

MQTT flow analysis has limited evidence on how tool-specific flow construction shapes traffic representations, despite flows being constructed rather than directly observed in packet captures. This paper compares flows generated from four public MQTT datasets using three tools and unsupervised clustering. Tool choice affects flow populations and clustering, though no tool consistently dominates.

  • Problem

    Flow records are constructed through tool-specific packet-grouping and termination rules, but the effect of flow-generation choice on MQTT characterization is not well established.

    Source: §1 · p. 2

  • Method

    The study compares Zeek-flowmeter, Argus, and NTLFlowLyzer on MQTT flows from four public datasets using K-means, DBSCAN, and HDBSCAN with and without PCA.

    Source: §7 · p. 19

  • Results

    Flow populations and clustering behavior vary by tool; Zeek-flowmeter is strongest in several settings, but no tool consistently dominates across datasets, algorithms, and metrics.

    Source: §1 · p. 2, §7 · pp. 19–20

  • Takeaways & Limitations

    MQTT flow analysis should account for dataset, flow-generation tool, preprocessing, clustering algorithm, and evaluation metric as interconnected factors.

    Source: §1 · p. 2, §7 · p. 20

    The study covers unencrypted MQTT on port 1883, four public datasets, and three tools used with default flow-construction settings.

    Source: §6.1 · p. 19

Abstract

from arXiv · show

Message Queue Telemetry Transport (MQTT) is widely used in IoT environments due to its lightweight publish-subscribe design and suitability for resource-constrained networks. While current MQTT security studies mainly focus on detecting attacks and optimizing classification performance, this work takes a different perspective by presenting a cross-dataset, tool-aware characterization of MQTT traffic at the flow level. This perspective is motivated by the fact that flow-generation is tool-dependent: flows are not directly observed in packet captures, but are constructed through tool-specific decisions about packet grouping, flow termination, and feature export. Using packet captures from 4 public MQTT datasets, we generated over 380K flows with 3 open-access flow-generation tools: Zeek-flowmeter, Argus, and NTLFlowLyzer. We analyzed the tool-specific flows using K-means, DBSCAN, and HDBSCAN, with and without PCA-based dimensionality reduction. We measured clustering quality using 2 metrics: (1) Silhouette score to assess label-independent cluster compactness and separation, and (2) Adjusted Rand Index to measure agreement with available labels. Our results show that the choice of flow-generation tool affects the resulting flow population, feature-space structure, and clustering behavior. Zeek-flowmeter achieves the best performance in several settings, particularly with K-means and PCA reduction to 10 components. Yet, no single tool consistently dominates across all datasets, algorithms, and evaluation metrics. These findings show that flow-level MQTT analysis should treat the dataset, flow-generation tool, preprocessing choices, clustering algorithm, and evaluation metric as interconnected experimental factors.

1. Introduction

MQTT flow-level characterization remains underexplored, and flow extraction can vary by tool rather than simply reflecting packets as observed. This study compares tool-generated MQTT flows across datasets using unsupervised clustering and finds that tool choice affects clustering behavior without one tool consistently leading.

1. Introduction

  • MQTT flow-level characterization is underexplored relative to packet-level studies and supervised intrusion detection, despite flows capturing temporal, directional, and volume-based properties.The paper aims to understand how flow-generation tools influence these representations and the insights derived from them.Source: §1 · p. 2
  • A flow is constructed by a tool that determines packet grouping, flow start and end, directionality, and exported features, so the same PCAP can yield different flow populations.Public MQTT datasets also differ in scale, realism, and attack diversity, making cross-dataset analysis challenging.Source: §1 · p. 2
  • Earlier work on HTTP- and DNS-based botnet traffic found that flow-generator choice influenced classifier performance, motivating consideration of dataset, algorithm, and tool together.Prior MQTT studies also report that flow-based representations can discriminate attacks better than packet-level representations.Source: §1 · p. 2
  • The study compares how three flow-generation tools shape MQTT traffic representations across four public datasets, using K-means, DBSCAN, and HDBSCAN with and without PCA.The analysis evaluates cluster compactness and separation with Silhouette score, and agreement with original labels using ARI.Source: §1 · p. 2
  • Zeek-flowmeter performs strongest in several settings, especially with K-means and PCA reduction to 10 dimensions, but no tool dominates across all datasets, algorithms, and metrics.The findings identify flow analysis as dependent on the dataset, algorithm, and flow-generation tool.Source: §1 · p. 2

2. Background

MQTT uses a lightweight client–broker publish–subscribe model, while network flows summarize communication at the network and transport layers. Flow records and their features depend on tool-specific construction and termination rules.

2. Background

  • MQTT clients publish to topics through a broker, which forwards messages to subscribers; QoS, retained messages, and Last Will and Testament support reliability under unstable connectivity.MQTT can also be vulnerable to protocol-specific attacks and inherited TCP-layer attacks, which may appear in connection duration, timing, directionality, or volume.Source: §2 · pp. 3–4

2.1. Network Flow

  • Flow records group packets sharing attributes, often a five-tuple, and tools may differ in whether they split long-lived flows or terminate them after inactivity.The study treats exported records as network- and transport-layer connection summaries, not direct representations of MQTT sessions or publish/subscribe interactions.Source: §2.1 · p. 4

2.2. Flow-Generation Tools

  • The study compares Argus, NTLFlowLyzer, and Zeek-flowmeter because they represent transaction-oriented reporting, bidirectional statistical features, and Zeek connection-state logic.The tools are open-source and documented for reproducible PCAP-to-flow conversion.Source: §2.2 · p. 4

2.2.1. Argus

  • Argus reports flow activity status records and periodically exports active long-lived flows when new activity occurs, with a default interval of 60 seconds.Each record contains timestamps for the first and last packet observed during the observation period.Source: §2.2.1 · p. 4

2.2.2. NTLFlowLyzer

  • NTLFlowLyzer creates bidirectional flows, assigning forward direction from the first packet and terminating TCP flows on teardown, maximum duration, or inactivity.Its default maximum duration is 120,000 seconds and its default inactivity timeout is 5,000 seconds.Source: §2.2.2 · p. 5

2.2.3. Zeek

  • Zeek represents TCP traffic as connection records and applies a default five-minute inactivity timeout; Zeek-flowmeter extracts features from flows identified by the Zeek engine.Zeek records connection state and excludes trailing non-productive packets after one direction closes when calculating TCP connection duration.Source: §2.2.3 · p. 5

3. Related Work

Prior MQTT security work emphasizes detection, while flow generation and clustering are less often treated as objects of study in their own right. This paper positions tool-aware unsupervised characterization as a way to examine how extraction shapes MQTT traffic structure.

3.1. MQTT Security and Attacks

  • Reported MQTT vulnerabilities include missing required-field, logical-error, and packet-length checks, while proposed security measures span packet validation, network protections, TLS, and external authentication.The reviewed countermeasures can add resource demands, complexity, or deployment challenges.Source: §3.1 · pp. 5–6

3.2. Supervised Intrusion Detection for MQTT

  • Supervised MQTT intrusion-detection studies commonly compare classifiers and report stronger discrimination from uniflow or biflow features than packet-level features, with some exceptions.One reviewed study reports accuracy as high as 98.9% for CNN-LSTM using binary classification on MQTTset.Source: §3.2 · p. 6

3.3. Unsupervised and Anomaly-Based MQTT IDS

  • Unsupervised MQTT IDS work includes packet-level anomaly detection and latent-representation methods, but the reviewed approaches focus on detector performance rather than comparing flow generators.The GAN-AE study documents Tranalyzer use but does not investigate alternative tools on the same traces.Source: §3.3 · pp. 6–7

3.4. Flow Generation and Tool Dependence

  • Flow records are tool-dependent abstractions, and prior work found that some flow tools omit feature categories or generate no flows under their predefined rules.MQTT studies comparing packet, uniflow, and biflow representations generally did not make the flow-generation tool an experimental variable.Source: §3.4 · p. 7

3.5. Unsupervised Clustering for Traffic Characterization

  • Traffic clustering has been used for classification and anomaly detection, but prior work generally treats the traffic representation as fixed rather than comparing how tools shape MQTT structure.The paper uses clustering as an analytical lens, not as a detector or label generator.Source: §3.5 · p. 7

3.6. Research Gap

  • Existing studies leave limited evidence on how different flow-generation tools affect MQTT flow populations and downstream clustering across datasets.The paper addresses this gap with a cross-dataset comparison of three tools and their effects on clustering outcomes.Source: §3.6 · p. 7

4. Methodology

The study evaluates tool effects through dataset preparation, flow generation and cleansing, then clustering and evaluation. It preserves original class proportions and limits flow-tool configuration changes to necessary additions or modifications.

4.1. Scope of the Study

  • The pipeline has three blocks: dataset selection and preparation, flow generation and cleansing, and clustering and evaluation.The study preserves original class proportions where possible and does not use balancing methods such as oversampling, undersampling, or SMOTE.Source: §4.1 · p. 8
  • The study focuses on unencrypted MQTT and uses flow-generation tools in basic settings, leaving tool tuning and MQTTS analysis for future work.The authors isolate flow-generation effects from encryption-related variability, noting that encrypted traffic requires separate analysis of TLS session behavior and visibility.Source: §4.1 · p. 8

4.2. Dataset Collection

The study collected publicly available datasets focused on MQTT attacks, excluding datasets or traffic that did not represent attacks on MQTT itself. The resulting collection contains four datasets with benign traffic and multiple MQTT-oriented attack scenarios.

4.2. Dataset Collection

  • The search retained datasets with attacks on the MQTT protocol and removed non-MQTT attacks from datasets containing mixed attack types.The process began by searching the literature for MQTT attack-based datasets and reviewing their descriptions.Source: §4.2 · p. 8
  • The final collection contains 4 datasets with 1.74M benign packets and 10.74M malicious packets, spanning several MQTT-oriented attack scenarios.These include flooding denial-of-service, brute-force authentication, malformed data, publish flooding, and SlowITe-style slow-rate attacks.Source: §4.2 · p. 8
  • The selected datasets differ in scale, testbed design, traffic-generation tools, and attack diversity.Table 1 summarizes publication year, simulation or real-testbed setup, application domain, attack types, packet counts, and malicious-traffic generation tools.Source: §4.2 · p. 8

4.3. Flow Generation and MQTT Filtering

The packet captures were converted to flows using Argus, NTLFlowLyzer, and Zeek-flowmeter, retaining captures all three tools could process. The resulting flow counts varied by tool, and MQTT filtering and sampling were applied before clustering.

4.3. Flow Generation and MQTT Filtering

  • The 14 retained captures produced 380,575 flow records across the three tools, with flow counts varying substantially by tool.The study excluded any capture that one of the tools failed to process.Source: §4.3 · p. 8
  • Across PCAP files, the Zeek/NTLFlowLyzer flow-count ratio was mostly concentrated near approximately 10^-0.3 with less spread, suggesting Zeek generates 0.5 of NTLFlowLyzer’s flow counts for any given PCAP; Argus-based ratios indicate greater variability.Source: §4.3 · p. 9
  • Flows were retained for MQTT analysis when either endpoint used unencrypted MQTT port 1883.For MQTT-IoT-IDS2020, the clustering pipeline used a class-ratio-preserving subset selected separately for each tool and class.Source: §4.3 · p. 9
  • The tools exported different numbers of fields: Zeek and Argus supported up to 81 and 84 fields, while NTLFlowLyzer supported up to 348.Zeek was modified to include source and destination ports.Source: §4.3 · p. 9

4.4. Feature Preprocessing

Feature preprocessing removed constant or unsuitable fields, transformed selected values, and reduced redundancy in NTLFlowLyzer’s larger feature set.

4.4. Feature Preprocessing

  • Columns with one unique value and fields such as explicit start and end timestamps were removed.The exclusions aimed to remove fields that did not contribute to behavioral characterization or could create artificial separation.Source: §4.4 · p. 9
  • Categorical fields were numerically encoded, numeric values were downcast, and ports were transformed into a direction-related feature for port 1883.Source: §4.4 · p. 9
  • For NTLFlowLyzer, redundant statistical features were removed when their underlying base fields were retained; otherwise, summary statistics such as mean and standard deviation were kept.This was intended to reduce feature duplication while preserving relevant behavioral information.Source: §4.4 · p. 9

4.5. Clustering Configuration

The experiments compare centroid-based, density-based, and hierarchical density-based clustering across dataset–tool pairs, with and without PCA. The configuration tests different clustering granularities and reduced feature spaces.

4.5. Clustering Configuration

  • Each dataset–tool pair was evaluated separately using K-means, DBSCAN, and HDBSCAN under consistent experimental conditions.The hyperparameter ranges were chosen to explore diverse configurations without excessive computational cost.Source: §4.5 · pp. 10–11

4.5.1. Clustering Algorithms

  • K-means tested K=2 through 16 rather than constraining cluster count to the number of labels, with each K run 10 times.The exploratory range allowed finer-grained groupings than the maximum of 6 labeled classes while limiting fragmentation and computational cost.Source: §4.5.1 · pp. 11–12
  • DBSCAN was evaluated over epsilon values from 0.1 to 2.5 and MinPts values from 2 to 10 to test different density assumptions.DBSCAN does not require a preset cluster count and can label low-density samples as noise.Source: §4.5.1 · p. 12
  • HDBSCAN varied min_cluster_size from 2 to 60, extracting stable clusters across multiple density levels rather than using one global radius.This shifts the main configuration choice toward the smallest group size considered meaningful.Source: §4.5.1 · p. 12

4.5.2. Dimensionality Reduction.

  • Experiments ran with and without PCA, reducing features to 10, 20, or 30 components when enough valid features were available.PCA was used to test whether a compact representation of dominant variance affected clustering performance.Source: §4.5.2 · p. 12

4.6. Evaluation Metrics

The section evaluates clustering with the Silhouette score for label-independent cluster quality and the Adjusted Rand Index (ARI) for agreement with dataset labels.

4.6. Evaluation Metrics

  • The Silhouette score identifies which setting of an algorithm generates the most coherent clusters; it evaluates cluster quality without class labels and ranges from -1 to 1, with higher scores indicating greater coherence.A score of -1 indicates a sample is in the wrong cluster, while 1 means it is in the correct cluster.Source: §4.6 · p. 12
  • For each sample, the Silhouette coefficient uses the mean intra-cluster distance a and mean nearest-cluster distance b.Source: §4.6 · pp. 12–13
  • ARI compares generated clusters with the dataset’s original labeled classes and ranges from -1 to 1.A value of 1 indicates perfect agreement, values near 0 indicate agreement similar to random labeling, and negative values indicate worse-than-chance agreement.Source: §4.6 · p. 13

5. Results

The experiments compare clustering results across datasets, tools, algorithms, and feature configurations. Zeek often leads, but the top tool varies by algorithm, dataset, and metric; execution time also scales with flow volume.

5.1. K-means

  • Most K-means top Silhouette and ARI values used Zeek PCA10; exceptions were GothX Silhouette (NTLFlowLyzer PCA10), MQTTset Silhouette (Zeek PCA20), and MQTT-IoT-IDS2020 ARI (Zeek original).Best K was 2 for both metrics on Artemis and MQTT-IoT-IDS2020; for GothX it was 12 for Silhouette and 8 for ARI, and for MQTTset it was 2 and 9, respectively.Source: §5.1 · p. 13, Table 3 · p. 14

5.2. HDBSCAN

  • Zeek with reduced dimensions supplied most top HDBSCAN ARI scores, but Silhouette winners split between Zeek for GothX and MQTT-IoT-IDS2020 and Argus for Artemis and MQTTset.Artemis was the ARI exception, where Argus with original features led; its best ARI was 0.02 versus 0.52 for the next-lowest dataset, and optimal MCS varied widely by metric.Source: §5.2 · p. 13, Table 4 · p. 15

5.4. Efficiency and Scalability

  • K-means had the highest execution times and lowest throughput, followed by DBSCAN and then HDBSCAN; DBSCAN’s execution-time curves rose more slowly than HDBSCAN’s overall.Across the evaluated datasets, K-means time tracked flow count more closely, while DBSCAN and HDBSCAN execution times increased more slowly.Source: §5.4 · pp. 14–15, Table 6 · p. 16

5.3. DBSCAN

  • For DBSCAN, Zeek with reduced dimensions had most top ARI scores, while Silhouette winners varied across NTLFlowLyzer, Argus, and Zeek.The ARI leaders used 21–55 clusters across datasets with 2–6 classes and isolated under 1% noise; Artemis’s best ARI was 0.13 versus 0.91 next-lowest.Source: §5.3 · pp. 13–14, Table 5 · p. 16

5.4.1. Correlations between Flows Count vs. Algorithm Execution Time

  • Execution time was strongly positively correlated with flow count for every tool and algorithm (r>0.89), with K-means and HDBSCAN correlations above 0.97.DBSCAN’s aggregate ‘All’ correlation was r=0.895; the authors identify flow volume as the primary determinant of computational overhead.Source: §5.4.1 · p. 16

6. Discussion

The discussion connects flow-construction choices to differences in flow populations and clustering, while reporting strong Zeek results in some settings and cautioning that runtime comparisons depend on the experimental pipeline.

6. Discussion

  • The unsupervised framing reflects traffic settings with incomplete or uncertain labels, and flow records summarize behavior over time rather than isolated packet events.The discussion highlights duration, packet and byte counts, directionality, rates, timing statistics, and endpoint interactions as flow features.Source: §6 · pp. 16–18
  • Zeek and NTLFlowLyzer flow counts were often proportional, but Argus-related counts varied sharply, including a case with over 1.8K times as many Argus flows as Zeek flows.The 1.8K-times comparison is for the Artemis benign capture; GothX’s normal capture had equal Zeek and Argus counts.Source: §6 · p. 18
  • Zeek with PCA-10 and K-means yielded the strongest results, while K-means mostly favored fewer than 10 clusters and HDBSCAN’s top scores mostly used more than 20.K-means assigns every sample to a cluster, whereas DBSCAN can designate samples as noise.Source: §6 · p. 18
  • Argus splits records beyond 60 seconds, NTLFlowLyzer can split flows beyond 120K seconds, and Zeek does not split ongoing flows.The discussion links these different splitting rules to the observed concentration of NTLFlowLyzer-to-Zeek flow-count ratios.Source: §6 · p. 18
  • Tool-specific end-of-flow and packet-discard rules can change the generated samples and therefore affect which clusters algorithms assign them to.The discussion notes that Zeek discards trailing non-productive packets after one direction closes a connection.Source: §6 · p. 19
  • Within this pipeline, clustering-algorithm choice affected throughput more than flow-generation-tool choice, but differing parallelization and search settings limit that comparison.K-means was evaluated across multiple K values with repeated runs per configuration.Source: §6 · p. 19

6.1. Limitations

  • The analysis covers unencrypted MQTT on port 1883, four public datasets, and three tools; encrypted MQTTS, other tools, and systematic parameter sensitivity remain outside its scope.The tools used default flow-construction settings, and some captures were excluded when not all three tools could process them.Source: §6.1 · p. 19

7. Conclusions

The paper characterizes MQTT flows across datasets, tools, clustering algorithms, and feature configurations. Its results show substantial tool dependence, but no tool dominates every setting.

7. Conclusions

  • The characterization converted packet captures from four public MQTT datasets into more than 380K flows using Zeek-flowmeter, Argus, and NTLFlowLyzer.The study included benign and malicious traffic.Source: §7 · p. 19
  • The number and structure of generated flows varied substantially across tools, showing that extraction depends on each tool’s internal rules and mechanisms.The authors conclude that flow extraction is not a neutral preprocessing step.Source: §7 · p. 19
  • Clustering behavior varied with the flow-generation tool as well as the dataset and algorithm; Zeek-flowmeter performed strongest in most experiments, particularly with PCA-10, but no tool dominated all settings.The comparison spans K-means, DBSCAN, and HDBSCAN, with and without PCA-based dimensionality reduction.Source: §7 · pp. 19–20
  • The authors conclude that flow-level MQTT analysis should account for datasets, tools, preprocessing, clustering algorithms, and evaluation metrics; future work will examine encrypted traffic and additional tools and configurations.Planned work also includes feature-level analysis and further examination of traffic-generation methods.Source: §7 · p. 20

CRediT authorship contribution statement

The contribution statement assigns conceptual, methodological, supervisory, and project roles across three authors.

CRediT authorship contribution statement

Loading 2610.29774v1…