PaperlayerView exact revision on arXiv

Source-linked AI summary

A Survey on Big Data for Network Traffic Monitoring and Analysis

Alessandro D'Alconzo, Idilio Drago, Andrea Morichetta, Marco Mellia, Pedro Casas

arXiv:2003.01648v1cs.DCcs.NI

TL;DR

NTMA faces growing, heterogeneous traffic and historical-data demands that make scalable monitoring and analysis difficult, while prior work lacks a systematic account of its combination with big data. This survey catalogs NTMA approaches and technologies, finding that work commonly reduces data early and then uses centralized or batch analysis rather than distributed or streaming analytics.

  • Problem

    NTMA must handle growing traffic, heterogeneous measurements, and historical datasets, but there is no systematic survey of how it combines with big-data approaches.

    Source: §I · p. 1

  • Method

    The survey catalogs NTMA research using big-data approaches, focusing on data management technologies and briefly covering analytics.

    Source: §I · p. 1

  • Results

    Most surveyed works use a single big-data source, reduce data early, and then apply traditional non-distributed machine learning; big-data analysis is mostly batch rather than streaming.

    Source: C. Single source, early reduction,… · p. 10, E. The rare cases of online analysis · p. 10

  • Takeaways & Limitations

    The survey provides application-oriented guidelines and identifies open issues and promising research directions for using big data in NTMA.

    Source: §I · p. 1

Abstract

from arXiv · show

Network Traffic Monitoring and Analysis (NTMA) represents a key component for network management, especially to guarantee the correct operation of large-scale networks such as the Internet. As the complexity of Internet services and the volume of traffic continue to increase, it becomes difficult to design scalable NTMA applications. Applications such as traffic classification and policing require real-time and scalable approaches. Anomaly detection and security mechanisms require to quickly identify and react to unpredictable events while processing millions of heterogeneous events. At last, the system has to collect, store, and process massive sets of historical data for post-mortem analysis. Those are precisely the challenges faced by general big data approaches: Volume, Velocity, Variety, and Veracity. This survey brings together NTMA and big data. We catalog previous work on NTMA that adopt big data approaches to understand to what extent the potential of big data is being explored in NTMA. This survey mainly focuses on approaches and technologies to manage the big NTMA data, additionally briefly discussing big data analytics (e.g., machine learning) for the sake of NTMA. Finally, we provide guidelines for future work, discussing lessons learned, and research directions.

I. INTRODUCTION

NTMA must handle growing, heterogeneous traffic streams and historical data, making scalable monitoring and analysis difficult. This survey examines how big-data approaches are used for NTMA, defines its literature scope, and offers application-oriented guidance.

I. INTRODUCTION

  • NTMA supports network visibility, anomaly and attack detection, usage monitoring, and historical analysis for traffic engineering and troubleshooting.Increasing traffic volumes, transmission speeds, service and attack changes, and diverse measurement sources complicate scalable, distributed analysis.Source: §I · p. 1
  • The survey addresses the lack of a systematic account of how NTMA and big data are combined and assesses how fully NTMA uses big-data technologies.It aims to give network researchers an overview of prior successes, pitfalls, challenges, and opportunities.Source: §I · p. 1
  • The survey catalogs NTMA work using big-data approaches to provide principled guidance by application requirements and purposes, and to identify open issues and research directions.The paper concentrates on managing big NTMA data and discusses analytics briefly.Source: §I · p. 1

A. Survey methodology

  • The survey reviews big-data surveys from the prior decade, then searches recent NTMA venues and digital libraries and selects papers by venue, citations, and scope.Its NTMA search focuses on the last five years of selected conferences, journals, workshops, and special issues.Source: A. Survey methodology · p. 2

A. Definition

  • Big data is characterized by volume, velocity, variety, and veracity; the survey adopts a five-V definition that also includes value.Its criteria cover large datasets, high-rate or real-time capture and analysis, heterogeneous sources and quality, and value extracted from data.Source: §I · p. 1, A. Definition · p. 2

B. Knowledge discovery in big data

The section frames knowledge discovery as a staged process from data input through analysis to output. It distinguishes data management from analytics and explains why big-data challenges make management consequential across the process.

B. Knowledge discovery in big data

  • KDD is organized as input, analysis, and output, grouping operations from gathering and preparation through mining, interpretation, and visualization,.The survey adapts this scheme to characterize NTMA papers by KDD stage.Source: B. Knowledge discovery in big data · pp. 2–3
  • Data input manages raw data into formats suitable for mining, including preparation, heterogeneous-source integration, and cleaning spurious data.Data analysis then extracts models, patterns, relations, and rules using statistical, machine-learning, or data-mining methods.Source: B. Knowledge discovery in big data · p. 2
  • Data output converts information into knowledge through quality measurement, concise displays, and analyst interpretation, but the survey does not evaluate this stage.Source: B. Knowledge discovery in big data · p. 2
  • The survey uses “data analysis” for information-extraction algorithms and “data analytics” for the broader process from analysis to knowledge discovery.Unlike, it does not use “data analysis” to mean the whole analytics process.Source: B. Knowledge discovery in big data · p. 3
  • Big-data volume, velocity, and variety make data management more complex and important because it reduces data volume and complexity and affects analysis speed and result value.The challenges call for frameworks and programming models that address the complete KDD cycle, while distributed frameworks are still expanding to cover all phases.Source: B. Knowledge discovery in big data · p. 3

C. Programming models and platforms

This section surveys four big-data processing models and gives a focused overview of Hadoop’s ecosystem, particularly Hadoop v2.0 and Spark.

C. Programming models and platforms

  • Big-data platforms are grouped into general-purpose, SQL-like, graph-processing, and stream-processing models.Examples include MapReduce, Dryad, Flink, and Spark for general-purpose processing; Hive and HAWQ for SQL-like processing; Pregel and GraphLab for graphs; and Storm and S4 for streams.Source: C. Programming models and platforms · p. 3
  • A comprehensive review of big-data programming models and platforms is outside this section’s scope.The section refers readers to and for complete surveys.Source: C. Programming models and platforms · p. 3

1) The Hadoop ecosystem:

  • Hadoop v2.0 combines the Hadoop kernel, MapReduce, and HDFS, with YARN managing cluster resources for competing jobs.Other resource managers and execution engines can also provide resources to Spark.Source: 1) The Hadoop ecosystem · p. 3
  • Spark uses in-memory data representations for multi-step transformations, unlike MapReduce’s disk-based batch operations.The section notes Spark’s use in iterative, real-time, and some batch applications that require more than a few MapReduce stages.Source: 1) The Hadoop ecosystem · p. 3
  • The Hadoop ecosystem adds high-level languages, libraries, and specialized tools for graph and stream processing.Examples include Pig Latin, Hive, Mahout, MLlib, GraphX, and Spark Streaming.Source: 1) The Hadoop ecosystem · p. 3

III. CATEGORIZING NTMA APPLICATIONS

The section organizes NTMA applications by their management purpose, then describes representative tasks and their challenges. Its survey focuses on traffic prediction, classification, fault management, and security.

A. Taxonomy

  • The survey uses eight network-management categories but covers only prediction, classification, fault management, and security.It excludes congestion control, routing, resource management, and QoS/QoE management because their dependence on NTMA varies and big-data literature was scarce at survey time.Source: A. Taxonomy · p. 4

B. NTMA applications

  • Table I presents the categories used to organize the survey’s examples of NTMA applications.The section proceeds to examples within those categories.Source: B. NTMA applications · p. 4

1) Traffic prediction:

  • Traffic prediction estimates future link status from prior measurements, but changes in network behavior make forecasting difficult.It supports traffic engineering, including decisions about when to add capacity to maintain QoS.Source: 1) Traffic prediction · p. 4

2) Traffic classification:

  • Traffic classification identifies services from traffic, but encryption, limited visible packet information, and changing services complicate identification.Approaches include packet inspection, supervised or unsupervised learning, and behavioral methods; unsupervised methods can explore services without training data.Source: 2) Traffic classification · p. 4

3) Fault management:

  • Fault management predicts, detects, isolates, and corrects faults to minimize downtime, with root-cause localization as a key step.Analytics may combine traffic, system logs, and active measurements to help identify causes amid cascading faults and other network changes.Source: 3) Fault management · p. 4

4) Security:

  • Security monitoring uses signatures for known attacks and anomaly detection to identify behavior that departs from a normal-network baseline.Signatures require well-documented attacks; anomaly detection can flag unknown threats but may miss stealth attacks and produce false positives.Source: 4) Security · pp. 4–5

C. Big data challenges in NTMA applications

NTMA applications face big-data challenges across data volume, speed, variety, quality, and value. The section then turns to data management, starting with collection systems and measurement approaches.

C. Big data challenges in NTMA applications

  • NTMA examples illustrate big-data challenges in volume, velocity, variety, veracity, and value.Classification may process large streams on-the-fly, while encrypted traffic reduces available information and security samples can be difficult to reproduce realistically.Source: C. Big data challenges in NTMA… · p. 5

IV. DATA MANAGEMENT FOR NTMA

  • The survey next examines NTMA data-management steps, leaving big-data analytics to a later section.Source: §IV · p. 5

A. Data collection

  • Internet measurement creates collection-scale challenges, including scaling acquisition and reducing data volumes without harming quality.The passage describes more than half a million networks, 1.6 billion websites, and more than 3 billion users.Source: A. Data collection · p. 5
  • Active measurement injects data and observes responses, while passive measurement continuously observes traffic and extracts indexes in real time.Exporters send packet copies or traffic statistics to collectors; high rates have encouraged preprocessing at exporters, including flow-based summaries.Source: A. Data collection · p. 5

1) Packet-based methods:

Packet-based monitoring preserves flexible access to traffic data but consumes substantial resources, while flow-based summaries reduce transmission needs at the cost of information. Ingestion into big-data frameworks still requires NTMA-format integration.

1) Packet-based methods:

  • Packet inspection offers the most flexibility but demands substantial resources; filtering, sampling, header-only export, and initial-flow-packet capture reduce what is collected.DPI can operate at multiple Gbit/s with hardware support, but encryption and privacy policies challenge its use.Source: 1) Packet-based methods · pp. 5–6
  • More than 600 GB of pcaps are saved per day even when capturing only 5 packets or 5 kB per flow; ISP header-only capture still yields around 45 GB per hour.The table compares packet-trace sizes across vantage points and collection methods.Source: 1) Packet-based methods · p. 6, Table II · p. 6

2) Flow-based methods:

  • Flow records aggregate traffic and lower transmission requirements, but lose information; Cisco estimated that about 90% of network traffic analyses were flow-based in 2013.NetFlow, sFlow, and IPFIX are common options; sFlow sampling may hurt some applications, while IPFIX permits field selection.Source: 2) Flow-based methods · p. 6
  • More than 70 Mbit/s of flow data can still result from one 50 Gbit/s vantage point using IPFIX, while aggregation across vantage points may require several Gb/s of input.Sampling reduces data volume considerably but limits NTMA applications.Source: 2) Flow-based methods · p. 6, Table III · p. 6
  • Archiving compressed flow data from the example vantage point grows linearly and consumes almost 30 TB after four years.Source: 2) Flow-based methods · p. 6

B. Data ingestion

  • NTMA ingestion requires plugins or scripts to adapt measurement formats; once these exist, standard tools such as Kafka can be used, and open-source plugins ease reuse.Spot uses user-provided parsing scripts, while PNDA offers plugins for popular NTMA formats.Source: B. Data ingestion · pp. 6–7

C. Data storage

Big-data storage systems scale across servers, but NTMA measurements often use formats that distributed frameworks cannot readily read or split. Existing approaches adapt frameworks or formats, yet remain specific or centralized.

1) Generic systems:

  • Big-data storage systems favor horizontal scaling across servers, making consistency, availability, and fault tolerance important; NoSQL alternatives trade rigid relational schemas for better scaling on large datasets,.NoSQL layouts include key-value, column-oriented, document-oriented, and graph-oriented databases.Source: 1) Generic systems · p. 7

2) NTMA storage in big data frameworks:

  • NTMA measurements often require special storage or transformation because ad-hoc collection formats are not directly readable by big-data frameworks.Pcap traces are not readily separable for parallel reading, and Hadoop-based systems cannot automatically split large binary nfdump files.Source: 2) NTMA storage in big data frameworks · p. 7
  • Format-specific APIs let Hadoop-based network traffic measurement applications process pcap and NetFlow traces in native formats without prior transformation; similar work is needed for each measurement format or analytics framework.Source: 2) NTMA storage in big data frameworks · p. 8
  • Specialized network-measurement storage can improve efficiency through compression, indexing, and retrieval, but the described systems are generally centralized and lack horizontal scalability.DBStream supports continuous and rolling analytics from data streams.Source: 2) NTMA storage in big data frameworks · p. 8

D. Pre-processing

NTMA preprocessing converts raw measurements into feature vectors, often through transformations before loading data into big-data frameworks. These transformations can improve analysis performance but also introduce information loss and integration costs.

D. Pre-processing

  • Preprocessing covers the steps that convert raw data into feature vectors describing instances such as network flows or contacted services.Source: D. Pre-processing · p. 8

1) Transformations:

  • Measurements may be transformed after collection into framework-ready or query-optimized formats, including Parquet, Avro, text, Cassandra, or indexed HDFS/HBase partitions.Some systems instead combine storage in Cassandra with APIs that read binary formats directly, or stream IPFIX records in JSON,.Source: 1) Transformations · p. 8
  • Transformations can boost analysis performance, but may lose information and create redundant copies that make integration with other systems harder.Source: 1) Transformations · p. 8

2) Feature engineering:

NTMA feature engineering remains fragmented, with few large-dataset studies and no holistic solution. Big-data analytics frameworks offer scalable options, but algorithms may require preprocessing or parallelization to handle large, high-dimensional inputs.

2) Feature engineering:

  • Some feature studies rely on older datasets, including datasets whose use has been criticized; one study found highly correlated features that were mostly unnecessary,.Source: 2) Feature engineering · p. 9
  • NTMA feature research lacks widely accepted features and systematic ways to compare or benchmark systems; existing studies generally use small datasets or tailor features to specific problems.Abt et al. report accuracy and recall above 92% for NetFlow feature selection in botnet C&C detection, while other work addresses DDoS and P2P traffic,.Source: 2) Feature engineering · p. 9

A. Generic big data frameworks

  • Big-data algorithms can demand substantial resources when they do not scale linearly, so common responses include simplifying inputs or parallelizing algorithms, sometimes with approximations.Source: A. Generic big data frameworks · p. 9
  • Available machine-learning tools include both non-distributed and distributed frameworks, with streaming-specific tools addressing strict time constraints.Examples include Weka and R, TensorFlow and MLlib, and StormMOA.Source: A. Generic big data frameworks · p. 9

B. Literature categorization

The survey finds that big-data techniques are appearing in NTMA, especially in network security, but big-data platforms are rarely used for machine-learning analytics. Most studies handle large data volumes, yet typically use one source, reduce data early, and run machine learning on traditional non-distributed platforms.

B. Literature categorization

  • The literature review checks data scale, use of popular big-data frameworks, machine-learning use, and whether big-data platforms support the machine-learning process.The survey treats datasets larger than tens of GBs, backbone-network cases, or months- to years-long studies as big data when size is unspecified.Source: B. Literature categorization · p. 9
  • The review also checks online analysis and use of multiple data sources, but excludes veracity and value because they are cumbersome to evaluate.Source: B. Literature categorization · p. 9
  • Network security attracts more big-data research, while adoption of big-data platforms for machine-learning techniques remains limited.The survey also reports that big-data techniques are starting to permeate NTMA applications.Source: B. Literature categorization · pp. 9–10

C. Single source, early reduction, sequential analysis

D. Big data platforms enabling big NTMA

The survey finds that big-data platforms can scale analysis on large datasets, but NTMA analytics is mostly batch-based and machine learning is often centralized after data reduction. It identifies gaps in generic platforms, distributed analytics, public datasets, and coverage of some NTMA application areas.

D. Big data platforms enabling big NTMA

F. Takeaway

  • Machine learning is widespread, especially for security and anomaly detection, but many algorithms scale poorly on large NTMA datasets and are difficult to parallelize.This constrains big-data platforms' use in the analytics stage and leaves many works performing data reduction during preprocessing.Source: F. Takeaway · p. 10

VI. CHALLENGES, OPEN ISSUES AND ONGOING WORK

  • Most NTMA solutions remain customized rather than using a generic platform, amid gaps in scalable data access, flexible storage, and standardization.Flow-based approaches scale better but lose detail; available APIs and layers for data formats are described as neither generic nor comprehensive.Source: §VI · pp. 10–11
  • Most studies use big-data techniques for preprocessing or feature extraction, then run machine-learning analysis centrally; distributed algorithms could support more scalable NTMA analysis.The survey notes that large-dataset machine learning could produce more accurate NTMA models, while some algorithms are hard to parallelize and platform support remains early.Source: §VI · p. 11
  • Few works apply big-data approaches to fault management, and categories such as QoS/QoE management are also rarely addressed.The survey suggests the lack of generic and standard NTMA platforms may be related to these gaps.Source: §VI · p. 11
  • Only two surveyed contributions disclose public datasets, and scarce or outdated datasets limit reproducibility and benchmarking of newer scalable solutions.Source: §VI · p. 11
  • Beam16 unifies batch and streaming models and improves portability, but the survey reports no NTMA applications using it.Source: §VI · p. 11
  • The survey concludes that practical big-data NTMA solutions require community synergies and consolidation around established big-data platforms.Source: §VI · p. 11

BIOGRAPHIES

The biographies describe four researchers whose interests include network traffic analysis, security, and big-data or machine-learning approaches for networking.

BIOGRAPHIES

  • Idilio Drago's research interests include Internet measurements, big-data analysis, and network security.Source: BIOGRAPHIES · p. 14
  • Andrea Morichetta's research interests include traffic analysis, security, and data analysis.Source: BIOGRAPHIES · p. 14
  • Marco Mellia's research interests include traffic monitoring and analysis, big data, and machine learning for traffic analysis, cybersecurity, and network monitoring.Source: BIOGRAPHIES · p. 14
  • Pedro Casas's work focuses on machine learning and data mining for networking, big-data analytics and platforms, network measurements, security, anomaly detection, and QoE monitoring.Source: BIOGRAPHIES · p. 14
Loading 2003.01648v1…