PaperlayerView exact revision on arXiv

Source-linked AI summary

Unified Host and Network Data Set

Melissa J. M. Turcotte, Alexander D. Kent, Curtis Hash

arXiv:1708.07518v1cs.CR

TL;DR

Operational cybersecurity research lacks useful data sets, while organizations face security and privacy concerns in releasing them. This chapter introduces LANL’s Unified Host and Network Data Set; in its flow records, 57% of TCP and approximately 70% of all biflows have a zero packet count in one direction.

  • Problem

    Useful cybersecurity research data sets are scarce, and released data may be stripped of information or restricted in ways that limit analysis and independent validation.

    Source: §1 · p. 1

  • Method

    The chapter introduces LANL’s Unified Host and Network Data Set and describes normalizing it for non-expert researchers while minimizing artifacts.

    Source: §1 · pp. 1–2

  • Results

    57% of TCP and approximately 70% of all biflows have zero SrcPackets or DstPackets, suggesting failure to find matching uniflows in both directions.

    Source: §2.2 · p. 4

  • Takeaways & Limitations

    The data set is intended to enable research and serve as an example for other providers, including research combining host logs with network data.

    Source: §5 · p. 13

    The unidirectional flow records do not reveal which endpoint initiated a bidirectional connection, complicating detection of behaviors such as lateral movement.

    Source: §2.1 · p. 3

Abstract

from arXiv · show

The lack of data sets derived from operational enterprise networks continues to be a critical deficiency in the cyber security research community. Unfortunately, releasing viable data sets to the larger com- munity is challenging for a number of reasons, primarily the difficulty of balancing security and privacy concerns against the fidelity and utility of the data. This chapter discusses the importance of cyber secu- rity research data sets and introduces a large data set derived from the operational network environment at Los Alamos National Laboratory. The hope is that this data set and associated discussion will act as a catalyst for both new research in cyber security as well as motivation for other organizations to release similar data sets to the community.

1 Introduction

Cybersecurity research lacks useful operational datasets, while security and privacy concerns complicate releases that preserve fidelity and utility. The chapter introduces LANL’s Unified Host and Network Data Set and explains how it fits a broader landscape of unevenly useful and documented data sets.

1 Introduction

  • LANL introduces a third data set, the Unified Host and Network Data Set, to stimulate operational cyber-data research and encourage other organizations to release useful data.Source: §1 · p. 1
  • The data set combines roughly 90 days of LANL host events and network flows, although the two subsets are not fully inclusive.Host logs came from most LANL Windows computers; network flows came from internal core routers and also include non-Windows computers and other devices.Source: §1 · p. 1
  • Identifying values are de-identified consistently across host and network data to preserve joint analysis, with selected ports, system usernames, and core hosts left identifiable.Source: §1 · p. 1
  • The chapter normalizes the data for researchers outside the domain while seeking to minimize artifacts introduced by normalization.Source: §1 · p. 2

1.1 Related public data sets

  • Existing public data sets include synthetic and dated collections, while some require vetting and carry use restrictions.Source: §1.1 · p. 2
  • Many existing data sets are inadequately characterized, motivating detailed discussion of their context, normalization, and idiosyncrasies.Such information can help researchers avoid false assumptions and reproduce similar data.Source: §1.1 · p. 2

2 Network Flow Data

This section describes LANL NetFlow-derived communication records and their transformation into directed biflows with device names. It also documents direction and mapping gaps alongside traffic-distribution diagnostics.

2 Network Flow Data

  • The network-flow release contains NetFlow V9 records of LANL device communications, without payload-level data.The fields considered include timestamps, source and destination IPs, protocol, ports, packets, and bytes.Source: §2 · p. 2, §2.1 · p. 2

2.1 Collection & Transformation

  • The authors stitch opposing uniflows and duplicates into directed biflows using port heuristics, then aggregate matched records.Flows are matched by a 5-tuple and periodically evicted after 30 minutes of inactivity; long-active flows are reported approximately every 3 hours.Source: §2.1 · pp. 3–4
  • Because IP addresses can be transient, shared, or multiple per device, the release maps them to FQDNs using DNS and DHCP logs over time.Where mapping fails, IP addresses remain in the data; where multiple FQDNs match, business rules favor the least-ephemeral name.Source: §2.1 · p. 4

2.2 Data Quality

  • 57% of TCP and approximately 70% of all biflows have a zero-packet side, largely because LANL sensors export only one direction on many routes.The bottom plot also tracks FQDN-mapping failures, which are more frequent for DstDevices than SrcDevices.Source: §2.2 · p. 4, Figure 1 · p. 5
  • Daily protocol proportions and top-port histograms characterize traffic; a non-uniform SrcPort histogram may reflect direction-selection errors or protocols using non-ephemeral source ports.The figures show daily protocol proportions and the top 50 source and destination ports.Source: §2.2 · p. 4, Figure 2 · p. 5, Figure 3 · p. 6
  • Per-protocol duration and byte plots include long-lived UDP and ICMP biflows, indicating persistent traffic sharing a 5-tuple; exemplar plots show in- and out-degree distributions.The persistent UDP and ICMP flows are attributed to applying biflow transformation beyond TCP uniflows.Source: §2.2 · p. 5, Figure 4 · p. 6, Figure 5 · p. 7

3 Windows Host Log Data

The chapter describes a 90-day LANL Windows host-log subset covering authentication and process activity, then explains how its fields and values were normalized for analysis.

3 Windows Host Log Data

  • The release contains JSON host events collected with Windows Logging Service from LANL Windows computers, covering authentication and process activity.WLS forwards event logs and administrator-defined contextual data to collection servers.Source: §3 · p. 7
  • Table 2 lists the included host-log Event IDs; records always contain EventID, LogHost, and Time, while attributes can vary by event.Figure 6 presents EventID percentages and LogonTypes for EventIDs 4624, 4625, and 4634; Figures 7 and 8 show unique-entity counts overall and per day.Source: §3 · pp. 7–8, Table 2 · p. 8, Figure 6 · p. 9
  • For directed authentication events, LogHost is the destination computer, while Source identifies the originating computer; TGS requests instead record the intermediary domain machine and requested service in ServiceName.A subsequent network logon to the computer named by ServiceName is common after a TGS credential grant.Source: §3 · pp. 8–10

3.1 Data parsing considerations

  • Host logs may record a parent process without an extension, which can be recovered through ParentProcessID; Source may be an IP address while LogHost is a hostname.Source: §3 · p. 10, §3.1 · p. 11
  • Parsing standardizes EventID-dependent attribute names to UserName and canonicalizes usernames and process names for consistency across records.Dates, version numbers, operating systems, and hexadecimal strings are removed from process names where possible.Source: §3.1 · pp. 10–11

4 Research directions

The chapter identifies anomaly detection as an evolving research area, noting the practical hurdles of false positives and opaque alerts. It outlines host-log and broader data-analysis directions, including interpretable anomalies, event sequences, and process trees.

4 Research directions

  • Operational cyber tools remain mostly rule- or signature-based because statistical anomaly detection faces high false-positive rates and alerts that lack interpretability.Anomaly methods can detect new attack variants by learning normal behavior and finding deviations, motivating research on false positives and interpretable alerts.Source: §4 · p. 11
  • For host-log time series, periodicity and correlations between event arrivals matter; credential renewal often accounts for periodic patterns.EventID 4624, LogonType 3 forms a significant share of events, and interpreting sequences as higher-level actions could aid modeling and analysts.Source: §4 · pp. 11–12, Figure 9 · p. 12
  • Process-start and stop records can support process-tree analysis, where atypical events could indicate cyber security anomalies.Processes form a hierarchy in which descendants can, in theory, be traced through ancestors to a root process.Source: §4 · p. 12
  • Other proposed directions include modeling network segmentation and risk, studying information loss during collection and normalization, and analyzing organizational and social behavior.Source: §4 · p. 12

5 Conclusion

The chapter presents operational network-flow and host-log data as resources for cyber security research and as examples for prospective data providers. It also highlights the relatively limited attention to host logs and to analyses combining data sets.

5 Conclusion

  • The chapter presents its network-flow and host-log data as resources intended to support cyber security research and provide an example for other data-set providers.Source: §5 · pp. 12–13
  • The conclusion says host-log data have received relatively little attention in the literature as endpoint security tools gain popularity.Source: §5 · p. 13
  • The chapter argues that meaningful analysis combining these and other data sets has received even less consideration and is important to advancing cyber defense.Source: §5 · p. 13

A Host log fields

The appendix defines common host-log fields for identifying records, users, sessions, authentication events, and processes.

A Host log fields

  • Time is the event epoch time in seconds; EventID identifies the event type, and LogHost names the computer where it was recorded.Source: A Host log fields · p. 13
  • UserName identifies the initiating account, DomainName its domain, and LogonID identifies the session; SubjectUserName and SubjectDomainName describe the mapped account.A username ending in $ denotes a computer account; LogonID is semi-unique across current sessions and LogHost.Source: A Host log fields · p. 13
  • For authentication events, Source identifies the originating computer; Status indicates success or failure, while LogonType and AuthenticationPackage characterize the authentication.FailureReason gives the reason for an unsuccessful logon.Source: A Host log fields · pp. 13–14
  • ServiceName names the account or service requested for a ticket, while Destination identifies the server accessed by mapped credentials.Source: A Host log fields · p. 13
  • ProcessName names the executable and ProcessID identifies a process within its current LogHost; ParentProcessName and ParentProcessID identify its parent process.A preceding EventID 4688 with a matching ProcessID can identify the parent process start event.Source: A Host log fields · p. 14

B Event Attributes

This section presents Table 3, titled “Event Attributes.”

B Event Attributes

Loading 1708.07518v1…