PaperlayerView exact revision on arXiv

Source-linked AI summary

Entropy/IP: Uncovering Structure in IPv6 Addresses

Pawel Foremski, David Plonka, Arthur Berger

arXiv:1606.04327v2cs.NIcs.AIcs.ITmath.IT

TL;DR

IPv6 address assignment practices and the vast address space make address structure difficult to understand. Entropy/IP models active-address samples with entropy and machine-learning techniques, then uses the models to generate scan candidates. The evaluation shows structure discovery in network and interface identifiers, with potential for targeted surveys.

  • Problem

    IPv6 address assignment features and address-space flexibility complicate understanding address structure, while analysts need that understanding for interpreting measurements and accounting for infeasible full-space scans.

    Source: §1 · p. 1

  • Method

    Entropy/IP analyzes active-address samples using positional entropy, clustering, and Bayesian Networks to model IPv6 address structure and generate candidate targets.

    Source: §1 · pp. 2–3

  • Results

    The evaluation demonstrates discovery of structure in both interface identifiers and network identifiers, including subnets.

    Source: §7 · p. 14

  • Takeaways & Limitations

    The initial evaluation suggests potential for targeted surveys of IPv6 address space and for improving analysts’ understanding of network operation and defense.

    Source: §7 · p. 14, §6 · p. 14

    The method treats address sets as active at one point in time and does not account for their temporal characteristics.

    Source: §6 · p. 14

Abstract

from arXiv · show

In this paper, we introduce Entropy/IP: a system that discovers Internet address structure based on analyses of a subset of IPv6 addresses known to be active, i.e., training data, gleaned by readily available passive and active means. The system is completely automated and employs a combination of information-theoretic and machine learning techniques to probabilistically model IPv6 addresses. We present results showing that our system is effective in exposing structural characteristics of portions of the IPv6 Internet address space populated by active client, service, and router addresses. In addition to visualizing the address structure for exploration, the system uses its models to generate candidate target addresses for scanning. For each of 15 evaluated datasets, we train on 1K addresses and generate 1M candidates for scanning. We achieve some success in 14 datasets, finding up to 40% of the generated addresses to be active. In 11 of these datasets, we find active network identifiers (e.g., /64 prefixes or `subnets') not seen in training. Thus, we provide the first evidence that it is practical to discover subnets and hosts by scanning probabilistically selected areas of the IPv6 address space not known to contain active hosts a priori.

1. INTRODUCTION

IPv6 assignment practices and address-space flexibility complicate structural analysis. Entropy/IP analyzes active-address samples with information-theoretic and machine-learning methods to discover address structure and generate scan targets.

1. INTRODUCTION

  • IPv6 structure analysis supports applications including network awareness, address-plan assessment, and measurement of active hosts and prefixes.The paper motivates this analysis in part because full IPv6 address-space scans are infeasible and address counts can be difficult to interpret.Source: §1 · pp. 1–2
  • Entropy across address positions and adjacent-nybble differences helps identify boundaries between semantically distinct address segments.Adjacent nybbles with similar entropy are grouped into larger segments.Source: §1 · p. 2
  • Address context matters: repeated characters may carry little structural information, while positional entropy can help distinguish address structure from pseudo-random content.The paper notes that stateless rules can misclassify addresses when similar addresses in the same prefix provide relevant context.Source: §1 · p. 2
  • Entropy/IP clusters segment values with DBSCAN and models their conditional probabilities across segments using Bayesian Networks.The system does not train on predefined address features, instead relying on entropy-based analysis to discover characteristics.Source: §1 · p. 2
  • The system presents entropy plots, segment-value frequency views, and a Bayesian Network for analyst exploration, alongside a target-address generator.Its workflow ingests addresses, computes entropy, discovers segments, and builds a model.Source: §1 · p. 2
  • Entropy/IP automatically discovers network IPv6 layout from observed active addresses and can generate candidate targets for broader scanning.Its intended scope includes CIDR prefixes, IGP subnets, network identifiers, and interface identifiers.Source: §1 · pp. 2–3

2. RELATED WORK

Prior work applies entropy to traffic features, detects pseudo-random IPv6 segments, and analyzes address aggregates. Entropy/IP instead models segments across whole IPv6 addresses and can generate unseen targets.

2. RELATED WORK

  • Prior entropy-based traffic analyses treat addresses as atomic features, unlike Entropy/IP’s use of entropy to expose address structure,,,.These works generally use entropy to characterize feature distributions or time-series variation.Source: §2 · p. 3
  • Privacy-address detection work identifies pseudo-random IPv6 segments; Entropy/IP also seeks these segments but uses entropy,, [27],.Privacy extensions introduce pseudo-random values that may have high entropy but are not pertinent to network structure.Source: §2 · p. 3
  • MRA Count Ratios summarize IPv6 address aggregates, while Entropy/IP separately analyzes nybble positions and groups them into variable-length segments, [27],,.The paper presents its analysis as a complementary viewpoint to MRA.Source: §2 · p. 3
  • Unlike IID-pattern scanning approaches that assume target /64 prefixes are known, Entropy/IP models whole addresses and can generate target /64 prefixes.The paper also distinguishes its probabilistic generation of unseen targets from prior approaches based on known or observed patterns,, [13],.Source: §2 · pp. 3–4

3. DATASETS

The study uses IPv6 address collections gathered in Q1 2016 from several sources, spanning server, router, and client networks. It combines operator-specific datasets with larger category aggregates and a BitTorrent client set.

3. DATASETS

  • The study collected 3.5 billion IPv6 addresses in Q1 2016 and assembled both real-world network sets and category-level aggregates.The small sets cover servers, routers, and clients, with five major operators represented in each category.Source: §3 · p. 4
  • Table 1 reports unique-address counts for small S*, R*, and C* datasets and aggregate AS, AR, and AC datasets.The small datasets represent servers, routers, and clients, respectively.Source: Table 1 · p. 4, §3 · p. 4
  • Server addresses came from DNS; router addresses came from DNS and traceroutes; client addresses came from CDN web requests and BitTorrent.The client web-request addresses were collected during 17–23 March 2016.Source: §3 · p. 4
  • Aggregate server, router, and client datasets contain 790K, 12M, and 3.5 billion addresses, respectively, across 4.3K, 5.5K, and 6.0K /32 prefixes.For Section 5.1, the authors stratified sampling by randomly selecting 1K IPs from the /32 prefixes to reduce network overrepresentation.Source: §3 · p. 4
  • The additional BitTorrent aggregate contains 220K peer addresses in 1.8K /32 prefixes, collected from trackers and DHT peers across three regions.Collection ran from 5–8 March 2016.Source: §3 · p. 4
  • Presented IPv6 addresses were anonymized by replacing their first 32 bits with a documentation prefix.Embedded IPv4 addresses were anonymized by changing their first byte to the 127.0.0.0/8 prefix.Source: §3 · p. 5

4. METHODOLOGY

Entropy/IP’s methodology combines entropy-based address segmentation, mining of common segment values, and Bayesian Networks to model dependencies and generate candidate addresses. Its interface supports visual exploration of these structures and conditional probabilities.

4. METHODOLOGY

  • The interface plots entropy by nybble, displays segment-value distributions, and shows Bayesian Network dependencies that can be explored by conditioning on segment values.The model can also generate candidate addresses for targeted scanning.Source: §4 · p. 5, §4.4 · p. 8

4.1 Entropy Analysis

  • Entropy/IP measures variability at each of 32 hexadecimal-character positions across an address set; lower per-position entropy means values are more likely to remain constant.It also defines total entropy to quantify address-set variability and the difficulty of guessing addresses by chance.Source: §4.1 · p. 6

4.2 Address Segmentation

  • Adjacent nybbles are grouped into segments when their normalized entropies remain similar, using thresholds and hysteresis, with fixed boundaries at bits 1–32 and after bit 64.The authors say the thresholds were developed on real-world networks and can be tuned to specific networks.Source: §4.2 · p. 6

4.3 Segment Mining

  • Segment mining selects popular values and ranges using frequency outliers, DBSCAN clustering of values, and DBSCAN clustering of value-count histograms.The procedure adds at most ten elements per step and stops when no more than 0.1% of values remain.Source: §4.3 · pp. 6–7

4.4 Modeling IPv6 Addresses

  • The system represents mined segment values as categorical random variables and learns a Bayesian Network to model their dependencies.To constrain learning, each segment can depend only on earlier segments; the model can be queried conditionally or used to generate addresses for scanning.Source: §4.4 · pp. 7–8

4.5 Discussion

  • Entropy/IP uses 4-bit nybbles, which the authors consider simple and sufficient; other bit widths can be used by adapting entropy analysis, while segmentation can adjust granularity.The chosen width matches the smallest chunk in IPv6 text representations.Source: §4.5 · p. 8
  • A preliminary windowing idea computes entropy for every possible bit-range, with each plotted point representing bits Y through Y+X; other variability measures could also be used.The authors suggest windowing may aid visual pattern discovery.Source: §4.5 · p. 8
  • The authors found threshold-based segmentation preferable to adjacent-entropy differences, while noting that segmentation without prior knowledge can impose an artificial model; they judged Bayesian Networks powerful, explorable, and concise relative to alternatives.They note that segment mining and Bayesian Network modeling may compensate for minor segmentation glitches; Probability Trees need abundant data, while Markov Models cannot directly capture non-adjacent dependencies.Source: §4.5 · p. 8

5. EVALUATION

Across client, server, and router datasets, Entropy/IP exposed varied address structures; scanning also found active addresses and previously unseen network identifiers. Results included predictable client prefixes and operator feedback that both confirmed some analyses and identified an address-plan error.

5.1 Big Picture: Aggregates

  • Aggregate address patterns differed by host type: clients were most random, routers showed dips consistent with some non-MAC-based IIDs, and servers were least random.Client entropy was near 1 in the lower 64 bits, while server entropy oscillated across addresses and increased steadily from bit 80 to 128.Source: §5.1 · pp. 8–9

5.2 Servers

  • Server datasets showed diverse structure: S1 had 10 segments and four addressing variants, while S2–S5 exhibited CDN, cloud, and service-specific patterns.In S1, 67% of addresses in the B4/B6 variant encoded literal IPv4 addresses; the other server datasets also showed structure.Source: §5.2 · pp. 9–10, Figure 8 · p. 10

5.3 Routers

  • Router datasets used operator-specific patterns, including predictable low bits, mostly zero-valued regions, and literal IPv4 addresses in R4 interface identifiers.For R1, bits 28–64 largely distinguished prefixes and bits 124–128 interfaces; the authors caution that apparently random values need not come from a random generator.Source: §5.3 · p. 10

5.4 Clients

  • C1 contained dependent interface-identifier segments associated with Android devices from one vendor; other client datasets generally had pseudo-random IIDs and structure in bits 32–64.The authors suspected the C1 pattern arose from client-side IID selection in a particular Android version and vendor; they observed no similar pattern in other evaluated datasets.Source: §5.4 · p. 11

5.5 Scanning IPv6 Networks

  • 770K addresses and 46K prefixes were found across 10 networks; S3 had a 43% scanning success rate, while S1 yielded no new addresses.The evaluation used 1K training addresses per network; S1's largely pseudo-random IIDs were cited as the reason no new addresses were found there.Source: §5.5 · p. 12, Table 4 · p. 12
  • The scanning results included active /64 prefixes absent from training, complementing prior approaches that predict interface identifiers but do not guess network identifiers.The scan generated 1M candidates from models trained on 1K real addresses.Source: §5.5 · pp. 11–12

5.6 Predicting IPv6 Client Prefixes

  • Client prefix prediction found thousands of /64 prefixes per network, with success rates from ~1% to 20%; training on 1K BitTorrent prefixes also predicted 120K prefixes for C4.Prediction was applied to the top 64 bits because sampled client networks extensively used pseudo-random interface identifiers.Source: §5.6 · pp. 12–13, Table 6 · p. 12

5.7 Validation

  • Operator feedback varied: one confirmed service-related low bits, while another said the analysis misassigned a /32 because the authors assumed the entire block belonged to the network.The authors said the feedback showed they could improve integration of RIR data and BGP-advertised prefixes.Source: §5.7 · p. 13

6. LIMITATIONS AND FUTURE WORK

The authors identify limitations in their segmentation assumptions and in their treatment of address dependencies and time. They also see candidate-address hit lists as a practical direction for active IPv6 surveys.

6. LIMITATIONS AND FUTURE WORK

  • Segmentation assumptions, including a hard-wired /32 break and 4-bit alignment, can make inferred segments disagree with actual subnet identifiers and affect candidate subnets.The authors suggest removing fixed parameters or searching parameter space against operator ground truth.Source: §6 · p. 13
  • The Bayesian Network models dependencies between segments, but the study does not model dependencies among nybbles within a segment.The authors identify mutual information or length-normalized entropy as possible future approaches.Source: §6 · p. 14
  • Address sets are treated as active at one point in time, leaving temporal patterns and changes in network deployments for future analysis.The authors note that temporal analysis could help uncover sequential and random assignments from dynamic pools.Source: §6 · p. 14
  • Evaluation supports using Entropy/IP to generate candidate-address hit lists for active IPv6 surveys, including from datasets smaller than those used in evaluation.The authors identify surveys of server farms, routing infrastructure, and selected client networks as possible applications.Source: §6 · p. 14

7. CONCLUSION

Entropy/IP automates analysis of sampled IPv6 addresses to discover address structure. Its initial evaluation suggests potential for targeted surveys of the address space and for improving analysts’ understanding of Internet operation and defense.

7. CONCLUSION

  • Entropy/IP automates network-structure discovery by analyzing sample IPv6 addresses gathered through standard means.Examples include server logs, passive DNS, and traceroute.Source: §7 · p. 14
  • The system demonstrates effectiveness in discovering structure in both interface identifiers and network identifiers, including subnets.Source: §7 · p. 14
  • Initial performance evaluation suggests potential for surgically surveying the vast Internet address space and improving analysts’ understanding of Internet operation and defense.Source: §7 · p. 14
Loading 1606.04327v2…