PaperlayerView exact revision on arXiv

Source-linked AI summary

Clusters in the Expanse: Understanding and Unbiasing IPv6 Hitlists

Oliver Gasser, Quirin Scheitle, Pawel Foremski, Qasim Lone, Maciej Korczynski, Stephen D. Strowes, Luuk Hendriks, Georg Carle

arXiv:1806.01633v2cs.NI

TL;DR

Exhaustive IPv6 scanning is infeasible, making the quality and representativeness of IPv6 hitlists central to Internet measurement. The paper combines diverse public sources with active longitudinal probing, aliased-prefix detection, entropy clustering, and crowdsourced client measurements. It finds a 50 M-plus-address hitlist with about half of addresses in aliased prefixes and six prevalent addressing schemes, while documenting important methodological and scope limitations.

  • Problem

    IPv6 cannot be exhaustively scanned, so measurement relies on hitlists whose representativeness can be affected by clustering and aliasing.

  • Method

    The paper constructs a public multi-source hitlist, probes responsiveness longitudinally, detects aliased prefixes, clusters addresses by entropy, and adds crowdsourced client addresses.

  • Results

    More than 50 M addresses were analyzed; about half reside in aliased prefixes, and entropy clustering identifies six prevalent IPv6 addressing schemes.

  • Takeaways & Limitations

    IPv6 hitlists should be curated for responsiveness, aliasing, source balance, and addressing structure rather than judged by address count alone.

  • Takeaways & Limitations

    Entropy-clustering results depend on the selected clustering parameters and tools, and cluster popularities reflect the hitlist more than the Internet itself.

Abstract

from arXiv · show

Network measurements are an important tool in understanding the Internet. Due to the expanse of the IPv6 address space, exhaustive scans as in IPv4 are not possible for IPv6. In recent years, several studies have proposed the use of target lists of IPv6 addresses, called IPv6 hitlists. In this paper, we show that addresses in IPv6 hitlists are heavily clustered. We present novel techniques that allow IPv6 hitlists to be pushed from quantity to quality. We perform a longitudinal active measurement study over 6 months, targeting more than 50 M addresses. We develop a rigorous method to detect aliased prefixes, which identifies 1.5 % of our prefixes as aliased, pertaining to about half of our target addresses. Using entropy clustering, we group the entire hitlist into just 6 distinct addressing schemes. Furthermore, we perform client measurements by leveraging crowdsourcing. To encourage reproducibility in network measurement research and to serve as a starting point for future IPv6 studies, we publish source code, analysis tools, and data.

1 INTRODUCTION

Because exhaustive IPv6 scanning is infeasible, hitlists provide practical targets but can be biased, clustered, aliased, and transient. This paper addresses these limitations through comprehensive discovery, entropy clustering, de-aliasing, longitudinal probing, visualization, diversification, and open sharing.

  • Motivation: IPv6’s enormous address space makes exhaustive scanning infeasible, so researchers use target lists called hitlists.IPv6 scanning is constrained by technical, storage, and ethical query-volume limits.
  • Motivation: IPv6 hitlists can be biased by imbalanced AS and prefix representation, address aliasing, oversized allocations, and short-lived address use.A single machine can respond across a large prefix, while some addresses may be used only briefly.
  • Approach: The paper builds a comprehensive hitlist from multiple state-of-the-art sources and uses entropy analysis to identify addressing schemes and aggregate clusters.The approach treats entropy fingerprints as a way to discover structure in the IPv6 address space.
  • Approach: A rigorous aliased-prefix detection method removes targets from prefixes where one machine may respond across many addresses.The method is intended to reduce aliasing’s impact on hitlist quality.
  • Approach: Longitudinal scans across several protocols identify reliably responsive addresses, since only a fraction of discovered addresses respond to probing.Responsiveness becomes an explicit filtering criterion for curating the hitlist.
  • Contributions: The work also introduces selective IPv6 visualization, evaluates address-generation and crowdsourcing methods, and publishes curated daily hitlists with code and data.These additions target exploratory analysis, address diversification, reproducibility, and future measurement studies.

2 PREVIOUS WORK

Prior work established that IPv6’s sparse address space requires targeted discovery rather than brute-force scanning. Existing approaches draw on DNS, structural address patterns, public and private hitlists, traceroutes, and crowdsourcing, while this work evaluates additional sources and capabilities.

  • Context: IPv4’s dense address space supports brute-force scanning, whereas IPv6’s sparse space motivates targeted measurement and comparisons among hitlist approaches.The paper situates its contribution against prior methods for discovering and measuring IPv6 addresses.
  • DNS techniques: DNS-based techniques have harvested IPv6 addresses from reverse-DNS trees, including 965 k addresses across 5531 ASes, with 56.7 % responsive in one study.Later work obtained 2.8 M IPv6 addresses from rDNS walking without probing them.
  • DNS techniques: The paper evaluates IPv6 rDNS as an active-measurement source and finds it a valuable addition to a hitlist.This addresses the earlier lack of overall responsiveness evaluation for rDNS-derived addresses.
  • Structural properties: Structural methods use rule mining or machine learning, including Entropy/IP, to infer IPv6 addressing schemes and generate new addresses.These methods exploit regularities inside otherwise sparse networks.
  • Hitlists: Earlier hitlists combined many sources but often relied on private data, whereas this work emphasizes publicly available sources for reproducibility.The comparison also tracks public and private addresses, client inclusion, active probing, and aliased-prefix detection.
  • Crowdsourcing: Crowdsourcing had previously collected only 38 IPv6 addresses in 2015, motivating broader use of client-oriented crowdsourced measurements.The paper builds on prior broadband-speed and IPv6-adoption measurement work.

3 IPV6 HITLIST SOURCES

The hitlist combines diverse public sources covering servers, routers, and clients, then analyzes their growth, distribution, clustering, and complementarity. Source composition reveals strong differences in AS and prefix representation, while visualization exposes over-represented prefixes and related structure.

  • Source selection: The authors select free, publicly accessible sources and seek balanced coverage of servers, routers, and clients to support reproducible hitlist construction.The source policy is designed to let other researchers deploy variations of the hitlist.
  • DNS-based sources: Domain lists yield about 9.8 M unique addresses, FDNS yields 2.5 M, CT contributes 16.2 M, and AXFR/TLDR contributes 0.5 M.These sources are derived from DNS names, forward lookups, certificate logs, and zone transfers.
  • Client and router sources: Bitnodes contributes 27 k client IPv6 addresses, while RIPE Atlas and ipmap add 0.2 M largely disjoint router addresses.Although Bitnodes is the smallest source, its client coverage makes it valuable.
  • Traceroute source: Scamper contributes 25.9 M unique addresses and grows strongly, with 90.7 % identified as SLAAC addresses.Vendor codes indicate that 47.9 % are ZTE and 47.7 % are AVM Fritzbox, suggesting mainly home-router and CPE equipment.
  • Visualization: Zesplots represent input IPv6 prefixes as rectangles, color them by address count, and expose possible over-representation without depicting the entire address space.Prefixes are ordered by prefix size and ASN, while unsized versions can reveal clusters.
  • Input distribution: AS distributions differ sharply across sources, and some prefixes contain unusually large address counts despite coverage of half of announced BGP prefixes.These patterns motivate further investigation of over-represented prefixes.
  • Comparison with DNSDB: The hitlist covers 12.9 % of DNSDB IPv6 addresses, 69.4 % of ASes, and 48.7 % of BGP prefixes, while remaining complementary to DNSDB.The authors attribute missing addresses largely to CDN coverage differences and do not add DNSDB because it is not public.

4 ENTROPY CLUSTERING

Entropy clustering represents IPv6 networks by per-nybble entropy fingerprints and groups networks with similar addressing patterns. Applied to the hitlist, it reveals a small set of recurring schemes, supports interpretation of DNS and BGP-prefix results, and remains parameterizable beyond the presented settings.

  • Method: The method computes normalized Shannon entropy across address nybbles, forming an entropy fingerprint for each network with at least 100 sampled addresses.Fingerprints can cover selected nybble ranges, and entropy 0 denotes a constant nybble while entropy 1 denotes equally probable values.
  • Method: K-means clusters networks by similar entropy fingerprints, while the elbow method selects k from the reduction in sum of squared errors.Each cluster is summarized by median entropy per nybble and relative popularity.
  • Results: 6 clusters emerge from full-address fingerprints, including counter-like, structured, pseudo-random-IID, and MAC-based addressing schemes.The most popular cluster has near-zero entropy except for a few terminal nybbles, while pseudo-random IIDs make address prediction impossible.
  • Results: 4 clusters emerge from IID-only fingerprints, with counter-based IIDs dominant and pseudo-random and MAC-based patterns associated with SLAAC.Counter-like IIDs are common in server pools and show potential for probabilistic scanning.
  • Results: DNS-responsive clusters usually have low entropy except at a few nybbles, making probabilistic scanning for IPv6 DNS servers easy.Entropy clustering is used to interpret scanning results and identify such addressing patterns.
  • Results: Larger BGP prefixes have more heterogeneous cluster mixtures, whereas smaller prefixes are more consistent, often indicating shared addressing schemes within an AS.The visualization orders prefixes by length and origin AS while using static box sizes for pattern spotting.
  • Discussion: Entropy clustering finds high-level patterns across networks and complements Entropy/IP, which finds low-level patterns within networks and can generate addresses for probing.The two methods can jointly identify networks susceptible to probabilistic scanning and produce hitlists for them.
  • Discussion: Results depend on clustering parameters and tools; the paper provides baseline settings while leaving alternative AS-, BGP-prefix-, and fingerprint-based analyses for future work.The technique is not confined to /32 prefixes or the fingerprint lengths presented.

5 ALIASED PREFIX DETECTION

The paper detects aliased IPv6 prefixes with fan-out probing, cross-protocol merging, and a short sliding window, then evaluates filtering and fingerprinting effects. The method addresses packet loss and prefix heterogeneity while substantially reducing address-count inflation, though validation remains indicative rather than conclusive.

  • Motivation: Aliased prefixes can artificially inflate IPv6 hitlists because many addresses may map to one server.This makes address count a poor proxy for distinct responsive systems.
  • 5.1 Multi-Level Aliased Prefix Detection: Fan-out probing sends pseudo-random probes across 16 more-specific subprefixes to reduce chance-driven aliasing classifications.The design distributes probes evenly while targeting addresses unlikely to respond by chance.
  • 5.2 Loss Resilience: Packet loss can create false negatives, so the method merges responses across protocols and uses a multi-day sliding window.The window requires an address to respond to any protocol during the preceding days.
  • 5.2 Loss Resilience: 3 days reduces unstable prefixes by almost 80%, leaving 14 of 909 aliased prefixes unstable.The short window is intended to limit delay when prefixes change nature.
  • 5.3 Impact of De-Aliasing: 29.4 M targets, or 53.4%, remain after filtering 55.1 M addresses, while AS coverage falls by only 13 ASes.Prefix coverage decreases 3.2%, from 25,465 to 24,648 announced prefixes.
  • 5.4 Fingerprinting Aliased Prefixes: Fingerprinting finds 13,202 of 20,692 prefixes with consistent behavior, supporting the interpretation that aliased prefixes have less diverse configurations.The validation also identifies 1,186 inconsistent prefixes excluding TCP timestamps, while many alleged non-aliased prefixes remain ambiguous.
  • 5.4 Fingerprinting Aliased Prefixes: The APD algorithm may produce false negatives because it does not probe low-density prefixes and because validation can also identify partially concentrated hosts.The authors report that fingerprinting is indicative rather than conclusive and do not feed it back into scanning.

6 ADDRESS PROBING

The study probes hitlist addresses daily across multiple protocols and evaluates responsiveness by prefix, protocol correlation, and stability over time. Responses are broadly distributed but show strong protocol-specific relationships and source-dependent persistence.

  • Measurement pipeline: The measurement pipeline preprocesses daily hitlists, removes aliased-prefix targets, traceroutes known addresses, and probes ICMP, TCP/80, TCP/443, UDP/53, and UDP/443.The process is repeated daily for longitudinal analysis.
  • Responsive addresses: 1.9 M responsive IPv6 addresses span 21,647 BGP prefixes and 9968 ASes.Most prefixes contribute dozens to hundreds of responsive targets, while a few contribute 12 k or more.
  • Cross-protocol responsiveness: At least 89 % of addresses responding to any probe also respond to ICMPv6, exceeding IPv4 values as low as 73 %.ICMPv6’s integral role in IPv6 is offered as a possible explanation.
  • Cross-protocol responsiveness: QUIC-responsive addresses are 98 % likely to provide HTTPS and HTTP, while HTTPS-to-HTTP correlation is 91 % and the reverse correlation is weaker.The HTTPS-to-HTTP correlation is higher than the reported IPv4 value of 72 %.
  • Cross-protocol responsiveness: DNS responsiveness resembles IPv4 overall, but DNS-to-HTTPS correlation is 54 % in IPv6 versus 78 % in IPv4.This is the stated exception to the broader similarity between IPv6 and IPv4 DNS correlations.
  • Longitudinal stability: Domain lists, FDNS, and RIPE Atlas retain responsiveness consistently over 14 days, while QUIC responses from CT and AXFR fluctuate more heavily.More than 80 % of fluctuating addresses are located in Akamai and HDNet prefixes.

7 LEARNING NEW ADDRESSES

The paper learns previously unknown IPv6 addresses by modeling addressing patterns with Entropy/IP and 6Gen, then probes the generated targets. The tools produce largely complementary address populations with different responsiveness and service profiles.

  • Generation method: Entropy/IP and 6Gen generate new addresses from non-aliased seed addresses split by AS, targeting probable patterns under a constrained scanning budget.Entropy/IP exhaustively walks its Bayesian network model, while 6Gen is used as a complementary generator.
  • Generated addresses: 239 M new unique addresses are learned: 116 M routable new addresses from Entropy/IP and 124 M from 6Gen.Only 675 k addresses overlap between the tools, equal to 0.2 % of all generated addresses.
  • Responsiveness: 785 k learned addresses respond to probes, a response rate of 0.3 %, underscoring the difficulty of finding responsive addresses through learning-based approaches.All learned addresses are probed on ICMP, TCP/80, TCP/443, UDP/53, and UDP/443.
  • Responsiveness: 6Gen finds almost twice as many responsive addresses as Entropy/IP: 489 k versus 278 k.The reported 6Gen response rate is a lower bound because the experiment uses random sampling rather than selecting its top generated addresses.
  • Complementarity: The tools share 17 k responsive addresses with a 2.5 % overlap response rate, compared with 0.3 % for the general learned population.The paper interprets this as evidence that the tools find complementary sets, while their small overlap is more likely to respond.
  • Host populations: 6Gen and Entropy/IP discover different responsive host populations: 6Gen has more ICMP-only hosts, whereas Entropy/IP finds relatively more DNS and QUIC-enabled web servers.The paper reports that Entropy/IP-responsive hosts are three times more likely to be DNS servers.
  • AS coverage: Responsive hosts span 1442 ASes for 6Gen and 1275 ASes for Entropy/IP, with 384 ASes found by only one tool.The paper therefore suggests running both tools to maximize discovered responsive addresses.

8 RDNS AS A DATA SOURCE

The study evaluates IPv6 reverse DNS as an additional active-measurement source because its responsiveness and bias characteristics had not been assessed comprehensively. rDNS contributes many new addresses with balanced structure and responsiveness broadly similar to the existing hitlist.

  • Motivation: IPv6 rDNS data are evaluated as a potential address source because harvesting them is costly and can strain important DNS infrastructure.The study compares rDNS structure and active responsiveness against other hitlist sources.
  • Address population: 11.1 M of 11.7 M rDNS addresses are new to the hitlist, while their prefix distribution is similar and AS distribution more balanced.The authors conclude that adding rDNS input would not introduce prefix or AS bias.
  • Active responsiveness: After filtering unrouted and aliased addresses, rDNS has a 10 % ICMP response rate versus 6 % for the non-aliased hitlist.HTTP(S) response rates are slightly lower for rDNS: 2 % (1 %) versus 3 % (2 %).
  • Client composition: Responsive rDNS addresses are not dominated by clients: top ASes are mainly hosting and service providers, and 60 % of TCP/80 addresses have IID hamming weight six or smaller.Only 6 %–9 % of responsive rDNS addresses show the evaluated SLAAC ff:fe sequence.
  • Conclusion: The responsive rDNS population adds a balanced set of IPv6 addresses, leading the authors to suggest adding rDNS data to hitlist inputs.

9 CLIENT IPV6 ADDRESSES

The study uses crowdsourcing to collect residential IPv6 client addresses, which are underrepresented in server- and router-heavy hitlists. Crowdsourced addresses add client targets, but most do not respond to incoming probes and disappear quickly.

  • Motivation and setup: Crowdsourcing is used to assess whether residential IPv6 addresses can supplement hitlists and support client-focused measurements.The study uses test-ipv6.com code through Amazon Turk and Prolific Academic.
  • Motivation and setup: 5781 Mturk users and 1186 ProA users participated, with IPv6 enabled for about 31 % and 20.6 % of users, respectively.The experiments ran from April 23 to May 23, 2018, with a $150 budget per platform.
  • Client distribution: IPv6 clients are concentrated in a few ISPs: Comcast accounts for 31.1 %, AT&T 13.2 %, and Reliance 7.8 %.The top five providers constitute only 30 % of IPv4 clients in the dataset, indicating greater IPv4 diversity.
  • Responsiveness: Only 352 addresses, or 17.3 %, respond to at least one ICMPv6 echo request.The majority of addresses gathered from residential networks do not respond to the probes.
  • Responsiveness: RIPE Atlas probes in the same ASes show a 45.8 % response rate, treated as an upper bound for crowdsourced responses.The authors suggest that local firewalls likely reduce user-device responses further.
  • Filtering: For 20 % of clients, the last responsive hop differs from the destination AS, indicating ISP filtering.
  • Stability and conclusion: Only 7 of 352 responsive addresses remain active for the full month; 39.4 % are active for 8 hours or less.The authors recommend probing crowdsourced addresses swiftly after collection because the responsive population shrinks quickly.

10 MEASUREMENT PRACTICES

The study applies ethical safeguards to active IPv6 measurements and makes its research artifacts publicly available to support reproducibility.

  • The researchers used a multiparty approval process to assess potential harms before active measurements.They limited query rates, used conforming packets, maintained a blacklist, and used dedicated servers.
  • The measurement study follows ethical scanning practices while limiting the likelihood of problems on scanned systems.The stated safeguards include constrained query rates and conforming packets.
  • The crowdsourcing study was reviewed by the university ethics committee, and participants could opt out of active probing.Collected crowdsourcing IPv6 addresses were excluded from public datasets.
  • The study publishes data, source code, and analysis tools to encourage reproducibility in network measurement research.Published artifacts include entropy clustering, the Entropy/IP generator, crowdsourcing documentation, and daily hitlist results.

11 DISCUSSION

The discussion emphasizes that IPv6 hitlists should be evaluated and tailored for responsiveness, stability, representativeness, and measurement goals rather than address count alone.

  • Time-to-Measurement: Client devices should be measured within minutes, whereas servers remain responsive over longer periods.The study reports that server addresses are more responsive and stable than CPE and client devices.
  • Hitlist Tailoring: Researchers should balance hitlists across prefixes and ASes and remove addresses in aliased prefixes to reduce measurement bias.The desired distribution can be adjusted toward address types or sources that match a study’s goal.
  • Unresponsive Addresses: Unresponsive addresses can still support analyses of prefix addressing schemes and address-learning algorithms.Algorithms such as Entropy/IP or 6Gen may use them to generate responsive addresses.
  • IPv6 Hitlist Service: Daily hitlists and aliased-prefix lists provide directly usable responsive-address subsets for future IPv6 measurements.Such subsets are useful when high-frequency periodic measurements cannot scan all known addresses.

12 CONCLUSION

The paper constructs a large public IPv6 hitlist, analyzes its aliasing and addressing structure, and uses longitudinal measurements to assess stability while continuing daily measurements.

  • More than 50 M addresses formed the largest IPv6 hitlist to date in the study.
  • About half of the addresses resided in aliased prefixes, and clustering identified six prevalent IPv6 addressing schemes.
  • Longitudinal measurements identified protocols and sources that were less stable over time.
  • The authors extended address-generation tools and found that they produced complementary address sets.
  • The project continues daily IPv6 measurements to provide updated hitlists.
Loading 1806.01633v2…