Source-linked AI summary
Deciphering Malware's use of TLS (without Decryption)
Blake Anderson, Subharthi Paul, David McGrew
TL;DR
Malware’s use of TLS limits plaintext-based detection, motivating methods that use observable encrypted-flow and handshake features instead. The paper studies malware and enterprise TLS behavior, accounts for sandbox bias, and finds that these features support detection and family attribution, including from a single encrypted flow.
Problem
Traditional pattern matching cannot inspect malware messages protected by TLS, creating a need to distinguish malicious from benign encrypted traffic while preserving encryption privacy.
Method
The paper analyzes TLS handshake, flow, packet-sequence, and byte-distribution features across malware sandbox data and enterprise TLS traffic, while accounting for sandbox bias.
Results
Malware TLS usage differs from enterprise usage, and family attribution reaches 90.3% accuracy from one encrypted flow and 93.2% using all flows in a 5-minute window.
Takeaways & Limitations
Passively observed TLS features can support malware detection and family attribution through rules or machine-learning classifiers without decrypting traffic.
Takeaways & Limitations
The results are most applicable to enterprise networks, and sandbox collection is biased by Windows XP, five-minute runs, and sample compatibility.
Abstract
from arXiv · showhide
The use of TLS by malware poses new challenges to network threat detection because traditional pattern-matching techniques can no longer be applied to its messages. However, TLS also introduces a complex set of observable data features that allow many inferences to be made about both the client and the server. We show that these features can be used to detect and understand malware communication, while at the same time preserving the privacy of benign uses of encryption. These data features also allow for accurate malware family attribution of network communication, even when restricted to a single, encrypted flow. To demonstrate this, we performed a detailed study of how TLS is used by malware and enterprise applications. We provide a general analysis on millions of TLS encrypted flows, and a targeted study on 18 malware families composed of thousands of unique malware samples and ten-of-thousands of malicious TLS flows. Importantly, we identify and accommodate the bias introduced by the use of a malware sandbox. The performance of a malware classifier is correlated with a malware family's use of TLS, i.e., malware families that actively evolve their use of cryptography are more difficult to classify. We conclude that malware's usage of TLS is distinct from benign usage in an enterprise setting, and that these differences can be effectively used in rules and machine learning classifiers.
I. INTRODUCTION
TLS obscures malware messages from traditional pattern matching, but its observable handshake and flow features support privacy-preserving detection, family attribution, and analysis of malware behavior.
- TLS encrypts network traffic while exposing parameters that support inferences about clients and servers.This creates observable signals without requiring plaintext inspection.
- ∼10% of malware samples in the dataset used TLS, making encrypted malicious traffic relevant to threat detection.TLS also makes deep packet inspection ineffective for those communications.
- 3,437 IPS rules included only 48 TLS-specific rules, with 6 detecting malware through strings in self-signed certificates.The rule-set analysis illustrates the limited investment in TLS-specific malware signatures.
- Malware typically used older and weaker cryptographic parameters than enterprise traffic, making its TLS usage distinct and classifiable.The study examined differences from both TLS client and server perspectives.
- 90.3% accuracy was achieved for malware family attribution from a single encrypted flow, rising to 93.2% using all encrypted flows within a 5-minute window.The attribution task used network-based data and treated each malware family as a class.
- The study combined sandbox malware data, enterprise TLS flows, flow features, packet statistics, byte distributions, and unencrypted TLS handshake data.It collected tens-of-thousands of malware samples and millions of enterprise TLS flows for comparison.
II. PRELIMINARIES AND ASSUMPTIONS
The study defines an enterprise-focused comparison using TLS flows collected from malware sandboxes and an enterprise network, while explicitly addressing sandbox and dataset assumptions.
- The analysis focused on TLS flows over port 443 to make enterprise and malicious TLS comparisons as unbiased as possible.The dataset contained 229,364 TLS flows across 203 ports, with port 443 the most common malicious TLS port.
- Results are most applicable to enterprise networks because the non-malware comparison data came from an enterprise network.The paper does not claim that its categorization and classification results generalize to service-provider data.
- The malware dataset comprised 18 families, 5,623 samples, and 25,793 encrypted flows that completed TLS negotiation and sent application data.Family labels were based on majority votes from VirusTotal signatures.
- The sandbox introduced operating-system and five-minute-run biases, which the study handled by analyzing affected cases separately or removing them.Accounting for this bias was described as essential to understanding malware TLS use.
- The authors removed flows matching the default Windows XP SChannel ciphersuite list when comparing general malware and enterprise TLS parameters.Family-level experiments retained such samples because the shared operating-system bias applied to all samples.
B. Feature Extraction
The feature-extraction pipeline combines conventional flow statistics, packet-sequence side channels, and representations derived from packet contents.
- The classifiers use traditional flow features, side-channel features, and features extracted from unencrypted TLS handshake messages.The extraction software processes live traffic or packet captures and exports data in JSON format.
- Flow features include inbound and outbound bytes and packets, source and destination ports, and total flow duration.These features are normalized to zero mean and unit variance.
- SPLT features represent the first 50 packets’ lengths and inter-arrival times, excluding zero-length payloads and retransmissions.The implementation collects packet-sequence information as a side-channel representation.
- A Markov-chain matrix models SPLT transitions after packet lengths and times are discretized into equally sized bins.Each matrix entry counts transitions between bins, and rows are then normalized.
3) Byte Distribution.:
The study represents encrypted flows using observable TLS handshake and traffic features, emphasizing byte distributions and client-side TLS differences between malware and enterprise traffic.
- 3) Byte Distribution.: Byte distributions count each payload byte value and can reveal encoding, header-to-payload ratios, application-header composition, and padding.The resulting 256 probabilities are used as machine-learning features.
- 4) Unencrypted TLS Header Information.: TLS client features include offered ciphersuites, advertised extensions, and client public-key length extracted from handshake messages.The public-key length comes from the client key exchange and depends on the negotiated ciphersuite.
- A. TLS Clients: Malware and enterprise clients differ in ciphersuite use, with malicious traffic more frequently offering the weak 3DES and RC4 suites.The comparison excludes flows matching the default Windows XP SChannel ciphersuite list.
- A. TLS Clients: Enterprise clients advertise more diverse TLS extensions, whereas malicious clients consistently advertise mainly signature_algorithms.Several extensions common in enterprise traffic are rarely observed in malicious traffic.
- A. TLS Clients: Enterprise traffic most often used 512-bit ECDHE_RSA keys, while malware almost exclusively used 2048-bit DHE_RSA keys.Client fingerprints also differed: enterprise configurations resembled recent browsers, whereas malware commonly matched older Opera, Firefox, and Tor configurations.
- A. TLS Clients: Across malware families, Internet Explorer 8 was the most frequent estimated client, while some families used Tor and others exhibited many distinct ciphersuite offer vectors.The Internet Explorer 8 fingerprint often reflected the underlying Windows environment.
B. TLS Servers
Malware-connected TLS servers differ from enterprise-connected servers in selected ciphersuites, extensions, certificates, and certificate subjects, providing family-level behavioral clues.
- B. TLS Servers: Approximately 90% of servers contacted by malware selected four ciphersuites, including 3DES, RC4, and DHE_RSA AES-256-CBC-SHA256 suites.These suites were rarely selected by servers contacted by enterprise hosts, and the RC4 suites are considered weak.
- B. TLS Servers: Malware-connected servers rarely selected TLS extensions, whereas enterprise-connected servers showed greater diversity, especially renegotiation_info and ec_point_formats.This pattern is consistent with the limited extensions advertised by malware clients.
- B. TLS Servers: Malware-connected certificate subjects often had DGA-like characteristics, although malware also communicated with inherently benign servers such as google.com.The certificate-subject distribution was long-tailed.
- B. TLS Servers: Self-signed certificates appeared in approximately 0.7% of malware sessions versus approximately 0.09% of enterprise sessions.The malware rate was roughly an order of magnitude higher.
- B. TLS Servers: Server behavior varied across families: Symmi and Dynamer contacted many unique IP addresses, while Virlock contacted one block.io address.Ten families used self-signed certificates, with ZBot the most frequent example.
- B. TLS Servers: Certificate subjects revealed family-specific tools and possible functionality, linking onion.to subjects with Tor usage and criteo.com with possible click-fraud.These are inferences from common certificate subjects and client configurations.
V. CLASSIFYING ENCRYPTED TRAFFIC
The classification experiments use logistic regression with an l1 penalty and compare feature subsets combining flow metadata, traffic patterns, byte distributions, and TLS information.
- V. CLASSIFYING ENCRYPTED TRAFFIC: Logistic regression with an l1 penalty was used for all classification results.
- V. CLASSIFYING ENCRYPTED TRAFFIC: The initial binary classifiers used different subsets of collected flow and TLS features.
- V. CLASSIFYING ENCRYPTED TRAFFIC: One classifier combined flow metadata, packet-length and inter-arrival sequences, and byte distributions, while another used only TLS information.
A. Malware versus Enterprise
The study evaluates malware-versus-enterprise classification using combinations of flow, side-channel, and TLS features, including a Windows XP sandbox-bias check. TLS header information materially improves detection, especially at a stringent false-discovery rate.
- Table V compares classifier accuracy across combinations of data features using overall accuracy and accuracy at a 0.01% FDR.
- 225,740 malicious and 225,000 enterprise TLS flows were used for the classification experiment.
- Using all available data views significantly improves classification results over models that omit TLS header information.
- Removing Windows XP SChannel flows leaves total accuracy unchanged but reduces performance at a 1-in-10,000 FDR by approximately 5%.
B. Malware Families
Malware families differ substantially in TLS-client behavior, and these differences affect detection difficulty. Older, weaker, stable configurations are easier to classify than varied or evolving cryptographic behavior.
- Combining flow metadata, side-channel information, and TLS features generally produced the best family-level models.
- Deshacop had the lowest reported true positive rate among families at 96.1%, while Tescrypt reached 97.6% among families resembling Windows XP SChannel clients.
- TLS-only performance can depend on operating-system artifacts: it worked well for Windows XP SChannel-like families but performed worst for most others.
- Virlock used one unchanged, Opera 12-like TLS client with outdated ciphersuites and was classified at 100% accuracy by all four classifiers.
- Dridex used multiple TLS clients and varied strong ciphersuites, making detection less effective than for Virlock.
- Self-signed server certificates provided an invariant that improved classification of Dridex despite its varying client behavior.
VI. FAMILY ATTRIBUTION
The paper measures similarity among 18 malware families using observable TLS-client parameters and finds structured groups alongside highly dissimilar families. TLS behavior therefore provides useful family-level attribution signals.
- 5,623 unique samples across 18 families generated 25,793 TLS flows that completed handshakes and sent application data.
- The similarity matrix uses offered ciphersuites, advertised extensions, and client public-key length as features.
- The matrix diagonal equals 1.0 because each family is perfectly self-similar.
- The matrix contains a Windows XP TLS-library cluster and a Tor 0.2.x-like cluster containing Skeeyah, Dynamer, Symmi, and Toga.
- Dridex and Virlock were the two most dissimilar families, although Dridex was difficult to classify and Virlock was trivial.
B. Multi-Class Classification
The paper treats malware-family attribution as multiclass classification from passive network data. A single encrypted flow yields strong but imperfect attribution, while aggregating flows in a short window improves accuracy.
- B. Multi-Class Classification: Using all data features achieved 90.3% accuracy for 18-class attribution from a single encrypted flow.
- B. Multi-Class Classification: Most TLS flows were attributed to their appropriate family with approximately 80–90% accuracy, except for Dridex and Virlock.
- B. Multi-Class Classification: Bergat and Dynamer were confused because similar communication patterns and packet-length sequences outweighed their differing TLS-client configurations.
- B. Multi-Class Classification: Yakes and Razy were difficult to distinguish because they connected mainly to baidu.com and exhibited mostly identical network behavior.
- B. Multi-Class Classification: A five-minute sliding-window majority vote increased multiclass accuracy from 90.3% for single flows to 93.2% for multiple flows.
VIII. LIMITATIONS AND FUTURE WORK
The study addresses sandbox and attribution limitations while showing that TLS-derived features support malware detection and family attribution. Future work includes broader environments and improved family labeling.
- Sandbox bias: Windows XP dependence and five-minute runs introduced bias, requiring explicit analysis of operating-system-reflective TLS features.The authors either removed affected cases or labeled and analyzed them separately.
- Attribution scope: Family attribution covered 5,623 unique samples across 18 families after family-name association reduced the original 20,548 TLS-using samples.The authors note that unreliable family labels reflect dynamic-analysis difficulty and propose robust clustering for future training data.
- Evasion and robustness: A motivated actor could evade detection by mimicking enterprise TLS parameters and certificates, so these methods are not exhaustive.The paper recommends combining TLS indicators with endpoint profiles based on unencrypted HTTP user-agent strings.
- Supported conclusions: Malware generally offers and selects weak ciphersuites and fewer extensions than enterprise clients, even after accounting for sandbox operating-system bias.The paper concludes that passively observed TLS features can support malware detection and family attribution through rules or classifiers.
- Results and future directions: 90.3% accuracy was achieved for attribution from a single encrypted flow, rising to 93.2% when all encrypted flows within five minutes were used.Some families share TLS configurations, while actively evolving families are more difficult to classify.
APPENDIX A CIPHERSUITE AND EXTENSION HEX CODES
The appendix provides hexadecimal mappings for ciphersuites and extensions used in the paper’s figures.
- Ciphersuites: Table VII maps hexadecimal codes to ciphersuites used in the figures.
- Extensions: Table VIII maps hexadecimal codes to extensions used in the figures.