Unsupervised Anomaly-based Malware Detection using Hardware Features

Adrian Tang; Simha Sethumadhavan; Salvatore Stolfo

Source-linked AI summary

Unsupervised Anomaly-based Malware Detection using Hardware Features

Adrian Tang, Simha Sethumadhavan, Salvatore Stolfo

arXiv:1403.1631v2Mar 7, 2014cs.CR

TL;DR

Existing hardware malware detectors rely on signatures of known malicious execution, leaving a need for detection that can handle unseen exploits. This paper learns normal execution profiles from performance-counter data and detects exploit-induced deviations, achieving near-perfect results on IE and Adobe PDF Reader while exposing vulnerability to mimicry attacks.

Problem
Signature-based hardware detectors compare execution against known malware patterns, motivating detectors that can recognize a wider range of attacks, potentially including novel ones.
Method
The paper uses unsupervised machine learning on performance-counter data to model normal program execution and applies power transforms, temporal aggregation, and selected event sets.
Results
100% of IE exploitation epochs were detected with 1.1% false positives and 1.5% slowdown, while Adobe PDF Reader reached an AUC of 0.999 with temporal modeling.
Takeaways & Limitations
Hardware anomaly-based detectors can complement signature-based predictors and may improve security by detecting novel, unseen malware exploits.
Takeaways & Limitations
Under generously optimistic assumptions, mimicry attacks can reduce detection performance by up to 6.5%, so anomaly detectors should not be the only defense.

Abstract

from arXiv · show

Recent works have shown promise in using microarchitectural execution patterns to detect malware programs. These detectors belong to a class of detectors known as signature-based detectors as they catch malware by comparing a program's execution pattern (signature) to execution patterns of known malware programs. In this work, we propose a new class of detectors - anomaly-based hardware malware detectors - that do not require signatures for malware detection, and thus can catch a wider range of malware including potentially novel ones. We use unsupervised machine learning to build profiles of normal program execution based on data from performance counters, and use these profiles to detect significant deviations in program behavior that occur as a result of malware exploitation. We show that real-world exploitation of popular programs such as IE and Adobe PDF Reader on a Windows/x86 platform can be detected with nearly perfect certainty. We also examine the limits and challenges in implementing this approach in face of a sophisticated adversary attempting to evade anomaly-based detection. The proposed detector is complementary to previously proposed signature-based detectors and can be used together to improve security.

1. INTRODUCTION

The paper introduces hardware anomaly-based malware detectors that learn normal execution profiles from hardware features and detect exploit-driven deviations without malware signatures. It evaluates this approach on vulnerable programs and considers evasion by sophisticated attackers.

1. INTRODUCTION: Hardware anomaly-based detectors learn baseline models of normal execution and identify malware through deviations in hardware-level behavior.They use microarchitectural features available through performance counters rather than pre-identified malware signatures.
1. INTRODUCTION: Anomaly-based detection can potentially identify novel attacks, whereas misuse-based detection is highly susceptible to small signature-deviating modifications.Misuse detection flags pre-identified execution signatures or attack patterns.
1. INTRODUCTION: Exploit code alters the victim program’s execution flow, producing observable perturbations that can support early malware detection.Detecting exploitation can provide more lead time than detecting a later payload.
1. INTRODUCTION: 100% of IE exploitation epochs were identified with 1.1% false positives at 1.5% sampling overhead using 512K-instruction epochs.Power transformation magnifies small deviations, while temporal aggregation separates exploit execution from larger program behavior.
1. INTRODUCTION: Under generously optimistic attacker and system assumptions, mimicry attacks reduce detection performance by up to 6.5%.The authors therefore position anomaly detection as part of an ensemble that can include signature-based predictors.

2. BACKGROUND

The background describes exploitation as a staged process in which shellcode hijacks a vulnerable program, downloads a payload, and executes it stealthily in memory. This structure explains why detection during exploitation is valuable.

2. BACKGROUND: A typical infection begins when an exploit triggers a vulnerability and transfers control to ROP and Stage1 shellcodes inside the victim program.The exploit commonly targets a memory-corruption bug delivered through a webpage or email attachment.
2. BACKGROUND: ROP shellcode reuses instructions from the original program binary to disable Data Execution Prevention.This code-reuse strategy circumvents DEP without directly executing untrusted data as code.
2. BACKGROUND: Stage1 shellcode downloads a larger Stage2 payload into memory and runs it through reflective DLL injection.The final payload may implement a backdoor, keylogger, or reconnaissance program.
2. BACKGROUND: Stage1 shellcode is small and fast because exploit writers face strict memory and reliability constraints.It is typically written in assembly and differs from the larger Stage2 payload in size, design, and function.
2. BACKGROUND: These exploits execute from memory in the host process context, write no files, and create no new processes, evading many file-based detectors.The described strategy is representative of many recent web exploit kits.

3. EXPERIMENTAL SETUP

The experiments build per-program baseline models from performance-counter measurements and evaluate whether shellcode stages produce detectable deviations. The study uses self-created exploits targeting common Windows applications and plugins.

3. EXPERIMENTAL SETUP: The study builds baseline per-program models with machine-learning classifiers and evaluates detection across operational factors.The central questions concern whether shellcode stages create observable deviations and whether they can be detected early.
3. EXPERIMENTAL SETUP: The authors created exploits for CVE-2012-4792, CVE-2012-1535, and CVE-2010-2883 targeting IE 8, Adobe Flash 11.3.300.257, and Adobe Reader 9.3.4.Metasploit was used because its exploitation techniques are representative of multi-stage real-world exploits.
3. EXPERIMENTAL SETUP: The generated exploits vary ROP shellcode, Stage1 shellcode, and the Stage2 payload.This variability is summarized in Table 1.
3. EXPERIMENTAL SETUP: Performance-monitoring hardware interrupts every N instructions to collect hardware-counter events and the executing process ID.Measurements were taken on a production Intel i7 Ivy Bridge system because architectural simulators could not run programs of this scale.
3. EXPERIMENTAL SETUP: The initial sampling interval was 512,000 instructions, chosen to provide sufficient measurements without excessive overhead.Each sample contains event counts, the process identifier, and the exploit-stage label.

Measurements

Measurements combine varied clean workloads with exploit runs collected under controlled conditions. The procedure reduces contamination and confounding while defining infection completion through process creation.

Measurements: Clean IE measurements vary websites, plugins, loading order, and browsing behavior to broaden normal-execution coverage.The workload includes the top 20 Alexa websites and additional plugin-using sites.
Measurements: 800 random PDFs were split evenly into training and testing sets for Adobe PDF measurements.The passage specifies 400 PDFs for each set.
Measurements: Infected and clean PDFs were matched in file-type distribution, including JavaScript content, while using Metasploit-generated exploits.This controls input differences between clean and exploit measurements.
Measurements: Measurements stop when a new process appears, indicating that the infection is complete after the target process becomes unstable or malware migrates.The new process supports persistence after Stage2 execution.
Measurements: VM restoration between runs keeps training and testing measurements independent and prevents contamination across clean and exploit runs.The same input sets are used for different measurements.
Measurements: Network bandwidth is throttled from 0.5 to 5Mbps through an internal Squid proxy to reduce latency-related confounding.The limits are varied during both exploit and clean runs.

4. BUILDING MODELS

The models are built from hardware performance-counter measurements by selecting discriminative event subsets, extracting temporal and non-temporal features, and learning clean execution profiles with unsupervised one-class classifiers.

Baseline modeling: Unsupervised learning builds baseline models from clean-environment hardware performance-counter measurements, avoiding the need for exploit-affected training data.The approach dynamically learns hidden structure in program-characterization data and uses it to detect deviations during exploitation.
Feature selection: 19 performance-counter events were shortlisted, then Fisher Scores ranked events by their ability to separate clean and infected measurements.The ranking was performed separately for malware execution stages and event categories, while the method does not account for feature dependence.
Feature selection: Nine candidate event sets were formed by selecting the top 4 events from each ranking row for IE baseline-characteristic models.Derived events such as %Misp Br were excluded because monitoring them requires two events and did not improve model efficacy.
Feature extraction: Non-temporal vectors contain four simultaneous event measurements from one epoch, while temporal vectors group N consecutive samples into 4N-feature vectors.The temporal extraction described here uses N = 4, producing 16-feature vectors.
Feature extraction: ROP shellcode was excluded from this feature-extraction test because it executes within one sample and contributes only a small portion of a grouped vector.This limits the temporal representation's coverage of the ROP stage.

5. RESULTS

The results show that malware-related deviations are difficult to see directly in skewed event distributions, but power transforms make some stage-specific differences more apparent for model evaluation.

5.1 Anomalies Not Directly Detectable: Most event distributions are positively skewed near zero, and exploit-induced deviations are not easily discerned from clean measurements.These observations explain why direct distribution inspection does not trivially reveal malware execution.
5.1.1 Power Transform: A rank-preserving power transform positively scales measurements to magnify slight deviations caused by malware execution.The transform is applied to alleviate the challenge posed by non-normal, heavily skewed distributions.
5.1.1 Power Transform: After the power transform, Misp Ret and Store show relatively larger deviations, especially during Stage1 exploit shellcode execution.These events are identified as likely having greater discriminative power for indicating malware code execution.
5.2 Evaluation Metrics for Models: ROC curves compare true-positive and false-positive rates, while AUC summarizes discrimination across classification thresholds.Non-temporal samples span one epoch; temporal samples span four epochs.

5.3 Detection Performance of Models

The models detect Stage1 shellcode most effectively, with temporal aggregation and selected architectural plus microarchitectural events improving detection. Coarser sampling reduces accuracy but can substantially lower monitoring overhead.

Different Stages of Malware Exploits: 99.5% detection accuracy is achieved for Stage1 shellcode with the AM-1 model.
Different Stages of Malware Exploits: ROP shellcode detection is mediocre because 512k-instruction sampling often dilutes its short execution with normal code measurements.ROP shellcode averages 2182 instructions, ranging from 134 to 6016 instructions.
Non-Temporal vs Temporal Modeling: Up to 59% AUC improvement is obtained for event set M-2 when temporal models replace non-temporal models.
Architectural vs Microarchitectural Events: 99.5% AUC is achieved for event set AM-1 by combining discriminative architectural and microarchitectural events.
Different Sampling Granularities: Coarser-grained sampling generally decreases detection performance because samples contain more measurements from context-switched processes.The reduction in sampling overhead at coarser rates nevertheless far exceeds the decrease in detection performance.

5.4 Results for Adobe PDF Reader

Adobe PDF Reader exploits are detected more accurately than IE exploits in the reported models, including near-perfect temporal AM-1 performance.

0.999 AUC is achieved for Adobe PDF Reader using temporal modeling with event set AM-1.ROP and Stage1 shellcode detection generally improves for Adobe PDF Reader compared with IE.

6. ANALYSIS OF EVASION STRATEGIES

The evasion analysis examines mimicry attacks that manipulate hardware-event characteristics to resemble benign execution. Padding and substitution are constrained, while grafting is powerful but operationally difficult, supporting layered defenses.

Mimicry attacks modify malware to exhibit architectural and microarchitectural characteristics resembling normal execution.
Adversary Assumptions: The adversary is assumed to know the target environment and modeling method, and to gather comparable HPC measurements, while exact modeled events may remain unknown.
Attack #1: Padding: Padding requires no-op instructions that jointly adjust monitored events toward ranges accepted by the models.
Attack #1: Padding: No-op padding cannot usually control events independently, and insertion into shellcode loops can sharply increase event counts and make execution appear more malicious.
Attack #2: Substitution: Substitution uses equivalent obfuscated code variants and a greedy HPC-measurement loop to retain variants acceptable to the models.
Attack #2: Substitution: The greedy substitution strategy eventually terminates, but whether its resulting shellcode modifications evade detection requires further examination.
Attack #3: Grafting: Grafting is considered the most powerful attack, but benign-code insertion can exceed exploit size limits, interfere with shellcode, or increase execution time.
Defenses: Deploying the detector with other anomaly-based predictors forces attackers to conform simultaneously to multiple normal-behavior constraints.

7. ARCHITECTURAL ENHANCEMENTS FOR DETECTION

The paper identifies hardware changes that could improve anomaly-based malware detection by increasing observability, reducing sampling overhead, supporting finer-grained analysis, and hindering evasion.

More performance counters: Adding more performance-counter events can better distinguish benign from malicious execution and potentially improve detection accuracy.The paper also suggests observing instruction and data working-set changes and basic-block execution frequencies.
Interrupt-less periodic access: Interrupt-less periodic counter access can eliminate interrupt-handling and sampling overhead while enabling finer-grained monitoring.Writing measurements periodically to a designated memory region allows direct sample access and reduces the noise effect.
Custom Accelerators: Finer-granularity sampling can improve detection accuracy, but increased data volumes may require energy-efficient hardware implementations.The current detector samples every 512K instructions and is implemented in software.
Secret Events: Using non-public microarchitectural events could make evasion attacks more difficult than when detectors rely on publicly available counters.The paper proposes a detector model analogous to hidden activity factors and logic in on-chip power controllers.

8. RELATED WORK

Related work applies hardware features to malware detection through misuse-based signatures, memory-bus monitoring, architectural modeling, and branch tracing, whereas this paper models protected programs and detects deviations.

Hardware malware detection: Prior hardware malware detection modeled microarchitectural signatures of Android malware, while this work models protected programs to detect deviations from their baselines.The paper also targets shellcode execution during exploitation rather than Android malware as complete programs.
Hardware malware detection: Other approaches detect malware installed in peripherals through additional memory-bus accesses or identify malicious program modification using architectural characteristics.These approaches address post-exploitation persistence or modification rather than the paper's anomaly-based exploit-detection setting.
Hardware-assisted exploitation defenses: Branch-monitoring facilities such as LBR and BTS have been used to detect runtime indirect-branch behavior and enforce control-flow integrity against exploitation techniques.The cited systems include kBouncer, CFIMon, and Eunomia.

9. CONCLUSIONS

The paper presents anomaly-based hardware detectors as complementary to signature-based methods and concludes that automated learning of program features supports malware detection while leaving room for stronger observability and evasion resilience.

Conclusions: Anomaly-based detectors use lower-level hardware features to detect exploits and can identify novel, unseen malware alongside signature-based predictors.The proposed combination is intended to improve security without relying exclusively on known malware signatures.
Conclusions: Recent hardware virtualization, exploit-reproduction toolkits, and computational capacity enabled performance-data collection and real-time machine-learning detection.These advances supported experimentation on infected production systems and practical detection strategies.
Conclusions: The detector achieved high-accuracy exploitation detection, but further gains may require additional counters, better algorithms, and hardware support against evasion.The paper specifically identifies non-linear classifiers and additional hardware support as opportunities for advancement.
Conclusions: Automated learning of program features may enable more performant and secure machines in the future.The paper frames this as a broader implication for computer-architecture research beyond malware detection.