Source-linked AI summary
Novel Feature Extraction, Selection and Fusion for Effective Malware Family Classification
Mansour Ahmadi, Dmitry Ulyanov, Stanislav Semenov, Mikhail Trofimov, Giorgio Giacinto
TL;DR
The paper addresses malware-family classification under polymorphic and metamorphic variation, where large numbers of modified samples complicate analysis. It proposes compact content and structural feature extraction with selective feature fusion, achieving 99.77% accuracy on Microsoft’s malware Challenge dataset.
Problem
Polymorphic and metamorphic modifications create large numbers of malware variants, making malware-family categorization important for analyzing the volume of samples received by security practitioners.
Method
The method extracts content-based and complementary structural features directly from packed executables, then selects and fuses relevant feature categories using a limited feature set.
Results
99.77% accuracy was achieved on Microsoft’s malware Challenge dataset.
Takeaways & Limitations
The proposed system provides compact structural representations that support classification of obfuscated and packed malware without deobfuscation or unpacking.
Takeaways & Limitations
The method was not tested for robustness against evasion or poisoning attacks, and the feature design excludes PE-header information because it was unavailable under the Microsoft challenge rules.
Abstract
from arXiv · showhide
Modern malware is designed with mutation characteristics, namely polymorphism and metamorphism, which causes an enormous growth in the number of variants of malware samples. Categorization of malware samples on the basis of their behaviors is essential for the computer security community, because they receive huge number of malware everyday, and the signature extraction process is usually based on malicious parts characterizing malware families. Microsoft released a malware classification challenge in 2015 with a huge dataset of near 0.5 terabytes of data, containing more than 20K malware samples. The analysis of this dataset inspired the development of a novel paradigm that is effective in categorizing malware variants into their actual family groups. This paradigm is presented and discussed in the present paper, where emphasis has been given to the phases related to the extraction, and selection of a set of novel features for the effective representation of malware samples. Features can be grouped according to different characteristics of malware behavior, and their fusion is performed according to a per-class weighting paradigm. The proposed method achieved a very high accuracy ($\approx$ 0.998) on the Microsoft Malware Challenge dataset.
1. INTRODUCTION
The paper presents a malware-family classification system designed for large-scale categorization despite polymorphic and metamorphic variation. It combines compact content and structural features with selective fusion and evaluates the approach on Microsoft’s malware challenge dataset.
- Motivation: Polymorphism and metamorphism continually modify or obfuscate executables, creating many variants within malware families and complicating malware analysis.The paper motivates family classification by the growth of malware samples and variants.
- Motivation: Content-based and dynamic/static analysis techniques can suffer false detections or limited information when malware is polymorphic, packed, or lacks APIs and assembly instructions.The cited example is a packed sample with no APIs and few assembly instructions.
- Proposed approach: The system extracts content-based and complementary statistical features directly from packed executables, avoiding costly unpacking while representing structural characteristics.The approach uses structural PE-related information to classify obfuscated malware.
- Evaluation: 99.77% accuracy was achieved on Microsoft’s malware Challenge dataset, which the paper assesses as a current and reliable testbed for malware classification.The evaluation is performed on data from Microsoft’s Kaggle-hosted challenge.
- Proposed approach: The method uses a limited number of features compared with other state-of-the-art systems, supporting its intended use in large-scale malware categorization.The paper presents this reduced feature set as a contribution.
- Proposed approach: Feature fusion selects an effective concatenation of feature categories associated with different malware aspects instead of combining every possible category.This selection provides a trade-off between classification accuracy and feature count.
2. RELATED WORK
Related work divides malware classification and detection approaches broadly into dynamic and static analysis. The paper notes practical limitations of dynamic analysis and the diversity of static-analysis features used in prior research.
- Analysis paradigms: Malware analysis systems are broadly grouped according to whether they use dynamic or static analysis.The distinction is presented in the context of malware detection and classification before signature development.
- Dynamic analysis: Dynamic analysis can be affected by malware’s anti-analysis defenses and by incomplete code coverage across executable execution paths.These limitations concern environments detected by malware and the inability to explore all or multiple paths.
3. SYSTEM ARCHITECTURE
The system represents malware using complementary content and structural features extracted from hex and assembly views, then selects feature categories before classification. It includes entropy, image, string, section, and data-define features, and uses XGBoost with bagging while excluding unavailable PE-header information.
- Malware representation: The method extracts complementary features from hex and assembly views to represent malware samples for accurate and fast classification.The two views provide information related to malware content, structure, and obfuscation.
- Design constraint: PE-header features are excluded because the Microsoft Challenge dataset does not provide the PE header.The system instead extracts content- and structure-based features directly from packed executable files.
- Features extracted from hex dumps: Hex-based features include N-grams, metadata, entropy statistics, image textures, and string-length histograms.Entropy is computed over 10000-byte sliding windows, with quantiles, percentiles, mean, variance, and whole-file entropy included.
- Features extracted from disassembled files: Assembly-based features include metadata, section characteristics, and data-define instructions that capture packed samples lacking API calls.The data-define category uses db, dw, and dd instructions and is proposed because it has high discriminative power for some malware families.
- Feature fusion: The fusion algorithm adds entire feature categories using forward stepwise selection and stops when adding a category no longer decreases logloss.This avoids irrelevant features, reducing computational complexity and potential accuracy loss compared with stacking every category.
- Classification: XGBoost is used as the classifier, with bagging applied to improve classification quality.The paper motivates XGBoost by its reported performance, speed, efficiency, and tunable parameters.
4. EXPERIMENTS AND RESULTS
Experiments on Microsoft’s malware dataset evaluate feature importance, fusion, classification performance, computational cost, and comparison with the competition winner. The proposed system achieves high accuracy and low logloss while using fewer and less complex features.
- 4.1 Data: 21,741 malware samples from nine families were released, with 10,868 samples used for training and the remainder for testing.The dataset contains nearly half a terabyte of hex and disassembled-code data, with PE headers removed.
- 4.2 Feature importance: SEC and DP are among the most important feature categories according to mean decrease impurity in Random Forest classification.Mean decrease impurity measures the impurity decrease associated with each feature.
- 4.3 Results: 99.77% accuracy and 0.0096 logloss were achieved using all feature categories, compared with 99.76% accuracy and 0.0094 logloss using the best categories.The best-category combination was selected through the feature fusion algorithm.
- 4.3 Results: 0.0064 test logloss was obtained with the best feature categories, compared with 0.0063 using all categories.Because Microsoft did not provide test labels, evaluation used predictions submitted to the competition website.
- 4.4 Computational cost: Feature extraction and classifier training can be time-consuming for complex feature structures and large datasets, including the 65K-feature 2-Gram category.The experiments were run on a quad-core 2 GHz laptop with 8 GB RAM.
- 4.5 Comparison and Discussion: The method achieves performance close to the competition winner while using lower computational complexity in its features and classification technique.The paper contrasts its limited feature set and XGBoost with bagging against the winner’s larger n-gram feature set and more complex ensemble.
5. CONCLUSION AND FUTURE WORK
The paper concludes that a limited-complexity malware classification system can represent discriminant family characteristics with novel structural features. Its reduced complexity is motivated by industrial usability and interpretability considerations.
- 5. CONCLUSION AND FUTURE WORK: The system uses compact novel features and a limited-complexity classification mechanism to represent discriminant characteristics between malware families.The conclusion describes limited complexity in both feature design and classification.
- 5. CONCLUSION AND FUTURE WORK: Structural features are easier to compute than content-based features and can classify obfuscated and packed malware without deobfuscation or unpacking.The conclusion links structural representations to both computational simplicity and handling obfuscated samples.
- 5. CONCLUSION AND FUTURE WORK: A light system is motivated by industrial use, where the trade-off between complexity and performance can be important and fewer features may improve analyst understanding.The paper also notes that complex systems may provide negligible validation gains over less complex ones.