PaperlayerView exact revision on arXiv

Source-linked AI summary

Learning the PE Header, Malware Detection with Minimal Domain Knowledge

Edward Raff, Jared Sylvester, Charles Nicholas

arXiv:1709.01471v2stat.MLcs.LG

TL;DR

Malware detection is challenging because executable formats are complicated and inconsistent, and PE parsing is difficult. This paper trains neural networks on raw bytes from a limited PE-header region using minimal domain knowledge. The networks match or surpass the domain-knowledge approach when both have equivalent information available.

  • Problem

    Malware detection is difficult because executable formats are complicated and inconsistent, and correctly parsing PE files is challenging.

    Source: §1 · p. 1

  • Method

    The authors train neural networks on raw bytes from a restricted PE-header region, using only the header length and PE-header offset to extract the bytes.

    Source: Domain Knowledge Needed for Other… · p. 4

  • Results

    Restricted to PE-header bytes, the networks matched or surpassed the domain-knowledge approach with equivalent information; Fully Connected accuracy rose 3 to 4 percentage points over that approach for Group B.

    Source: §5.1 · p. 8, §8 · p. 13

  • Takeaways & Limitations

    The results show that neural networks can learn from raw PE-header bytes and match or surpass a domain-knowledge approach with equivalent information.

    Source: §8 · p. 13

    The cause of the observed calibration fluctuations remains unresolved and is left for future work.

    Source: §7 · p. 13

Abstract

from arXiv · show

Many efforts have been made to use various forms of domain knowledge in malware detection. Currently there exist two common approaches to malware detection without domain knowledge, namely byte n-grams and strings. In this work we explore the feasibility of applying neural networks to malware detection and feature learning. We do this by restricting ourselves to a minimal amount of domain knowledge in order to extract a portion of the Portable Executable (PE) header. By doing this we show that neural networks can learn from raw bytes without explicit feature construction, and perform even better than a domain knowledge approach that parses the PE header into explicit features.

1 Introduction

The introduction frames malware detection as difficult because executable formats are complicated and inconsistent, making PE parsing challenging. The paper tests neural networks on raw bytes from a restricted portion of the PE header, reporting higher accuracy than its baselines without explicit domain-knowledge features.

1 Introduction

  • Malware detection is difficult because executable formats are complicated and inconsistent, and correctly parsing a PE file is challenging.The Windows operating system does not always enforce its specification, which may change, and malware may intentionally violate the PE standard or loading process.Source: §1 · p. 1
  • Whole-file byte n-gramming imposes a computational burden and has not performed as well as earlier results suggested.Byte n-gramming is also commonly used because it requires no domain knowledge.Source: §1 · p. 1
  • The study tests neural networks on raw byte sequences from the MS-DOS, COFF, and Optional PE headers.The Optional header is usually required despite its name.Source: §1 · p. 2
  • The authors report that their neural network achieves higher accuracies than the baselines without needing domain knowledge.They also investigate whether the networks learn concepts similar to those in the domain-knowledge approach from raw byte sequences.Source: §1 · p. 2

2 Related Work

Prior malware-detection work uses PE-header features, n-grams, and other engineered inputs, while this paper focuses on learning features from raw byte sequences. Related sequence-model work motivates the neural-network approach and distinguishes it from earlier RNN applications.

2 Related Work

  • PE-header features have supported effective malware classifiers, including prior systems using imports and other header fields.PE-Miner used 189 features and reported an overall AUC of 0.991 for distinguishing benign and malicious files.Source: §2 · p. 2
  • Neural networks have been applied to malware detection using byte n-gram inputs, but those approaches still require initial feature specification and selection.N-gram approaches retain partial loss of character-sequence information and become computationally burdensome as n grows beyond about 4.Source: §2 · p. 2
  • Earlier work on raw-input learning in images and text reported gains over classical approaches, motivating a domain-knowledge-free malware classifier [21].The paper notes potential flexibility for related domains such as iPhone, Android, or Linux malware, with minimal work.Source: §2 · p. 2
  • HMM malware approaches model byte n-grams or assembly op-codes, but their parameter requirements grow as O(S^m+1) for mth-order dependencies over S states.The authors contrast this with RNNs learning higher-order sequence dependencies and hierarchical features.Source: §2 · p. 3
  • Prior RNN malware detection used human-engineered features collected through dynamic analysis, whereas this work uses static analysis and seeks learned features.The paper states that, to its knowledge, no prior work had used neural networks for malware detection from raw byte sequences.Source: §2 · p. 3

3 Baseline Approaches

The section establishes PE-header-only baselines so the approaches have equivalent information available. Its primary domain-knowledge baseline parses selected header fields, while the other baseline uses the bytes themselves.

3 Baseline Approaches

  • The baseline section compares PE-header-only methods and limits the approaches to a shared byte range so their available information is equivalent.The primary baseline parses header features; the other baseline uses minimal-domain-knowledge methods on the extracted bytes, which also feed the neural networks.Source: §3 · p. 3

3.1 Domain Knowledge Approach

  • The primary domain-knowledge baseline extracts 115 PE-header features with PortEX, including 112 numerical and 3 categorical features.It uses the MS-DOS, COFF, and Optional headers; PortEX was selected because it handled the most malicious binaries in a small test and was easy to use.Source: §3.1 · p. 3
  • The authors tested Random Forests and Extra Random Trees with 100 trees each; boosted trees and non-tree methods performed worse or took longer to train.Tree-based models were favored because they support numerical and categorical inputs and are scale invariant.Source: §3.1 · p. 3

Domain Knowledge Needed for Other Approaches

The other approaches need only the header length and PE-header offset to extract a fixed byte region, with imports omitted to limit domain knowledge. The n-gram baseline represents that region through binary n-gram presence features and regularized logistic regression.

Domain Knowledge Needed for Other Approaches

  • The minimal extraction uses the 64-byte MS-DOS header and 264 bytes after the PE-header offset, for 328 bytes total, padding short files with zeros.The only domain knowledge needed is the header length and the PE-header offset at byte 0x3C; imports are excluded to minimize domain knowledge.Source: Domain Knowledge Needed for Other… · pp. 3–4
  • The raw-byte approaches must learn to recover the information represented by the parsed features from this lower-level representation.Comparable accuracy could help avoid practical difficulties in extracting parsed features.Source: Domain Knowledge Needed for Other… · p. 4

3.2 Byte N-Gramming Approach

  • The byte n-gram baseline uses binary presence-or-absence features from the PE-header byte region and Elastic-Net-regularized Logistic Regression.The regularization performs automatic feature selection; smaller C values reduce effective degrees of freedom and can force coefficients to zero.Source: §3.2 · p. 4

4 Neural Network Approach

The paper tests neural networks on low-level PE-header bytes, aiming to learn useful representations with minimal domain knowledge. It compares a fixed-length fully connected model with a sequential LSTM approach.

4 Neural Network Approach

  • Neural networks could learn how PE files are used in practice, reducing reliance on hand-built processing for corrupted headers and corner cases.The authors motivate this approach by noting that malicious files may violate format rules.Source: §4 · p. 4

4.1 Fully Connected Neural Networks

  • The fully connected model receives 328 ordered bytes, embeds each byte into a 16-dimensional vector, and concatenates them into a 5,248-dimensional input.The embedding is optimized during learning; larger embedding dimensions showed no performance difference.Source: §4.1 · pp. 4–5
  • Four fully connected hidden layers use dropout, batch normalization, ELU activations, DeCov, and weight decay, with L1 regularization on the first layer.The authors report that each added regularization technique increased the highest generalization accuracy achieved.Source: §4.1 · p. 5

4.2 Recurrent Neural Networks

  • The recurrent model uses three LSTM layers and attention, while variable-length sequence handling motivates RNNs as candidates for future domain-knowledge-free classification.The authors restrict the input to the PE header because processing a typical executable byte by byte would require millions of time steps and make LSTM learning computationally intractable.Source: §4.2 · p. 5, Figure 1 · p. 6
  • Each LSTM model took about 11 days to train, compared with less than an hour for the fully connected network; DeCov inhibited learning in this RNN task.The authors do not consider this an intrinsic incompatibility, having used DeCov successfully with RNNs on other tasks.Source: §4.2 · p. 5

Attention Mechanism

The attention mechanism adapts sequence attention to a single hidden-state sequence by combining local step information with a sequence-wide summary. It assigns normalized importance weights to time steps and uses them to produce a weighted average of hidden states.

Attention Mechanism

  • The mechanism adapts attention to one input sequence: each step's hidden state is considered alongside a summary of the whole sequence.The authors describe this as a simplification for sequence classification.Source: Attention Mechanism · p. 6
  • A one-layer network combines local and global hidden-state information to score each time step, and softmax turns the scores into weights that sum to one.The output is a weighted average of the hidden states.Source: Attention Mechanism · p. 6
  • Experiments with multi-layer attention networks inhibited learning and performed worse than the one-layer approach.Source: Attention Mechanism · p. 6

5 Experimental Methodology and Results

The study compares parsed PE-header features, header byte n-grams, and raw-header neural networks across test sets, training models on Group B data. The Fully Connected network outperforms the domain-knowledge approach, while results also show that training-set choice affects generalization.

5 Experimental Methodology and Results

  • Evaluation compares parsed-header Extra Trees and Random Forests, header-byte n-gram Logistic Regression, and raw-header Fully Connected and LSTM networks.The tree methods use PE-header features parsed into logical components; both neural networks use raw PE-header bytes.Source: §5 · p. 6
  • Group A-trained header-feature models overfit: Group B-trained models retain predictive ability on Group A, but the converse does not hold.The authors therefore use Group B training data and evaluate generalization on Group B, Group A, and Open Malware test sets.Source: §5 · p. 7, Table 2 · p. 7
  • Models train on Group B data and are tested on Groups A and B and Open Malware; Group A uses Microsoft binaries as benign data, while Group B is partner-sampled.Group B is intended to better represent common client-machine files.Source: §5 · p. 7
  • Neural networks use three LSTM layers and four fully connected layers; the selected hidden-state size is 256, and final models train for 35 epochs.The authors report that larger networks did not significantly improve performance and were less tractable; the best-performing epoch was selected for evaluation.Source: §5 · pp. 7–8

5.1 Model Results

  • On Group A, the feed-forward network dominates the ROC curves while 3-grams are dominated; on Group B, it nearly dominates, though Random Forest does better in a small region.The Group A plot is zoomed into the top-left corner because all models have high AUC.Source: §5.1 · p. 8, Figure 2 · p. 8, Figure 3 · p. 8
  • Fully Connected gains 3 to 4 percentage points in accuracy and 0.053 AUC on Group B over domain-knowledge methods; Attention LSTM trails those methods but beats byte n-grams.Table 3 reports final test-set scores, with balanced accuracy giving the two classes equal contribution.Source: §5.1 · p. 8, Table 3 · p. 9

Comparison with Whole File Byte N-Grams

PE-header information achieved generalization described as comparable to whole-file byte n-grams, though Group B test performance was lower than expected. The authors regard Group B as more representative of benign and malicious software populations, leaving that lower performance an issue.

Comparison with Whole File Byte N-Grams

  • Whole-file byte n-gram models trained on Group B reached balanced accuracies of 87.3%, 94.5%, and 81.1% on Group A, B, and Open Malware, respectively.The authors describe PE-header-only generalization as comparable at some level, while identifying Group B test accuracy as the significant difference.Source: Comparison with Whole File Byte N-Grams · p. 9
  • Group B test performance was slightly worse than on the other test sets, contrary to the expectation that same-source test data would perform best.The authors identify why these approaches differ in generalization as a question for future work.Source: Comparison with Whole File Byte N-Grams · p. 9
  • PE-header information generalized to new data especially well with an appropriate training set, but Group B performance remained lower than on the other test sets.The authors say the weaker Group B result remains an issue because they believe Group B better reflects benign and malicious software populations.Source: Comparison with Whole File Byte N-Grams · p. 9

6 Inferring Relative Feature Importance

The section compares feature importance from an Extra Trees model with byte-level attention from an LSTM to assess what the neural network learned. The analyses find substantial overlap, while noting limits in aggregate interpretation and in interpreting the Fully Connected network.

6 Inferring Relative Feature Importance

  • The analysis uses Extra Trees MDI and averaged LSTM attention to rank features, but the methods differ in their feature units and aggregate information.MDI uses impurity decreases weighted by the proportion of data reaching a tree node, while LSTM byte importance averages attention weights across training examples; the table gives Extra Trees features, byte ranges, and relative importance.Source: §6 · p. 10, Table 4 · p. 10
  • The LSTM selected three single-bit fields, which it cannot directly observe as raw bit values through its embedding layer.The selected fields were IMAGE_FILE_DLL, TERMINAL SERVER AWARE, and NO SEH; the authors view this as evidence that the LSTM can learn finer-grained meaning in byte information.Source: §6 · p. 10
  • For several multi-byte fields, the LSTM selected the most significant bytes, possibly indicating that large values matter more than lower-order bytes.Selected examples include the Certificate Table size, Subsystem, Debug table size, and Import Table size; the full CLR Runtime Header offset and size range was also selected.Source: §6 · p. 10
  • Only the first and last MS-DOS header bytes ranked among the LSTM's top 70, and their MDI was near zero in relative importance.The authors infer that the MS-DOS header did not play a major role in the decision process.Source: §6 · p. 10
  • LSTM attention emphasized bytes associated with nine of the ten highest-ranked features selected by the domain-knowledge approach.The authors report meaningful, though not perfect, overlap between the features the LSTM learns from raw bytes and those selected by Extra Trees.Source: §6 · p. 11
  • The authors expect the Fully Connected network learned similar features, but its feedback is less directly interpretable than the Attention mechanism.They leave inferring feature importance for other neural architectures in the malware domain to future work.Source: §6 · p. 11

7 Network Calibration

Calibration with a small number of labeled Group A test examples substantially improved accuracy for both networks, including the Fully Connected model with only 10 points. Across test distributions, however, accuracy fluctuations and accuracy–AUC discrepancies remain, and the cause of the calibration issue is unresolved.

7 Network Calibration

  • Fully Connected model AUC was stable and reached 0.91 on Group B and 0.98 on Group A, while accuracy fluctuated; Open Malware accuracy swung widely and has no AUC because it contains only malware.The authors report this behavior consistently across runs and also observe it for the LSTM; the figures distinguish AUC with dashed lines and balanced accuracy with solid lines.Source: §7 · p. 11, Figure 4 · p. 11, Figure 5 · p. 11
  • Group B AUC was lower than Group A AUC despite Group B being the training distribution; the authors argue Group A is objectively simpler, with most benign files from Microsoft Windows.They consider the higher Group A AUC reasonable for a model that generalizes well, but note that Group B better reflects benign and malicious software populations.Source: §7 · pp. 11–12
  • AUC can remain high despite poor accuracy because it measures ranking, whereas accuracy depends on the decision threshold; the authors report an example with AUC 0.96 and accuracy 60%.Their example illustrates how a poorly calibrated decision threshold can produce low accuracy even when rankings are strong.Source: §7 · p. 12, Figure 6 · p. 12
  • Accuracy swings indicate that the decision threshold is not always well calibrated across distributions: changes are minor on Group B, more pronounced on Group A, and especially large on Open Malware.Open Malware contains only malicious examples, so accuracy falls below 50% when the decision point favors benign labels and rises when it favors malicious labels.Source: §7 · p. 12
  • With 20 calibration points, Group A test accuracy rose from 54.1% to 93.3% for the Fully Connected network and from 57.5% to 75.1% for the LSTM.The Fully Connected network also performed better after calibration with just 10 labeled data points.Source: §7 · p. 12
  • The calibration issue's cause remains unknown, and Random Forest and n-gram models also show lower accuracy than expected from their AUCs.The authors suspect it may relate to the PE-header feature source rather than the model or feature representation, but leave determining the cause open for future work.Source: §7 · p. 13

8 Conclusions and Future Work

The networks classify executables from raw PE-header bytes without feature engineering, matching or surpassing a domain-knowledge approach with equivalent information. The authors propose extending the input and exploring recalibration for malware-family classification.

8 Conclusions and Future Work

  • Using a manageable range of PE-header bytes, the networks classified executables without feature engineering and matched or surpassed a domain-knowledge approach with equivalent information.The authors describe this as the first application of neural networks on raw byte-sequence features for this task.Source: §8 · p. 13
  • The model’s underlying representation was robust enough for recalibrating models to new domains, which may matter to organizations facing targeted or otherwise unique intrusion attempts.Source: §8 · p. 13
  • Future work will increase the information available to the network and pursue processing whole executables with minimal domain knowledge.Source: §8 · p. 13
  • The authors also propose investigating recalibration for malware-family classification, including whether a robust model can recognize new families from a few examples and estimate their spread without manually constructed signatures.Source: §8 · p. 13

Appendix A N-Gram Details

The appendix details how PE-header byte n-grams provide a strong baseline and compares their performance across n-gram sizes and datasets. Smaller n-grams performed better on Open Malware recall, while larger ones showed weaker cross-dataset generalization and greater overfitting.

Appendix A N-Gram Details

  • The n-gram analysis aimed to maximize byte n-gram performance as a strong baseline; warm starts made training across regularization values incrementally efficient and produced regularization paths.Source: Appendix A · p. 16
  • Restricting n-grams to 328 header bytes made experiments with sizes 2–6 feasible and allowed concatenating sizes to address fields stored in 1–4 contiguous bytes.The training data contained over 4 billion unique 6-grams, but only 1,218 occurring in at least 1% of headers; for 2-grams, the corresponding counts were 65,532 and 563.Source: Appendix A · p. 16
  • 5- and 6-grams had an almost 2-point advantage over 3-grams on Group B, but performed considerably worse on Groups A and Open Malware; 6-gram recall fell below 50%.Source: Table 5 · p. 16, Appendix A · p. 17
  • 2- and 3-grams performed best on Open Malware recall across n-gram sizes, but generalized less well to Groups A and B than larger n-grams and less well across datasets than domain-knowledge models.Source: Table 5 · p. 16, Appendix A · p. 16
  • Regularization paths show stronger overfitting for larger n-grams and support 2- and 3-grams over sizes of 4 or more.The paths compare cross-validation and test-set performance; the 2-gram and 3-gram plots are Figures 8 and 9, and the larger-size plots are Figures 10 and 11.Source: Figure 8 · p. 16, Appendix A · p. 16, Figure 9 · p. 17

Appendix B Keras Model Definitions

The appendix provides Keras code for a fully connected network and an LSTM attention network. Both consume the PE-header byte sequence; the LSTM model forms a weighted context from its hidden states before classification.

Appendix B Keras Model Definitions

  • Both code snippets use a 328-byte input sequence and a 16-dimensional embedding; the input variable represents the initial byte sequence.Source: Appendix B · p. 18
  • The fully connected network uses four dense layers, with batch normalization and ELU activations shown in its definition.Source: Appendix B · p. 18
  • The LSTM attention code computes attention weights over local hidden states, weights and sums those states into a final context, then applies a sigmoid output layer.Source: Appendix B · p. 18
  • The LSTM attention network uses four recurrent layers, with S denoting the number of neurons in each LSTM layer.Source: Appendix B · p. 18
Loading 1709.01471v2…