PaperlayerView exact revision on arXiv

Source-linked AI summary

SAGEGAN: Style-Based Anomaly Detection with Gaussian Embeddings using Generative Adversarial Networks

Thesath Wijayasiri, Kar Wai Fok, Vrizlynn L. L. Thing

arXiv:2609.27357v1cs.CR

TL;DR

Rare or previously unseen malware variants weaken the assumptions of supervised multiclass classification, while analysts need models whose behaviour can be inspected and related to malicious-code characteristics. SAGEGAN converts binaries into three-channel images and trains a style-conditioned adversarial reconstruction model on benign samples, using layer-wise Gaussian styles and deterministic style-space analysis. On the self-collected test set, the Gaussian variant achieved 89.76% AUC and 88.19% balanced accuracy.

  • Problem

    Rare or previously unseen malware variants weaken the assumptions of supervised multiclass classification, while analysts need models whose behaviour can be inspected and related to malicious-code characteristics.

    Source: §I · p. 1

  • Method

    SAGEGAN converts binaries into three-channel images and trains a style-conditioned adversarial reconstruction model on benign samples, using layer-wise Gaussian styles and deterministic style-space analysis.

    Source: §I · pp. 1–2

  • Results

    On the self-collected test set, the Gaussian variant achieved 89.76% AUC and 88.19% balanced accuracy.

    Source: A. Primary results on the… · p. 7

  • Takeaways & Limitations

    The analyses suggest malware families differ in how anomaly is distributed across spatial scales and in how coherently each family is organized in the latent space.

    Source: §IX · p. 12

    The study covers static PE files, and its anomaly score is a ranking signal rather than a calibrated probability; adaptive perturbations and operational constraints remain unevaluated.

    Source: §VII · p. 9

Abstract

from arXiv · show

Malware evolves faster than rule-based and signature-driven detection pipelines. This paper presents SAGEGAN, a benign-only trained malware anomaly detection framework that converts portable executable files into compact three-channel images and models benign structure through style-conditioned adversarial reconstruction. The representation combines Hilbert-mapped byte values, benign-referenced byte-transition surprise, and entropy deviation from benign software. The model encodes each image into a layer-wise style tensor aligned with a seven-stage modulated generator, rather than a single latent bottleneck. A Gaussian style prior, moment-based prior alignment, and latent consistency are used to reduce mismatch between encoded benign styles and the generator's sampled manifold. For interpretation, a deterministic encoder pathway maps each executable to a fixed style tensor, enabling repeatable layer-wise family distance, gradient sensitivity, principal component, and class-behaviour analyses. On a self-collected portable executable corpus containing malware from 214 families, the Gaussian variant achieves 89.76% area under the receiver operating characteristic curve and 88.19% balanced accuracy, while the genome-style variant reaches 88.03% and 84.09%, respectively. Without refitting model weights, benign reference statistics, or decision thresholds, the same checkpoints are evaluated on DIKE, Microsoft BIG 2015, and Lester malware subsets. The results suggest that layer-wise style modelling supports both anomaly ranking and structured post hoc analysis of how malware families depart from the benign manifold.

I. INTRODUCTION

Unseen malware variants and the need for interpretable analysis motivate benign-only anomaly detection. SAGEGAN combines a byte-aware image representation with style-conditioned reconstruction and uses its deterministic style space for repeatable family-level analysis.

I. INTRODUCTION

  • Rare or previously unseen malware variants challenge supervised classification, while analysts need models whose behaviour can be inspected and related to malicious-code characteristics.Signature-based systems and handcrafted heuristics struggle against packed, obfuscated, and polymorphic binaries; dynamic analysis is costly to scale and susceptible to sandbox evasion.Source: §I · p. 1
  • SAGEGAN maps binaries along a Hilbert curve, encoding raw bytes, benign-referenced byte-bigram surprise, and deviation from benign entropy in three image channels.The Hilbert path is intended to preserve locality, keeping nearby elements in the byte sequence more likely to remain spatially adjacent in the image.Source: §I · p. 1, §II · p. 3
  • A deterministic encoder maps each executable to a fixed style tensor, supporting repeatable layer-wise family distances, sensitivity measures, and dispersion analyses.The style hierarchy is used to examine how anomaly separates across generator depths, not to identify exact source-code instructions or byte-level causes.Source: §I · pp. 1–2
  • Its Gaussian model encodes each image as a seven-layer style tensor and uses moment-based prior alignment and latent consistency to reduce mismatch with the generator's sampled style manifold.The layer-wise representation is aligned with the generator hierarchy rather than compressed into a single latent bottleneck.Source: §I · pp. 1–2

II. RELATED WORK

  • Prior malware-image research largely framed the task as supervised family classification, while adversarial anomaly methods did not directly address malware-byte statistics or benign-only screening.SAGEGAN instead combines literal byte structure with benign-referenced statistical cues and studies the encoder's deterministic style representation.Source: §II · pp. 2–3
  • SAGEGAN's related-work contribution is the combination of benign-only anomaly detection, layer-wise style inference, and deterministic style-space analysis for family-level interpretation.The paper positions this combination, rather than malware imaging or adversarial reconstruction alone, as its contribution.Source: §II · p. 3

III. METHODOLOGY

SAGEGAN converts binaries into three-channel images, then trains a style-conditioned reconstruction model on benign samples. The channels preserve byte structure and encode local and global deviations from benign reference statistics.

A. Overview

  • The two-stage pipeline converts binaries into three-channel images and trains a style-conditioned adversarial reconstruction model on benign samples only.At test time, reconstruction residual and discriminator response provide the anomaly score.Source: A. Overview · p. 3

B. Binary to Image Conversion

  • Hilbert mapping places raw bytes on a two-dimensional grid to preserve literal content and improve spatial locality; files are resized to 256 × 256 when needed.The red channel carries byte values, the green channel highlights transitions unlikely under a benign sequence model, and the blue channel centers file entropy on the benign mean.Source: B. Binary to Image Conversion · pp. 3–4
  • Benign reference statistics are computed from the benign training split and then frozen for subsequent validation and testing.The representation combines local byte-transition surprise with a global entropy deviation from benign software.Source: B. Binary to Image Conversion · p. 4

C. Model Architecture

SAGEGAN uses a shared style-conditioned encoder, generator, and discriminator backbone with seven generator modulation stages. Layer-wise styles are aligned to the learned Gaussian prior to connect benign reconstructions with the generator’s sampled style space.

C. Model Architecture

  • The seven-stage style hierarchy distributes latent influence across generator depths, supporting multiscale benign reconstruction and later layer-wise analysis.The backbone uses seven modulated convolution blocks and encodes each input as seven 512-dimensional style vectors.Source: 1) Shared backbone · p. 4
  • The encoder maps each image directly to a seven-layer style tensor aligned with the generator, rather than compressing it through a single latent bottleneck.This alignment puts anomaly scoring and post hoc layer-wise analysis in the same latent space.Source: 2) Architecture A: layer-wise Gaussian… · p. 5
  • Moment matching regularizes encoded benign styles toward the learned Gaussian sampler, while latent consistency re-encodes generated samples.These terms reduce latent drift and the risk that encoded styles occupy a reconstruction-only region disconnected from the generator’s adversarially trained style manifold.Source: 2) Architecture A: layer-wise Gaussian… · p. 5

3) Deterministic style inference for explainability:

SAGEGAN uses deterministic encoding for repeatable style-space interpretation, while its second variant replaces the continuous Gaussian prior with a discrete codebook. The genome-style model uses reconstruction and adversarial training without the Gaussian model’s alignment and consistency terms.

3) Deterministic style inference for explainability:

  • Deterministic encoding maps the same executable to the same style tensor given fixed weights and preprocessing, enabling repeatable layer-wise analyses.The reported analyses include family distance, gradient sensitivity, principal components, and family compactness.Source: 3) Deterministic style inference for… · p. 5
  • The Gaussian model’s deterministic style representation supports inspection of how malware families differ from benign software, not identification of exact source-code instructions or byte-level causes.Its layer-wise modulation hierarchy serves as an internal coordinate system for family analysis.Source: 3) Deterministic style inference for… · p. 5
  • The genome-style variant replaces the continuous prior with a discrete codebook that samples one learned embedding variant per gene.Applying Gaussian-model moment alignment and latent consistency degraded performance in early experiments, so this variant uses reconstruction and adversarial training only; it is not a strictly matched optimization objective.Source: 4) Architecture B: genome-style latent… · p. 5

D. Training Objectives

Training alternates discriminator, adversarial generator, reconstruction-based encoder-generator, and latent-consistency updates. At test time, an anomaly score combines reconstruction residual with an inverted discriminator logit, while a frozen validation-selected threshold determines balanced accuracy.

D. Training Objectives

  • Training alternates discriminator learning, adversarial generator learning, reconstruction-based encoder-generator learning, and latent-consistency learning.The stated Gaussian-variant weights are λrec = 10, λprior = 0.01, and λlat = 1.0.Source: D. Training Objectives · p. 5
  • Prior alignment and latent consistency reduce the risk that encoded benign styles occupy a reconstruction-only region disconnected from the generator’s adversarially trained style manifold.Training exposes the generator to both sampled prior styles and encoded benign styles.Source: D. Training Objectives · p. 5

E. Anomaly Scoring

  • At test time, the score combines reconstruction residual and an inverted discriminator logit; higher scores indicate stronger deviation from the learned benign manifold.The threshold for balanced accuracy is selected on held-out benign and malware validation data, then frozen for the self-collected test and transfer evaluations; AUROC uses continuous scores and does not depend on that threshold.Source: E. Anomaly Scoring · p. 5

IV. DATASETS AND EVALUATION PROTOCOL

The study evaluates Windows PE malware against benign executables using one benign-only training partition and frozen benign reference statistics. Its datasets span 214 malware families and three external malware sources.

A. Data sources

  • The five datasets comprise self-collected malware, benign executables, and the DIKE, Microsoft Malware Classification Challenge 2015, and disjoint Lester malware subsets.The external corpora provide evaluation across benchmark malware families and independently sourced collections.Source: A. Data sources · p. 6

1) Self-collected malware corpus:

  • The self-collected corpus contains 10,820 malicious PE files across 214 MalwareBazaar families, while the benign corpus contains 20,000 Lester executables matched to the malware size range.The malware files were restricted to 2–50 MB to reduce trivial size effects and keep image conversion comparable across sources.Source: 1) Self-collected malware corpus · p. 6, 2) Benign corpus · p. 6

2) Benign corpus:

  • The 20,000-file benign selection came from the Michael Lester collection and supplied entropy statistics used to reduce extreme covariate shift in the blue channel.Source: 2) Benign corpus · p. 6

B. Splits and leakage control

  • Only 10,000 benign files trained the model; the remaining benign files were split between validation and test, and malware never updated the model or benign reference statistics.Bigram and entropy references were computed from the benign training partition and then frozen.Source: B. Splits and leakage control · p. 6
  • A disjoint malware validation subset selected the single balanced-accuracy threshold without changing model parameters or benign preprocessing statistics.Final self-collected testing used the remaining malware and benign test split.Source: B. Splits and leakage control · p. 6

C. Primary and transfer test composition

The primary evaluation combines held-out benign files with malware, while transfer tests replace the malware source and retain the trained model and operating threshold. AUC and balanced accuracy are reported for the unbalanced mixtures.

C. Primary and transfer test composition

  • The primary test combines held-out benign files with held-out malware from the same overall source, testing benign-only separation in an intentionally imbalanced mixture.Source: C. Primary and transfer test composition · p. 6
  • Transfer tests replace the malware portion with DIKE, Microsoft Malware Classification Challenge, or disjoint Lester samples while keeping the benign test set fixed.Model weights, benign statistics, score normalization, and the decision threshold are not retuned on target datasets.Source: C. Primary and transfer test composition · p. 6

D. Evaluation metrics

  • AUC and balanced accuracy are reported for the unbalanced test mixtures; balanced accuracy weights positive and negative classes equally.AUC measures ranking quality independently of class prior and the fixed balanced-accuracy threshold.Source: D. Evaluation metrics · pp. 6–7

V. EXPERIMENTAL SETUP

  • The reported Gaussian model was selected from checkpoints within the first 500 epochs by validation AUC, with its threshold fixed before final testing.Unless otherwise stated, comparisons were checked across three seeds; Gaussian remained consistently ahead of genome-style on the self-collected test set, with a modest margin.Source: §V · p. 7

VI. RESULTS

SAGEGAN’s Gaussian variant leads the self-collected comparison and the tested generative baselines, while transfer results remain favorable under an unchanged evaluation setup. Comparisons with prior published methods are contextual rather than protocol-matched.

A. Primary results on the self-collected dataset

  • 89.76% AUC and 88.19% balanced accuracy were achieved by the Gaussian variant, versus 88.03% AUC and 84.09% balanced accuracy for genome-style.The self-collected test set comprised 5,000 held-out benign and 10,820 MalwareBazaar malware samples across 214 families.Source: Table IV · p. 7, A. Primary results on the… · p. 7

B. Comparison with additional generative baselines

  • The Gaussian SAGEGAN variant achieved the highest AUC among tested generative models, and both SAGEGAN variants exceeded the strongest non-SAGEGAN balanced-accuracy baseline.The genome-style variant also exceeded simpler GAN-based baselines; ECBiGAN was the closest competing encoder-generator anomaly detector.Source: B. Comparison with additional… · p. 7, Table V · p. 8

C. Zero-shot transfer

  • On DIKE, Gaussian exceeded genome-style by more than nine AUC points; both models performed near ceiling on Microsoft BIG and Lester, with a small Gaussian advantage.The transfer tests retained the same checkpoints, frozen benign statistics, and unchanged balanced-accuracy threshold.Source: C. Zero-shot transfer · pp. 7–8, Table VI · p. 8

D. Comparison with prior published methods

  • Prior-study comparisons appear favorable, particularly on Microsoft BIG 2015 and DIKE, but differing datasets and protocols prevent a definitive superiority claim.These literature comparisons are contextual, not strict head-to-head evaluations.Source: D. Comparison with prior published… · p. 8

E. Statistical validation of the anomaly score

The score analyses show useful ranking behavior, shifted but overlapping class distributions, and a relatively stable operating region. The discussion links this behavior to structured style modelling while noting limits on interpretation and evaluation scope.

E. Statistical validation of the anomaly score

  • The ROC curve shows useful ranking across operating points, scores are shifted but overlapping between classes, the selected threshold lies in a stable region, and risk bins preserve monotonic ordering without representing calibrated probabilities.The precision-recall curve remains informative under class imbalance.Source: E. Statistical validation of the… · p. 8, Figure 6 · p. 8, Figure 7 · p. 8

VII. DISCUSSION

  • The three image channels combine Hilbert-mapped byte structure with benign-referenced transition surprise and entropy deviation.Red captures coarse layout and repeated motifs, green highlights locally surprising transitions, and blue summarizes entropy deviation.Source: §VII · p. 8
  • The Gaussian model’s explicit encoder alignment and latent consistency appear to stabilize reconstruction and improve transfer, while genome-style remains a complementary discrete alternative.The present discussion does not position genome-style as a stronger replacement for the continuous variant.Source: §VII · pp. 8–9
  • Style modulation provides a layer-wise hierarchy for studying multiscale family separation, rather than identifying exact source-code instructions or byte-level causes.Source: §VII · p. 9
  • Transfer used no parameter refitting, benign-statistic recomputation, or threshold retuning, though high Microsoft BIG and Lester results may reflect easier separation from contemporary benign software.Source: §VII · p. 9
  • The study is limited to static PE files and lacks evaluation of adaptive perturbations, scanning latency, memory limits, and analyst triage workload.Packed or heavily obfuscated binaries also require separate robustness evaluation, and the score is a ranking signal rather than a calibrated probability.Source: §VII · p. 9

VIII. EXPLAINABILITY ANALYSIS

The explainability analysis uses deterministic encoder outputs to examine how malware families differ from benign software across the style hierarchy. It combines layer-wise distance, anomaly-score sensitivity, and family compactness with PCA projections.

VIII. EXPLAINABILITY ANALYSIS

  • All measures use the deterministic encoder output, giving each executable a fixed representation and making family analyses repeatable without sampling ambiguity.The method does not claim byte-level attribution, symbolic rule extraction, or guaranteed semantic disentanglement.Source: §VIII · p. 10

A. Layer-wise family distance

  • Family distance compares each malware family’s mean style vector with the benign mean at each layer, locating departures across the generator hierarchy.The distance is computed separately for each family and style layer.Source: A. Layer-wise family distance · p. 10

B. Gradient-based layer sensitivity

  • Gradient-based sensitivity measures how locally responsive the anomaly score is to perturbations of the deterministic style vector at each layer.The analysis treats this as post hoc sensitivity, not a causal explanation of the binary.Source: B. Gradient-based layer sensitivity · p. 10

C. Family compactness and principal component structure

  • Family compactness and PCA characterize within-family dispersion and whether benign samples and malware families occupy distinguishable regions of flattened style space.Lower compactness values indicate a tighter family region; higher values indicate broader variation or possible subgroups.Source: C. Family compactness and principal… · p. 10

D. Style-space analysis across malware families

Across the self-collected corpus and Microsoft BIG 2015, deterministic style analyses show family-level organization and nonuniform separation from benign software. Distance, sensitivity, and PCA results reveal different profiles across families and datasets.

D. Style-space analysis across malware families

E. Class behavioural patterns

The paper interprets malware family behaviour through distance, sensitivity, and dispersion profiles across the style hierarchy. It presents these as structured post hoc analysis, bounded by the fact that they do not establish causal mechanisms in binaries.

E. Class behavioural patterns

  • Families show late-layer, multiscale, or more diffuse profiles, which distinguish fine-grained irregularity from broader structural mismatch and varying within-family consistency.Compact PCA clusters indicate stronger internal regularity, while diffuse families may include broader variation or structurally distinct subgroups.Source: E. Class behavioural patterns · p. 12

F. Scope of the explainability claims

  • Class-behaviour patterns support post hoc interpretation and error analysis, while the analysis does not assign fixed semantics to layers or prove causal mechanisms in binaries.The contribution is a disciplined, repeatable analysis of family-level structure in the internal representation.Source: F. Scope of the explainability claims · p. 12

IX. CONCLUSION

  • SAGEGAN combines byte-aware images with style-conditioned adversarial reconstruction; its Gaussian layer-wise latent with moment-based alignment performs strongest on the self-collected benchmark and transfers without refitting.The explainability results suggest that families differ in how anomaly is distributed across spatial scales and how coherently they are organized.Source: §IX · p. 12
  • Future work will extend the framework to additional executable formats and dynamic traces, and examine gene-aware scoring and robustness under packing, obfuscation, and adversarial perturbation.Source: §IX · p. 12
Loading 2609.27357v1…