PaperlayerView exact revision on arXiv

Source-linked AI summary

MARS: Malware Analysis with Rule-Based Scoring of LLM Claims

Hyeongjun Choi

arXiv:2610.09553v1cs.CRcs.AI

TL;DR

Malware triage requires decisions from incomplete evidence, while comparisons between systems can confound decision strategies with differences in evidence collection and inputs. MARS compares direct classification with claim-based policy scoring on shared evidence across six models. Direct classification achieves higher overall accuracy for all six models and higher malicious alert recall in ten of twelve platform-model combinations.

  • Problem

    Comparisons of malware analysis systems are complicated by differences in evidence collectors, models, and inputs, while triage itself requires reconciling often-incomplete observations.

    Source: §1 · p. 1

  • Method

    MARS compares direct LLM classification with typed behavioral claims scored by an explicit policy, using identical static evidence bundles and a shared collector across six models.

    Source: §1 · p. 1

  • Results

    Direct classification has higher overall accuracy for all six models and higher malicious alert recall in ten of twelve platform-model combinations.

    Source: §1 · p. 1, §7 · pp. 15–16

  • Takeaways & Limitations

    Retained claims make verdict computation inspectable and revisable, allowing policy rescoring without another model call, but do not ensure stability when model-generated inputs change.

    Source: §7 · p. 16

    The perturbation study tests four fixed modifications to stored bundles and does not establish realizable binary changes or characterize adaptive manipulation of the collector and decision paths.

    Source: §5.4 · p. 15

Abstract

from arXiv · show

Large language models can triage malware through direct verdicts or behavioral claims scored by an external policy. We present MARS, a malware triage framework, and compare direct classification with single-pass claim scoring using the same evidence collector and identical static evidence bundles for each model. The evaluation covers 1,195 PE and ELF binaries grouped into 1,001 near-duplicate clusters and six language models, with deterministic rules providing a baseline. Direct classification is more accurate for all six models. On samples with usable outputs from both paths, its accuracy advantage ranges from 3.7 to 20.9 percentage points, with all 95% cluster-bootstrap confidence intervals for the differences above zero. It also achieves higher malicious alert recall in ten of twelve platform and model combinations. Claim mediation provides no consistent reduction in performance variation across models. Separate subset studies find more consistent alert decisions for direct classification and a larger recall loss for the claim path when predefined indicator fields are removed. In a family identification probe, claims yield higher accuracy than verdict labels but lower accuracy than evidence text. Retained claims expose the inputs to verdict computation and permit policy revision without another model call. We reproduce archived verdicts exactly and apply a revised policy to the same records, including outputs from two additional models withdrawn by their provider. Under the evaluated claim taxonomy and additive policy, these results favor direct classification when only a verdict is required, while demonstrating that retained claims support explicit policy inspection and revision.

1. Introduction

Malware triage combines incomplete, difficult-to-reconcile observations, and comparisons of analysis systems can confound decision strategy with differences in evidence collection and inputs. MARS compares direct classification with claim scoring on shared evidence; direct classification achieves higher overall accuracy across all six models.

1. Introduction

  • Direct classification leaves evidence interpretation and verdict assignment to one model response, while a plausible rationale does not establish that it faithfully reflects the decision process.Malware observations can be incomplete because packing conceals static content and dynamic analysis observes only behavior triggered in a particular environment.Source: §1 · p. 1
  • Typed claims make each claim’s contribution to the downstream score explicit and can be rescored under another policy without another model call, but do not establish claim correctness or eliminate generation variability.The evaluation also examines processing cost, family-information probes, repeated execution, policy replay, and evidence perturbation.Source: §1 · pp. 1–2
  • Holding the evidence collector and LLM constant addresses the attribution problem created when independently developed systems use different evidence and inputs.The comparison covers complete decision paths, including prompts and output schemas.Source: §1 · p. 1
  • MARS compares deterministic rules, direct LLM classification, and claim extraction followed by policy scoring on 1,195 PE and ELF binaries using basic static evidence.The two LLM paths are evaluated across six models; the claim path uses a confidence-free additive score and a model-specific benign threshold calibrated on a disjoint development set.Source: §1 · pp. 1–2
  • Direct classification has higher overall accuracy on all six models, with every model’s paired cluster-bootstrap 95% confidence interval excluding zero, and higher malicious alert recall in ten of twelve platform-model combinations.These findings apply to the evaluated evidence collector, claim taxonomy, and scoring policy.Source: §1 · pp. 1–2
  • In the family-identification probe, claim sets outperform verdict-only representations but score below raw evidence text, while direct justifications score close to claim sets.Repeated execution shows that a fixed scoring policy does not ensure end-to-end stability; indicator hiding produces greater recall loss on the claim path than on the direct path.Source: §1 · p. 2

2. Background

Malware analysis provides bounded observations rather than complete accounts of program behavior, so conclusions depend on the available artifacts. The background distinguishes explicit claim scoring from evidence support judgments and explains what retained claims do—and do not—establish.

2.1. Bounded Observation Context

  • Static artifacts may expose capabilities without showing whether they are exercised, and monitored execution records only behavior triggered under supplied conditions.Packing and analysis evasion can further restrict observations; holding evidence fixed permits comparison of decision strategies, though generated claims remain interpretations of available observations.Source: §2.1 · p. 2

2.2. Structured Outputs and Faithfulness

  • A fixed policy scores specified claim fields, making their contributions inspectable and replayable, but this procedural link neither establishes semantic correctness nor reveals internal reasoning.Reproducibility applies to a fixed claim set and policy; claim generation and severity assignment can vary across model calls.Source: §2.2 · p. 2

2.3. Claim Decomposition and Support

  • MARS support assessment labels claims supported, hypothesis, or unsupported according to their support in the supplied artifacts, but these model judgments do not establish behavioral correctness across executions.Extraction and assessment use the same model family and overlapping context, so their errors may be correlated; support states are annotations and do not affect verdict assignment.Source: §2.3 · pp. 2–3

3. Methodology

MARS holds evidence bundles constant while comparing rule-based, direct LLM, and claim-mediated decision paths for PE and ELF malware triage. Its methodology also specifies how evidence is collected, claims are scored, and archived outputs and policy sensitivity are examined.

3.1. System Overview

  • MARS compares deterministic rules, direct LLM labels, and scored claims across repeated-run, dynamic-evidence, and full-corpus studies.The full-corpus comparison uses basic static evidence; additional studies examine repeated runs and selected dynamic-evidence cases.Source: §3.1 · p. 3

3.2.1. Basic Static Evidence

  • Static bundles combine executable metadata, imports or dependencies, symbols, strings, and other signals; optional decompilation adds selected function bodies and relationships.Common fields group conceptually related PE and ELF artifacts while retaining platform-specific details; ground-truth labels are excluded from bundles and prompts.Source: §3.2.1 · p. 3, §3.2.2 · p. 3

3.2.3. Optional Dynamic Evidence

  • Dynamic collection observes PE or ELF activity during a 60-second run, but selected-case results do not estimate its benefit across the corpus.Unavailable runtime evidence is marked unavailable and does not establish behavioral absence.Source: §3.2.3 · pp. 3–4

3.2.4. Unified Evidence Bundle

  • Each sample and evidence setting receives one serialized, hashed, cached bundle, and matched models and paths use the same verified input.Fixed preprocessing limits and untrusted-data prompts apply before downstream branching.Source: §3.2.4 · p. 4

3.3.1. Direct LLM Classification; 3.3.2. Structured Claim Extraction

  • The direct path returns a label, confidence, and justification, while the claim path extracts atomic typed propositions with category, severity, and confidence.The scoring interface uses claim categories and severity; category weights are prespecified rather than learned from the evaluation corpus.Source: §3.3.1 · p. 4, §3.3.2 · pp. 4–5

3.3.3. Second-Pass Claim Support Assessment

  • An optional second pass annotates claims as supported, hypothesis, or unsupported but does not change categories, severities, scores, or verdicts for a fixed claim set.The independently executed configurations can still differ because each performs a new first-pass extraction.Source: §3.3.3 · p. 5

3.4. Fixed Verdict Policy

  • The fixed policy weights claims by category and severity, sums their contributions, and maps the score to labels using calibrated or specified thresholds.Confidence is excluded from the score; the main comparison’s policy differs from the fuller deployed procedure, which is not characterized by the evaluated support-assessment configuration.Source: §3.4 · p. 5

3.5. LLM Settings and Output Artifacts; 3.6. Scoring Policy Sensitivity

  • Run artifacts support reconstruction of scoring, while stored claims enable sensitivity analyses that vary weights and thresholds without additional model calls.Floating model tags and unavailable serving details constrain exact reproduction; threshold sensitivity also includes an evaluation-label oracle used only diagnostically.Source: §3.5 · p. 6, §3.6 · p. 6

4. Evaluation

The evaluation uses a clustered PE/ELF corpus and holds evidence and decision-path conditions fixed for the main comparison. It also includes exploratory dynamic-evidence and repeated-run studies with narrower sampling.

4.1. Dataset Construction and Selection

  • 1,195 evaluation binaries are grouped into 1,001 near-duplicate clusters, while corpus composition does not represent deployment prevalence and omits some difficult benign categories.The corpus is neither family- nor time-balanced; its benign collection does not exhaust potentially unwanted programs and administrative tools.Source: §4.1 · pp. 6–7

4.2. Experimental Design and Metrics

  • The main comparison processes each binary through rules and each model’s direct and single-pass paths using identical cached evidence and fixed claim-policy thresholds.The six-model panel uses basic static evidence; the claim thresholds are calibrated on a disjoint development set.Source: §4.2 · p. 7
  • The dynamic-evidence follow-up uses 20 selected Gemini 2.5 Flash cases, but its selection rule is not reproducible, so it is descriptive rather than representative.The repeated-run study samples 15 benign and 15 malicious binaries per platform and runs three paths ten times for three models; support assessment is not evaluated on the full corpus.Source: §4.2 · p. 7
  • Accuracy treats unknown outputs and errors as incorrect, while alerts combine suspicious and malicious labels; paired accuracy intervals resample near-duplicate clusters.Individual rate intervals use Wilson intervals, and stability is agreement across ten runs on either the four-class label or binary alert outcome.Source: §4.2 · p. 7

4.3. Verdict Results

Direct classification leads the paired accuracy comparison across the six-model panel, while platform results show exceptions in recall and trade-offs in benign labeling. The ELF rules reach the alert boundary but do not alert on any malicious sample in this corpus.

4.3.1. Shallow-Metadata Baselines

  • 93.6% PE and 88.8% ELF balanced accuracy: the metadata classifier exceeds mean single-pass performance on both platforms and mean direct performance on PE.Architecture alone reaches 86.9% on PE and 84.3% on ELF; the metadata classifier uses shallow features and is trained on corpus folds, unlike MARS’s no-fine-tuning setup.Source: §4.3.1 · p. 7, Table 5 · p. 8

4.3.2. Results by Platform

  • Direct classification has higher malicious alert recall in ten of twelve platform/model combinations, with exceptions for Gemma4-31B on PE and DeepSeek-V4-Pro on ELF.On PE, Gemma4-31B claims raise recall from 76.7% to 96.7% while benign exact-label rate falls from 98.3% to 72.4%; Gemini 2.5 Flash exhibits the reverse trade-off.Source: §4.3.2 · p. 7, Table 6 · p. 8

4.3.3. ELF Rule Coverage

  • 0.0% malicious recall on ELF: the rules-only baseline’s alert boundary is reachable, but no malicious ELF sample in the corpus reaches it.Recomputing the rules reproduces all 598 recorded labels; no malicious sample exceeds 1.5, while 40 benign samples reach the 3.0 alert threshold.Source: §4.3.2 · p. 7, §4.3.3 · p. 8, Figure 3 · p. 9
  • Only three YARA rules match ELF bundles, and none of the 298 malicious ELF samples with a YARA match also receives the symbol bonus.The implemented bonus predicate misses the taxonomy’s process-manipulation category; this identifies limited coverage by these rules, not an inherent lack of information in ELF binaries.Source: §4.3.3 · p. 9

4.3.4. Paired Comparison

  • 3.7–20.9 percentage points: direct classification leads single-pass accuracy on paired usable outputs, with all six cluster-bootstrap intervals above zero.The ordering also persists with errors retained in the denominator and when the claim threshold is selected using evaluation labels; archived Gemma3-27B results instead favor claims by 16.3 points.Source: §4.3.4 · p. 9, Table 7 · p. 9

4.4. Checks on the Verdict Comparison

Checks within architecture strata and on a metadata-hard subset preserve direct classification’s accuracy advantage. Removing collector-failure cases changes the absolute metrics but does not explain the ordering.

4.4.1. Fixed Architecture Strata

  • Direct classification retains higher mean balanced accuracy in all four architecture strata, leading in 21 of 24 model/stratum combinations.Margins are 12.1 points on PE x64, 24.3 on PE x86, 7.6 on ELF x86-64, and 4.4 on ELF x86-32; other metadata cues remain available within strata.Source: §4.4.1 · pp. 9–10, Table 8 · p. 10

4.4.2. Metadata-Hard Subset

  • 11.2 percentage points: the median direct-classification accuracy gap on the metadata-hard subset, compared with 9.8 points on the full corpus.The subset is defined by low confidence from this metadata classifier; the result does not establish difficulty independent of that classifier or demonstrate behavioral reasoning.Source: §4.4.2 · p. 10, Table 8 · p. 10

4.4.3. Collector Failures

  • Excluding 11 malicious samples that fail in both LLM paths leaves direct classification higher in recall in ten of twelve combinations and higher in accuracy for all six models.The largest change in an accuracy gap is 0.20 percentage points; the collector defect affects absolute performance but does not explain the ordering.Source: §4.4.3 · p. 10

4.5. Family Information in Retained Outputs

Claims retain family-discriminative content beyond verdict labels, but evidence-bundle text scores higher than claims in the probe. Claim processing generally costs more under assumed rates, while selected dynamic-evidence cases show mixed effects.

4.5. Family Information in Retained Outputs

  • 61.1% PE and 83.1% ELF: evidence-bundle text has higher family-identification accuracy than every claim representation, while claims outperform verdict-only representations.Claim accuracy ranges from 44.2% to 57.9% on PE and 72.3% to 81.9% on ELF; the probe uses matched platform-specific samples and cluster-grouped cross-validation.Source: §4.5 · pp. 10–11, Table 9 · p. 11

4.6.1. Full-Corpus Processing Cost

  • 1.20–1.41 times: claim processing’s estimated token cost relative to direct classification on five models under the assumed rates.Median latency is 1.3–2.3 times as high on those models; Qwen3.5-397B reverses the pattern, with lower claim-path latency and slightly lower estimated cost. Estimates do not establish actual deployment expenditure.Source: §4.6.1 · pp. 11–12, Table 10 · p. 12

4.6.2. Dynamic Evidence on Selected Cases

  • On selected PE cases, adding runtime observations raises malicious alerts from 5/7 to 7/7 for single-pass processing and from 4/7 to 6/7 for support-assessed processing.ELF effects are mixed: support-assessed alerts rise from 4/7 to 6/7, while single-pass alerts fall from 6/7 to 5/7; the selected cases do not support a corpus-wide estimate.Source: §4.6.2 · p. 12

4.7. Repeated-Run Stability

Across repeated runs, direct classification retains an accuracy advantage, while verdict consistency varies and changing outputs are associated with greater score and claim-count variation. Direct classification also has the highest alert consistency across the reported platform and model combinations.

4.7. Repeated-Run Stability

  • Exact verdict consistency ranged from 36.7% to 93.1%, and direct classification had the highest alert consistency in every platform and model combination.The claim configurations used independent first-pass extractions, so their consistency differences reflect claim-generation variation, not the effect of support assessment.Source: §4.7 · p. 12
  • Exact verdict consistency was lower for malicious samples in 17 of 18 combinations; most configurations were more consistent at the alert boundary than at the four-class boundary.The exception was direct classification with Gemini 2.5 Flash on PE, with 40.0% consistency for benign samples and 86.7% for malicious samples.Source: §4.7 · p. 12
  • Changing samples had greater score and claim-count variation on both platforms, while the PE threshold relationship was absent on ELF.On PE, changing samples were closer to the benign threshold than stable samples for DeepSeek-V4-Pro and GLM-5.2: median distances were 0.28 versus 3.40 and 0.80 versus 3.30, respectively.Source: §4.7 · p. 12

4.7.1. Variation in the Accuracy Difference

  • Across all 30 model-run combinations, direct classification favored direct classification over single-pass processing, with a minimum accuracy difference of 8.5 points.Median differences were 16.9 points for Gemini 2.5 Flash, 17.8 for DeepSeek-V4-Pro, and 11.9 for GLM-5.2; within-model ranges spanned 8.4–15.2 points.Source: §4.7.1 · p. 12

4.8. Replay, Rescoring, and Evidence Perturbations

Archived records permit exact replay of the evaluated claim policy and rescoring under a revised policy, while renewed model calls can vary. In the tested evidence perturbations, recall losses differed by decision path, and effects depended on the modification and model.

4.8.1. Replay and Rescoring

  • On renewed direct-classification calls for malicious samples, exact label agreement was 91.8% and alert agreement was 98.0%.The 1,759 paired records had identical bundle hashes; results combine decoding variability and possible provider-side model changes and apply only to malicious samples.Source: §4.8.1 · p. 12
  • Recomputing the evaluated policy from archived records reproduced the recorded labels in all 9,456 cases, including 2,366 from two withdrawn models.This verifies deterministic score replay, not stability under renewed LLM calls; the direct label can also be retrieved unchanged.Source: §4.8.1 · pp. 12–13
  • A revised policy agreed with the deployed default on 4,682 of 9,456 labels (49.5%), and archived claims enabled comparison without renewed model access.The disagreement reflects a policy change, not a replay failure.Source: §4.8.1 · p. 13

4.8.2. Evidence Perturbations

  • Under S1, malicious alert recall fell 34.1 points for rules only, 29.2 for single-pass processing, and 10.0 for direct classification.The larger single-pass loss occurred in all four architecture strata; the study does not isolate claim extraction, taxonomy, and scoring as separate causes.Source: §4.8.2 · p. 13, Table 12 · p. 14
  • S2 reduced direct-classification recall by 4.4 points, while S4 pooled recall intervals spanned zero despite lower benign exact-label rates for both LLM paths.Under S4, benign exact-label rates fell by 5.5–11.0 points for direct classification and 4.4–9.9 for claims; one single-pass configuration gained 16.7 points. Under S3, neither pooled recall interval excluded zero, and single-pass losses varied by model. The experiments used four fixed bundle modifications on a 182-file subset and do not establish robustness to binary changes or adaptive manipulation.Source: §4.8.2 · p. 13

5. Discussion

Under matched evidence conditions, direct classification outperformed claim mediation in accuracy across the six-model panel, while claims retained policy inputs for inspection and revision. Policy sensitivity, system-design implications, and validity boundaries qualify these comparisons.

5.1. Comparison of Decision Architectures

  • Direct classification had higher accuracy across the six-model panel, with positive cluster-bootstrap intervals for every paired difference, while claim mediation showed no consistent stability or uniformity advantage.Archived Gemma3-27B results favored claims when direct classification produced substantial benign escalation, so the ordering did not extend to every model examined.Source: §5.1 · p. 14
  • In the family probe, claims outperformed verdict labels but scored below evidence-bundle text, while retained claims expose policy inputs for inspection and revision without another model call.Direct justifications achieved scores similar to claim sets.Source: §5.1 · p. 14

5.2. Sensitivity of the Fixed Policy

  • With joint weight perturbations up to ±50% and threshold refitting, the 5th–95th percentile evaluation balanced-accuracy interval was 3.7–8.5 points wide across models.Mean perturbed performance remained within 2.5 points of baseline for every model; individual sweeps changed panel mean balanced accuracy most for benign indicators (3.0 points), suspicious API usage (2.5), and capability hints (2.0).Source: §5.2 · p. 14
  • Development balanced accuracy exceeded evaluation balanced accuracy by 10.6–18.3 points, and choosing thresholds on evaluation labels still left direct classification ahead by 2.2–16.1 points across models.Evaluation-label threshold selection raised single-pass balanced accuracy by 0–5.1 points, with a median improvement of 1.5; these analyses do not identify the cause of the development–evaluation gap or bound other policies.Source: §5.2 · p. 14

5.3. Implications for System Design

  • Direct classification is the primary baseline when only a sample-level label is required; claim mediation fits systems needing explicit, revisable rules over retained outputs.Weights and thresholds can change independently of claim generation, but post-processing direct confidence or justifications creates a new decision rule rather than reconstructing the original label computation.Source: §5.3 · p. 14

5.4. Limitations and Threats to Validity

  • The corpus is not balanced by family or collection time, does not represent operational prevalence, and leaves difficult benign categories incompletely covered.Unmasked identity cues and strong metadata baselines limit attribution to behavioral interpretation; family annotations were not independently verified, and verdict accuracy was not benchmarked against external detectors or production services.Source: §5.4 · p. 14
  • Matched evidence and models do not isolate representation format because prompts, schemas, budgets, parsing, and verdict assignment differ between paths.Collected evidence was not validated against independent reverse engineering, and the study does not measure semantic claim precision, completeness, expert agreement, or analyst effects; linkage to score computation does not establish claim correctness or usefulness.Source: §5.4 · p. 15
  • Perturbation findings apply to four fixed bundle modifications, not binary changes or adaptive manipulation, and archived records cannot recover withdrawn model services or guarantee future outputs.The perturbation results do not isolate the causal contribution of extraction, taxonomy, or scoring; floating model tags also limit exact reexecution for the remaining panel.Source: §5.4 · p. 15

6. Related Work

Prior work applies LLMs to malware analysis, binary understanding, and multi-model orchestration, but studies different tasks or pipeline choices. This study compares decision architectures for native PE and ELF triage using shared evidence and a corpus.

6. Related Work

  • Prior malware-analysis work includes static analysis, deobfuscation, behavioral summaries, and Android categorization, whereas this study evaluates evidence from native PE and ELF binaries.The cited Android studies use application code and platform-specific evidence.Source: §6 · p. 15
  • Binary-understanding work evaluates decompilation, function-name recovery, code summarization, and behavior localization, while MARS studies how behavioral claims contribute to a whole-sample security verdict.TTPDetect focuses on recognizing and localizing behaviors within decompiled functions.Source: §6 · p. 15
  • Concept bottleneck models use human-specified concepts, unlike MARS claims generated by a general LLM without concept-level supervision and scored by a prespecified policy.Other orchestration work compares coordination among models, whereas this study examines where verdict assignment occurs in the pipeline; its contribution is an empirical comparison using a shared collector and corpus.Source: §6 · p. 15

7. Conclusion

With identical evidence bundles, direct classification outperformed claim scoring on overall accuracy across all six models and had higher malicious alert recall in ten of twelve platform–model combinations. Retained claims enable policy inspection and revision, but do not consistently reduce variation or establish robustness to binary modifications.

7. Conclusion

  • Direct classification achieved higher overall accuracy for all six models and higher malicious alert recall in ten of twelve platform and model combinations.These comparisons used identical serialized evidence bundles and a fixed scoring policy for the claim path.Source: §7 · pp. 15–16
  • Claim mediation did not consistently reduce variation across models, and the results favor direct classification when only a sample-level verdict is required.The policy’s explicit control must be weighed against observed costs in accuracy and stability.Source: §7 · p. 16
  • Retained claims make verdict computation inspectable and revisable, allowing archived verdicts to be replayed exactly and rescored under a revised policy without another model call.Exact replay was possible for outputs from two models withdrawn by their provider.Source: §7 · p. 16
  • Removing selected predefined indicator fields caused a larger malicious alert recall loss for the claim path, but the study does not establish robustness to binary modifications.Future work will test adaptive adversaries and binary modifications before evidence collection.Source: §7 · p. 16

Related papers

Loading 2610.09553v1…