PaperlayerView exact revision on arXiv

Source-linked AI summary

From Billion-Scale to Million-Scale Language Models: Operational Capabilities for Embedded Post-Exploitation

Gabriel Hortea, Simon Pietro Romano, Juan Tapiador, Mario Varlese

arXiv:2610.23534v1cs.CRcs.AI

TL;DR

The paper examines whether ultra-small language models can support autonomous post-exploitation despite the practical constraints that motivate embedded use. It evaluates compact models and interaction strategies, finding that capability depends on more than model size and that current models remain constrained for deployment.

  • Problem

    Whether current uSLMs can realize embedded post-exploitation in practice remains open, and prior evidence on their offensive viability and transfer across Python and Lua is limited.

    Source: §1 · pp. 2–3

  • Method

    The study analyzes one-shot, incremental, and closed-loop feedback interactions, using prior failures and validation diagnostics to inform self-repair.

    Source: §5 · pp. 11–12

  • Results

    Successful code generation depends on the interaction of model capacity, task formulation, prompting strategy, and programming language, rather than model size alone.

    Source: §8 · p. 25

  • Takeaways & Limitations

    Current compact models remain insufficient for autonomous generation of sophisticated malware in deployment-motivated resource-constrained post-exploitation scenarios.

    Source: §8 · p. 25

    The functional-correctness metrics do not assess code quality, robustness, stealthiness, evasion, efficiency, or maintainability, and expanded tests could change reported Pass@1 and Pass@5 values.

    Source: §7.8 · p. 25

Abstract

from arXiv · show

Small Language Models (SLMs) are increasingly attractive for resource-constrained environments due to their low computational footprint. These characteristics also make them candidates for future autonomous malware, where connectivity, latency, privacy, or operational constraints may preclude cloud-based inference. Yet the ability of ultra-compact models to support malware during post-exploitation remains poorly understood. We systematically evaluate seven open-source SLMs (20M-1.5B parameters) on two post-exploitation code-generation tasks: Directory Enumeration and Backdoor Generation. We compare Python (high-resource) and Lua (low-resource) across prompting strategies, task formulations, iterative reasoning, grammar-constrained decoding, and execution-feedback loops. Results show that offensive code-generation capabilities do not increase linearly with model size. Successful execution depends on the interaction of model capacity, task formulation, and prompting strategy, with task formulation exerting a greater influence than prompt engineering alone. Algorithmic formulation combined with few-shot prompting enables Qwen2.5-Coder 1.5B to reach Pass@5 of 100% in both Python and Lua for Directory Enumeration, 100% in Python for Backdoor Generation, and 99.2% in Lua. Although the evaluated models remain too computationally demanding for widespread deployment in contemporary malware, rapid progress in compact models may reduce this barrier. These findings help anticipate AI-assisted malware and inform future countermeasures.

1. Introduction

Cloud-dependent language-model applications face latency, privacy, and connectivity constraints, while compact models could make local inference inside malware feasible. This study tests whether ultra-small models can autonomously generate effective post-exploitation code and systematically characterizes their capabilities and limits.

1. Introduction

  • The study asks whether ultra-small language models can autonomously generate effective malicious code for post-exploitation, a capability whose practical viability remains unclear.Prior work largely assumes cloud-hosted or server-grade models, leaving embedded, resource-constrained use less explored.Source: §1 · pp. 2–3
  • The paper systematically evaluates uSLM operational capabilities and architectural limits across model scales in representative post-exploitation scenarios.Its contributions also examine interaction strategies, Python–Lua differences, and execution-driven feedback.Source: §1 · p. 3
  • The evaluation examines whether cognitive scaffolding improves or degrades code generation, and how performance changes between Python and lower-resource Lua.It also investigates execution-driven feedback and its implications for current model safety mechanisms.Source: §1 · p. 3
  • The authors release a sandbox-based evaluation framework and execution dataset to support reproducible offensive-security assessment of embedded language models.The study aims to raise awareness of uSLMs’ potential misuse in next-generation malware development.Source: §1 · p. 3

2. Background and Related Work

Prior work covers language-model adaptation, malware automation, and compact-model code generation, but largely leaves embedded post-exploitation under severe resource constraints unexplored. This paper positions its study around that gap and examines compact models across task and language settings.

2.1. LLMs and Adaptation Techniques

  • Prior LLM research explores prompting strategies such as few-shot learning and Chain-of-Thought, while adaptation work includes fine-tuning methods such as LoRA.LoRA freezes original weights and learns additional low-rank matrices to approximate updates, reducing computational cost and memory use.Source: §2.1 · p. 4

2.2. LLMs for Malware Generation and Automation

  • The reviewed work spans malware generation, polymorphism, attack-chain generation, and runtime code synthesis, but some locally run models still require server-class hardware.That computational footprint prevents realistic deployment in resource-constrained post-exploitation settings.Source: §2.2 · pp. 4–5
  • Earlier offensive-security studies generally rely on cloud-hosted APIs, server-grade local models, or external reasoning infrastructure rather than embedded uSLMs.This work shifts the focus to resource-constrained deployment and representative post-exploitation tasks.Source: §2.2 · p. 5

2.3. The Emergence of SLMs for Code Generation

  • Existing compact-model code-generation studies cover broader benchmarks and models, whereas this work narrows evaluation to post-exploitation code generation with models below 2B parameters.The study also evaluates prompting strategies in this focused setting.Source: §2.3 · p. 5
  • Unlike prior multilingual findings, this study reports statistically significant Python–Lua differences for every evaluated model below 2B parameters.The authors note that earlier benchmarks heavily represented high-resource languages and excluded niche or domain-specific languages.Source: §2.3 · p. 5

3. Operational Scenario: Embedded SLM Malware

The operational scenario embeds a pretrained compact model in malware intended to function without external communications, generating post-exploitation code for directory enumeration and backdoor deployment. Generated code is validated and can be revised through an execution-feedback cycle, under assumptions that initial compromise has occurred and Internet access is unnecessary.

3. Operational Scenario: Embedded SLM Malware

  • The proposed malware embeds an SLM to generate payloads locally, avoiding external communications and static payloads in an air-gapped setting.The analysis restricts models to off-the-shelf pretrained systems below 1.5 billion parameters to limit memory and resource-monitoring concerns.Source: §3 · pp. 5–6

3.1. Execution and Self-Correction Pipeline

  • Generated code is validated with static and syntactic checks, then unit-tested in an internal sandbox; failures trigger feedback containing the error and faulty code for self-correction.Python is used when available, with Lua as a fallback.Source: §3.1 · p. 6

3.2. Operational Threat Model

  • The threat model represents post-compromise persistence and exploration through directory enumeration and backdoor deployment, assuming successful initial execution and no Internet requirement.These assumptions bound the scenario to an already-compromised, isolated target environment.Source: §3.2 · p. 6

4. Methodology

The study evaluates off-the-shelf compact models on two post-exploitation code-generation tasks across languages, task formulations, prompting strategies, and interaction paradigms. It uses controlled experiments and execution-centric validation to assess functional correctness and computational efficiency.

4.1. Experimental Objectives; 4.3. Evaluated Models; 4.4. Target Programming Languages Rationale; 4.5. Tasks

  • Seven off-the-shelf uSLMs under 2B parameters, without fine-tuning, are tested on Directory Enumeration and Backdoor Generation in Python and Lua.Python represents a high-resource language and Lua a low-resource counterpart; the tasks target file-system enumeration and network programming, respectively.Source: §4.1 · p. 6, §4.3 · p. 7, §4.5 · p. 8

4.1. Experimental Objectives; 4.10. Validation Pipeline

  • The evaluation uses a multi-stage, sandboxed pipeline that fails code at the first validation error and dynamically executes only structurally sound programs.Static checks cover syntax, function signatures, and required elements; dynamic tests run in an isolated process with a timeout.Source: §4.1 · p. 6, §4.10 · pp. 9–10

4.2. Experimental Variables; 4.6. Task Formulation and Prompting Strategies; 4.7. Prompting Strategies

  • The factorial design compares model behavior across task formulations, prompting strategies, languages, and decoding conditions.The task formulations are basic, constrained, and algorithmic; prompting includes zero-shot, few-shot, thinking, and few-shot thinking.Source: §4.2 · p. 6, §4.6 · p. 8, §4.7 · p. 8

4.8. Structured Decoding

  • Structured decoding tests whether an EBNF grammar can reduce syntax demands on smaller models by constraining generated output.The proposed mechanism offloads syntactic burden to a deterministic parser so model capacity can focus on algorithmic logic.Source: §4.8 · p. 8

4.9. Experimental Phases and Interaction Paradigms

  • Three interaction paradigms are evaluated: one-shot generation, sequential Least-to-Most decomposition, and a bounded feedback loop for self-correction.Least-to-Most disables rigid syntax constraints; the feedback phase tests whether models can parse runtime errors and patch code.Source: Figure 3 · p. 9, §4.9 · p. 9

4.10. Validation Pipeline

  • For iterative generation, intermediate artifacts receive static checks, while sandbox execution is deferred until the final complete script.The final dynamic stage appends task-specific unit tests and executes the combined script in the language’s native interpreter.Source: §4.10 · p. 10, Figure 4 · p. 10

4.11. Evaluation Metrics; 4.11.2. Computational and Telemetry Metrics

  • Computational efficiency is assessed through generated token volume and end-to-end inference time alongside functional success.The measures are used to compare task completion with computational overhead across interaction paradigms.Source: §4.11 · p. 10, §4.11.2 · pp. 10–11

4.11.1. Functional Correctness: Pass@𝑘

  • Pass@k estimates the probability that at least one of k samples passes all validation checks, using n ≥ k samples to reduce variance.The study reports k ∈ {1, 3, 5} at task level and as a global average.Source: §4.11.1 · p. 10

5. Framework Architecture and Implementation

The framework coordinates prompt construction, SLM inference, code extraction, validation, and metric collection through modular components. It supports three interaction paradigms and adapts prompting, decoding, and validation to the target language.

Interaction Flow

  • The framework supports one-shot generation, incremental Least-to-Most construction, and closed-loop self-repair using validation diagnostics and failed code.Incremental artifacts are checked at each micro-step, while final dynamic evaluation occurs after completion.Source: §5 · p. 11
  • The Orchestrator routes tasks through a Prompt Builder that layers examples, reasoning constraints, or feedback and formats prompts for the model type.Dialogue models use native chat templates; other models receive source-language comment lines and a completion trigger.Source: §5 · p. 11
  • The VLLMRunnerProxy pre-tokenizes inputs and adjusts generation limits to prevent context overflow.It serves as a hardware-protection and execution-stability layer.Source: §5 · pp. 11–12
  • The Output Enforcer uses EBNF specifications to constrain sampling to strings that satisfy grammar-defined syntax.Language-specific parsing rules, comment tokens, and execution commands are injected through the Language Profiler.Source: §5 · pp. 11–12
  • The Code Extractor isolates executable code, which passes through ordered static, heuristic, and dynamic checks that stop at the first failure.The pipeline categorizes errors by the layer where execution is interrupted.Source: §5 · p. 12
  • The Metrics Collector aggregates inference statistics and validation success rates into reports and calculates metrics such as Pass@k.The framework architecture spans experiment configuration through metric collection and supports language-specific adaptation.Source: Figure 5 · p. 11, §5 · p. 12

6. Experimental Evaluation

Across six evaluation phases, the study tests model behavior across tasks, languages, formulations, prompting, and interaction strategies. Algorithmic formulation most benefits the largest model, while gains from iterative reasoning and feedback depend on model capacity and task.

6.1. Experimental Setup and Reproducibility

  • The experiments use a factorial design and repeat configurations ten times with deterministic seeds, reporting mean metrics and standard deviations.Sampling uses T=0.2 and n=10 candidate generations per prompt across the One-Shot, Iterative, and Reflection phases.Source: §6.1 · p. 12

6.2. Architectural and Domain Limits

  • Adding implementation constraints systematically reduces Pass@1 and Pass@5, including Python Directory Enumeration Pass@5 falling from 72.2%±11.7 to 0% for Qwen2.5-Coder 1.5B.The constrained formulation also reduces Lua Directory Enumeration Pass@5 for Qwen2.5-Coder 1.5B from 30.0%±25.8 to 5.0%±15.8.Source: §6.2 · p. 14, Figure 6 · p. 14, Figure 7 · p. 14
  • Task formulation has a substantially stronger influence on code-generation performance than prompting strategy, and algorithmic guidance improves the largest model while widening its gap from smaller models.Under the basic formulation, leading models are statistically indistinguishable on Python Directory Enumeration; algorithmic formulation establishes a significant hierarchy favoring Qwen2.5-Coder 1.5B.Source: §6.2 · pp. 14–15, §6.3 · p. 15

6.3. Interaction Strategies

  • For Qwen2.5-Coder 1.5B, Few-Shot prompting with the Algorithmic formulation raises Lua Directory Enumeration Pass@5 from 0% under Basic to 100%, and Lua Backdoor Generation from 0% to 99.2% ± 2.6.For Qwen2.5-Coder (0.5B), Python Directory Enumeration falls from 85.5% ± 15.3 under Basic to 0% under Algorithmic; smaller models do not consistently benefit from the reformulation.Source: §6.3 · pp. 15–16
  • Across tasks, grammar-constrained decoding reduced functional success, while iterative Least-to-Most prompting provided limited additional benefits and depended strongly on model capacity.Smaller models often fail within the first few decomposition steps through interaction-protocol violations; the largest model sustains longer sequences but may fail when constructing the final program.Source: §6.3 · pp. 17–18

6.4. Autonomy and Reflection

  • Execution feedback increases Qwen2.5-Coder 1.5B's Python pass rate monotonically across refinement iterations, but reflection yields marginal benefits for smaller models and virtually none on Lua.In Lua, algorithmic formulations appear to provide the largest improvements under feedback.Source: §6.4 · p. 19

6.5. Efficiency and Failure Analysis

  • More generated reasoning does not systematically improve functional success; its computational overhead translates into higher Pass@5 only for Qwen2.5-Coder 1.5B.Among unsuccessful generations, the largest model has a higher proportion of runtime failures, while smaller models fail mainly during syntax or constraint validation.Source: §6.5 · pp. 19–20
  • Smaller models fail earlier in the more demanding Backdoor Generation task, while Qwen2.5-Coder 1.5B maintains a nearly unchanged proportion of runtime failures across tasks.Backdoor failures for the largest model include runtime, syntax, and code-generation errors, often at the final program-construction stage.Source: §6.3 · pp. 18–19, §6.5 · p. 20, Figure 15 · p. 20

7. Discussion

The discussion identifies sharp capability boundaries and shows that outcomes depend on model capacity, task formulation, and interaction strategy. It also examines efficiency, failure stages, defensive implications, and threats to validity.

7.1. The Capability Threshold in Malicious Code Generation

  • Models below 300M parameters remain at the statistical floor; Backdoor Generation is harder than Directory Enumeration, and Lua generally requires the 1.5B model.Python has a lower effective capability threshold than Lua.Source: §7.1 · p. 20

7.2. The Paradox of Prompting and Constraints

  • Constraints reduce successful generations, while algorithmic guidance helps Qwen2.5-Coder 1.5B but degrades smaller models; task decomposition matters more than prompt strategy alone.Grammar-constrained decoding also reduces functional success, particularly for the most capable evaluated models.Source: §7.2 · p. 20

7.3. The Limits of Iterative Reasoning and Cognitive Load

  • Algorithmic reformulation, not iterative prompting, is the primary source of performance improvement; only the largest evaluated model reliably manages the added planning demands.Smaller models may lose conversational structure, while Qwen2.5-Coder 1.5B can preserve the reasoning chain and fail at final synthesis.Source: §7.3 · p. 21

7.4. The Boundaries of Autonomous Self-Correction

  • Reflection helps mainly when a model can already generate partially correct solutions; Qwen2.5-Coder 1.5B can use feedback for repair, especially with algorithmic formulations.SmolLM2 360M and Qwen 0.5B gain virtually no benefit from extra iterations when baseline ability is absent, particularly in low-resource settings.Source: §7.4 · pp. 21–22

7.5. Efficiency Trade-offs and the Illusion of Verbosity

  • For Qwen2.5-Coder 1.5B, longer reasoning can yield valid solutions, but below the capability threshold it mainly adds verbose output and computational overhead.The Figure 18 evaluation maps Pass@5 against token efficiency across models, tasks, formulations, prompting strategies, and languages.Source: §7.5 · p. 22, Figure 18 · p. 23

7.6. The Qualitative Shift in Failure Modes

  • Smaller models’ failures are dominated by early constraint errors, especially in Lua, whereas larger models more often reach runtime before failing.Figure 19 categorizes unsuccessful generations by their earliest failure stage, independently of overall success rate.Source: §7.6 · pp. 22–23, Figure 19 · p. 23

7.7. Implications for Defenders

  • Current uSLMs are not reliable replacements for handcrafted malware, but their capabilities reflect interactions among model size, task formulation, and prompting strategy.The discussion proposes behavioral signals such as inference footprints and repeated execution-feedback cycles for defensive research, while noting that capability thresholds may decline as compact models advance.Source: §7.7 · pp. 23–24

7.8. Threats to Validity

  • Manually designed prompts, decompositions, and fixed few-shot examples may bias results, so observed differences do not establish universally optimal designs.Fixed demonstrations improve reproducibility but may affect smaller models more because their behavior is more sensitive to in-context examples.Source: §7.8 · p. 25

8. Conclusion and Future Work

The paper characterizes ultra-small language models’ post-exploitation code-generation capabilities and finds that success depends on more than model size. It highlights task formulation, prompting, and programming language as important factors, while noting limits in the evaluation’s scope.

8. Conclusion and Future Work

  • Successful malicious code generation reflects interactions among model capacity, task formulation, prompting strategy, and programming language rather than model size alone.Source: §8 · p. 25
  • Algorithmic task reformulation consistently yields larger performance gains than increasing reasoning depth through iterative prompting.Source: §8 · p. 25
  • Reliable generation across both evaluated tasks is observed only for Qwen2.5-Coder-1.5B, while 360M and 0.5B models show limited but non-negligible capabilities.Source: §8 · p. 25
  • Moving from Python to Lua substantially increases task difficulty in the evaluated tasks.Source: §8 · p. 25
  • The study’s framework combines functional validation, iterative reasoning analysis, computational-efficiency measurements, and a failure taxonomy.Source: §8 · p. 25
  • The evaluation covers two post-exploitation tasks, open-weight models below two billion parameters, and controlled prompting without fine-tuning or reinforcement-based adaptation.Future work includes broader post-exploitation primitives and malware-oriented languages, parameter-efficient adaptation, deployment-oriented evaluation, and compressed deployment formats.Source: §8 · pp. 25–26

Software and Data Availability

The authors report that the study’s source code and artifacts have been archived on Zenodo, with code also maintained on GitHub.

Software and Data Availability

  • The source code and artifacts are archived on Zenodo, and code is also maintained on GitHub.The private repository is to become fully public upon publication.Source: Software and Data Availability · p. 26

A. Detailed Task Definitions

The appendix defines directory enumeration tasks through progressively more structured prompts, from unconstrained natural language to artificial constraints and algorithmic scaffolding. It also outlines least-to-most prompting as a sequence of implementation micro-steps.

A. Detailed Task Definitions

2. Constrained

  • The constrained formulation introduces artificial friction through banned APIs and required command execution.Its example requires io.popen or os.popen and disallows standard directory-reading libraries such as lfs or os.listdir.Source: §2 · p. 29

3. Algorithmic

  • The algorithmic formulation supplies strict pseudocode for creating a list, running ls -1, processing output, closing the handle, and returning results.Source: §3 · p. 29

Iterative Evaluation (Least-to-Most Prompting)

B. Runtime Validation Pseudocode

The pseudocode algorithms correspond to the paper’s directory enumeration and backdoor tasks.

B. Runtime Validation Pseudocode

C. EBNF Grammar Definitions

The section describes using EBNF grammars to constrain SLM logits during Lua and Python script generation.

C. EBNF Grammar Definitions

D. Other Matherials

This materials section presents tabular results with standard deviations and outlines the backdoor task and its prompting formulations.

D. Other Matherials

  • The section reports results in tables 9–12, including standard deviations.Source: D. Other Matherials · p. 29
  • Table 8 is identified as the task definition card for a TCP command execution server.Source: Table 8 · p. 30
  • The basic unconstrained prompt asks for a TCP command execution server on port 7456 that receives commands, executes them, and returns results.Source: D. Other Matherials · p. 30

2. Constrained

  • The constrained formulation adds strict requirements to use io.popen() in Lua or subprocess.Popen in Python, capture output, and avoid os.execute() with temporary files.Source: §2 · p. 30

3. Algorithmic

  • The algorithmic formulation supplies an ordered seven-step pseudocode sequence from socket creation through connection closure.Source: §3 · p. 30

Iterative Evaluation (Least-to-Most Prompting)

The section describes incremental implementation and unit-test validation, alongside results on task formulation and Lua inference overhead.

  • The incremental implementation instructions cover socket setup, accepting a client, receiving its command, executing it, returning the result, and closing the connection.Source: Iterative Evaluation (Least-to-Most… · p. 30
  • The Directory Enumeration validation algorithm sets up a controlled filesystem and checks that the generated result includes a file and a subfolder, then tears down the environment.Source: Iterative Evaluation (Least-to-Most… · p. 31
  • The Backdoor validation algorithm runs the generated server in a daemon thread, simulates a client, and checks for the expected command output while recording crashes.Source: Iterative Evaluation (Least-to-Most… · p. 31
  • The appendix includes simplified Lua and Python EBNF grammar figures used in Outlines.Source: Figure 20 · p. 32, Figure 21 · p. 33
  • The Lua Directory Enumeration overhead table highlights Qwen 0.5B contextual collapse under Few Shot + Thinking and SmolLM2 360M over-generation without success under Thinking.Source: Table 9 · p. 34
  • The Lua Backdoor Generation overhead table highlights Qwen 0.5B contextual collapse under Few Shot + Thinking and high SmolLM2 360M thinking costs without corresponding performance gains.Source: Table 10 · p. 35
  • Retention ≥91.3%: Process-Driven Scaffolding substantially reduces the Python–Lua performance gap on Directory Enumeration.Models under 300M parameters are aggregated because they consistently exhibit zero capability.Source: Table 11 · p. 36
  • 90.0%: Qwen 1.5B’s backdoor success rate in Lua under the Process-Driven (Algorithmic) formulation.Source: Table 12 · p. 37

Related papers

Loading 2610.23534v1…