PaperlayerView exact revision on arXiv

Source-linked AI summary

Evaluating Context Segmentation in Locally Deployable SLMs for Cybersecurity CTF Tasks

Sebastiano Nordio, Michele Lotto

arXiv:2609.12839v3cs.CRcs.AI

TL;DR

Locally deployable open-weight SLM agents face context bloat and cognitive degradation during long-horizon cybersecurity tasks, amid concerns that they can operate beyond proprietary API safeguards. The paper introduces context segmentation, separating strategic planning from isolated execution attempts. It reports a reward–token-efficiency trade-off and success on complex tasks that single-pass baselines fail to solve.

  • Problem

    Tool outputs and trial-and-error logs can overwhelm SLM agents during multi-step CTF tasks, while locally deployed open-weight models raise cybersecurity concerns beyond proprietary API safeguards.

    Source: §1 · p. 2, §2 · p. 4

  • Method

    Context segmentation uses an Explorer to plan strategies and summarize attempts while independent Worker loops execute them from blank contexts.

    Source: §1 · p. 2, §3 · p. 4

  • Results

    Context segmentation achieves a superior trade-off between absolute reward and token efficiency compared with brute-force repetition, and resolves complex long-horizon tasks that single-pass baselines fail to address.

    Source: §7 · p. 8

  • Takeaways & Limitations

    The reported capability lowers the hardware barrier for automated cyberattacks using open-weight SLMs.

    Source: §1 · p. 2, §7 · p. 8

    The 6 GB RAM constraint limited evaluation to two models that reliably completed the benchmark, and broader validation beyond CTF challenges remains planned.

    Source: §6 · p. 8

Abstract

from arXiv · show

The proliferation of highly capable open-weight Small Language Models (SLMs) democratizes access to advanced cybersecurity capabilities, posing an escalating risk as these models can bypass proprietary API guardrails when deployed locally. However, SLMs deployed as autonomous agents often struggle with long-horizon, exploratory tasks like cybersecurity Capture The Flag (CTF) challenges due to context bloat and cognitive degradation from accumulated tool-call outputs. To understand and mitigate this cybersecurity threat, we introduce context segmentation, a two-level agentic framework that divides complex exploitation tasks into manageable, contextually isolated sub-problems. Evaluating on the picoCTF dataset using memory-constrained gemma-4 models, we demonstrate that for the E4B model, our strategy acts as an intelligent search, achieving competitive rewards with superior token efficiency compared to brute-force retries, and successfully solving 18.52% of tasks that standard agentic execution fails to complete. Code is available at https://github.com/9xeb/context-segmentation.

1 Introduction

Locally deployable open-weight SLMs raise cybersecurity concerns because they can operate beyond proprietary API safeguards, while context segmentation is proposed to improve their performance on complex exploitation tasks. The reported contributions include greater token efficiency than brute-force retries and success on edge cases that the reiterated baseline does not solve.

1 Introduction

  • Locally deployable open-weight models can bypass proprietary API guardrails, raising concern about automated cyber threats.The paper links this risk to models that can be fine-tuned and deployed on consumer-grade hardware.Source: §1 · p. 2
  • The proposed prompt-level strategy divides intricate exploitation tasks into manageable, context-specific sub-problems processed sequentially.Source: §1 · p. 2
  • Context segmentation is more token-efficient than brute-force iteration, although brute-force baseline loops can yield higher absolute rewards.This is presented as a trade-off between reward and token efficiency.Source: §1 · p. 2
  • Segmentation resolves complex edge-case tasks that the reiterated baseline fails to address entirely.Source: §1 · p. 2
  • The paper provides empirical evidence that SLM capabilities warrant re-evaluating cybersecurity threat models.Source: §1 · p. 2

2 Background

The background describes how SLM agents use tools to execute cybersecurity tasks, but repeated tool interactions can overwhelm their contexts. It motivates context segmentation as a way to divide those tasks into smaller, isolated sub-problems.

2 Background

  • SLMs operate under hardware constraints, and their definitions as “small” or “large” vary with context and time.Agents can issue structured tool calls that execute code, with results appended to the model context.Source: §2 · p. 3
  • The threat model assumes limited VRAM and compute, requiring locally hosted SLMs to use efficient, tool-assisted interactions rather than solving complex vulnerabilities in one pass.Source: §2 · p. 3
  • CTF challenges test cybersecurity skills across domains, with a hidden flag serving as proof of successful compromise.Their deterministic, isolated environments and clear success criteria make them benchmarks for autonomous exploitation capabilities.Source: §2 · p. 3
  • During multi-step CTF tasks, verbose tool outputs and trial-and-error logs consume token budgets, while attention degradation can lead to instruction drift, hallucinated syntax, and repeated failures.Source: §2 · p. 4

3 Methodology

The method separates strategic planning from tactical execution while using one locally hosted SLM. An Explorer agent directs independent Worker attempts and receives concise reports rather than parsing each attempt’s full history.

3 Methodology

  • The architecture separates high-level strategic reasoning from low-level execution so verbose terminal outputs do not pollute long-term planning.Source: §3 · p. 4
  • Explorer retains summarized high-level reports of attempts and proposes new instructions, while Worker executes each instructed problem-solving attempt in a blank context.Source: §3 · p. 4
  • The Explorer invokes run_strategy to start a Worker loop, which interacts with the bash CLI and returns a summary of its actions and reason for stopping.Explorer then decides whether to try another strategy or halt; both agents can signal when they have exhausted viable options.Source: §3 · p. 4
  • Context segmentation is intended to mitigate context bloat by dynamically pruning irrelevant information and helping SLM agents maintain coherent high-level problem-solving.Source: §3 · p. 4

4 Experimental Setup

The evaluation uses quantized Gemma-4 instruction-tuned models and picoCTF challenges in a modified Intercode environment. It compares a single-loop baseline with a fallback strategy that invokes Explorer after a Worker fails.

4 Experimental Setup

  • The evaluation uses 4-bit gemma-4-E2B-it and gemma-4-E4B-it models within a memory envelope below 6 GB of RAM.Source: §4 · p. 5
  • The benchmark uses picoCTF challenges in the modified Intercode CTFEnv, where agents interact with a bash shell, examine binaries, and retrieve hidden flags.The authors selected this setup amid a fragmented, model-centric evaluation ecosystem and limited availability of agent-oriented cybersecurity benchmarks.Source: §4 · p. 5

4.1 Strategies

  • The Plain baseline runs one Worker sequentially and marks a task successful only when the correct flag is submitted.A task is marked failed if the Worker gives up.Source: §4.1 · p. 5
  • The standalone Explorer strategy was excluded after observations that it underperformed on simple problems by overcomplicating straightforward solutions.The evaluation instead used Plain+Explorer.Source: §4.1 · pp. 5–6
  • Plain+Explorer first runs the Plain strategy and falls back to Explorer if the Worker gives up.Success requires submitting the correct flag; the strategy fails if Explorer concedes.Source: §4.1 · p. 6

5 Results

Plain+Explorer improves on plain execution but uses more tokens; against brute-force retries, the E4B model achieves a competitive reward with greater token efficiency and solves tasks missed by the plain baseline.

5 Results

  • Plain+Explorer achieves higher reward than plain execution at the cost of significantly higher token consumption.The strategy iterates multiple plain instances, producing the reward-versus-token trade-off.Source: Table 1 · p. 6, §5 · p. 6
  • E4B plain+explorer earns 0.39 reward with 8,652.52 tokens, beating plain best-of-4’s 0.38 with over 2,200 fewer tokens, though below best-of-5’s 0.41.The comparison is against brute-force plain iterations; plain+explorer’s reward is mean reward, while the plain best-of-k reward is reward@k.Source: §5 · p. 6, Table 2 · p. 7
  • 18.52% of tasks missed by plain execution are solved by E4B plain+explorer, while plain execution uniquely solves 2.88%.For E2B, 77.78% of tasks remain unsolved by either approach; plain+explorer recovers 7.82% uniquely versus 3.29% for plain.Source: Table 3 · p. 7, §5 · p. 7

6 Limitations and Future Work

The authors characterize the results as preliminary and identify prompt refinement, broader model and domain testing, long-term memory, and reassessment of the give-up tool as future work.

6 Limitations and Future Work

  • The results remain preliminary, with prompt optimization and refinement of Explorer and Worker prompts identified as future work.The planned refinements target strategic coherence and execution reliability.Source: §6 · p. 8
  • Future work includes adding agentic long-term memory and reassessing whether the give-up tool encourages premature stopping on exceptionally hard tasks.The authors also plan to explore new roles and support general problem-solving without domain-specific assumptions.Source: §6 · p. 8
  • A 6GB RAM constraint limited the study to two models that reliably completed the current benchmark.The authors plan to test a wider variety of models on low-end hardware and extend benchmarking beyond CTF challenges.Source: §6 · p. 8

7 Conclusion

The conclusion presents context segmentation as a way to separate strategic reasoning from execution clutter, improving outcomes on complex tasks and the reward–token trade-off. The accompanying pseudo-Python code depicts an Explorer loop that triggers strategy runs using clean Worker agents.

7 Conclusion

  • Context segmentation separates high-level strategy from low-level execution clutter and improves the reward–token trade-off against brute-force repetition.The conclusion says it resolves some complex, long-horizon tasks that single-pass baselines fail to address and frames the capability as lowering the hardware barrier for automated cyberattacks.Source: §7 · p. 8

A Pseudo Python code

  • The pseudo-code has the Explorer run the main loop and trigger strategy runs, with a clean Worker agent spawned for each strategy.The code identifies context_segmentation as the entry point and run_strategy as the mechanism for triggering sub-agent loops.Source: A Pseudo Python code · p. 11

B Hardware and Software Infrastructure

Experiments used consumer-grade AMD GPUs, llama.cpp inference with Vulkan, and Docker-isolated CTF tasks.

B Hardware and Software Infrastructure

C Models

Several candidate models were excluded from final benchmarking after 4-bit tests within a 6GB RAM envelope made them infeasible on consumer-grade hardware.

C Models

  • Qwen-3.5-4B8, Qwen-3.5-9B9, Ling-3.0-Tiny10, G9v3-3B11, and LFM2.5-8B-A1B12 were excluded from final benchmarking.The exclusions followed preliminary testing.Source: C Models · p. 12
  • At 4-bit precision, these models exhausted the context window through runaway generation, doom loops, or excessive reasoning.They were quantized to fit within the 6GB RAM envelope.Source: C Models · p. 12
  • The reported feasibility problem arose when fitting the candidate models to consumer-grade hardware using 4-bit quantization and 6GB RAM.Source: C Models · p. 12

D Intercode

The evaluation uses Intercode’s picoCTF environment for autonomous challenge solving, while the context-segmentation strategy evolved from agent-compose’s subproblem workflow into a dual-agent approach.

D Intercode

  • Intercode’s CTFEnv provides roughly one hundred Dockerized picoCTF challenges, with a Python SDK for initializing and resetting the container.Source: D Intercode · p. 12
  • Agents use Bash execution and flag submission tools, and submissions are checked against each challenge’s ground-truth flag.Incorrect submissions can prompt another attempt or termination.Source: D Intercode · p. 12

E Strategies

  • The context-segmentation idea began in agent-compose, where subproblems were solved in separate contexts and their results merged, sometimes with dependencies between subproblems.Source: E Strategies · pp. 12–13
  • After LLM-generated YAML schemas remained underwhelming, the approach retained a dual-agent split between high-level planning and low-level execution and removed the rigid YAML.Source: E Strategies · p. 13

F Additional Results

Table 4 reports mean reward and mean token usage for individual experiment runs; plain+explorer runs achieve higher reward than plain runs at higher token cost.

F Additional Results

Related papers

Loading 2609.12839v3…