Source-linked AI summary
BUSTA: Teaching Small Language Models to Reason About Binary Execution
Ritika Poddar, Aidan Fennelly, Deirdre Murphy, Eric Wolos, Corre L. Steele, Michael Doyle
TL;DR
Reverse engineering requires low-level binary reasoning, while constrained and isolated environments can limit access to frontier-scale APIs. This paper fine-tunes open models with GRPO on inverse basic-block execution; knowable-register accuracy rises substantially, though frontier models still lead overall.
Problem
Reverse engineering requires reasoning over low-level program behavior, and resource-constrained isolated workflows limit the use of frontier-scale solutions.
Source: §1 · pp. 1–2Method
The authors use GRPO to fine-tune open models to predict input register values from x86-64 basic blocks and desired output register states.
Source: §1 · pp. 1–2, §2.2 · p. 2Results
Knowable-register accuracy reaches 85.3% for BUSTA-BB-Qwen3-1.7B and 95.2% for BUSTA-BB-Nemotron-3-Nano, while frontier models remain strongest overall at up to 99.2%.
Source: §3.1 · p. 4Takeaways & Limitations
Targeted post-training can improve small open-weight models on low-level binary reasoning and may be practical in compute-constrained reverse-engineering environments.
Source: §4 · p. 4The dataset covers 100 Assemblage binaries; evaluation on more diverse unseen binaries and in actual deployment environments remains future work.
Source: §5 · p. 4
Abstract
from arXiv · showhide
Large Language Models (LLM) are increasingly applied to Reverse Engineering (RE) workflows. RE, however, differs from common LLM coding tasks as it requires models to reason about low-level program behavior. Furthermore, the deployment of tooling in resource-constrained sandboxed RE environments limits the ability to use frontier-scale solutions. We examine whether a targeted LLM training approach can transfer distinct RE skills to open models via inverse basic-block execution: given an x86-64 basic block and a desired output register state, the model predicts the input register values that produce it. Fine-tuning \texttt{Qwen3-1.7B} and \texttt{Nemotron-3-Nano} with Group Relative Policy Optimization (GRPO) raises knowable-register prediction accuracy from 73.2\% to 85.3\% and from 63.3\% to 95.2\%, respectively. Frontier models still score highest (up to 99.2\%), but our fine-tuned \texttt{Nemotron-3-Nano} comes within four points at a fraction of the parameter count. We release both fine-tuned models and the training dataset.
1 Introduction
The paper asks whether targeted training can teach open models reverse-engineering skills that general prompting may not provide. It studies inverse execution of binary basic blocks, motivated by low-level reasoning demands and constrained deployment settings.
1 Introduction
- RE requires reading assembly, registers, and traces and reasoning over low-level program state, unlike many common software tasks.Prior LLM-based RE work has emphasized zero- and few-shot prompting more than fine-tuning.Source: §1 · p. 1
- The study tests whether models can infer input register values from a binary basic block and desired output state, a task arising in vulnerability research.The authors frame this as a simplified first step toward determining inputs or initial machine states that produce a desired execution state.Source: §1 · p. 1
- Sensitive or malicious binaries are often analyzed in isolated environments where frontier APIs are unavailable, motivating locally runnable open-weight models.Smaller open-weight models can fit tighter compute budgets.Source: §1 · p. 2
- The authors fine-tune Qwen3-1.7B and Nemotron-3-Nano with GRPO and benchmark them alongside open and frontier models.Their comparison aims for equitable and reliable evaluation across differing model-generation styles.Source: §1 · p. 2
2 Method
The method builds labeled x86-64 basic-block examples, trains models with GRPO and a weighted task reward, then evaluates prediction quality and token use against base and frontier models.
2.1 Data Curation
- From 100 Windows PE binaries, the authors generate 287,137 x86-64 basic blocks and retain 282,741 validated samples split by source function into training, validation, and test sets.Training data are further downsampled using stratified sampling.Source: §2.1 · p. 2
- Each sample records a block’s disassembly and input/output register states after emulation with randomized initial registers, while RIP, RSP, and RBP are fixed.Registers are labeled unknowable when multiple inputs can yield the same output; RIP, EIP, IP, and RFLAGS are always treated as unknowable.Source: §2.1 · p. 2
2.2 Training Method
- Models receive on-policy GRPO post-training against sparse task-level rewards, with prompts specifying the expected input and output schema.The authors use the NeMo RL post-training library with the Megatron backend.Source: §2.2 · p. 2
2.3 Reward Function
- The final reward weights register-value prediction more heavily than knowability classification and penalizes omitted registers and invalid responses.Correct knowable values receive +1 and incorrect values −1; correct values for unknowable registers receive +1.1 without an incorrect-value penalty.Source: §2.3 · p. 3
2.4 Evaluation
- Evaluation uses 635 downsampled held-out samples and measures response validity, register coverage, knowable-register accuracy, unknowable classification, and token use.The comparison includes each fine-tuned model’s base model and frontier models, with token use serving as a deployment-agnostic inference-cost proxy.Source: §2.4 · p. 3
3 Experiments
The experiments fine-tune Qwen3-1.7B and Nemotron-3-Nano on inverse register inference, then compare them with base and frontier models. Both fine-tuned models improve knowable-register accuracy, while frontier models remain strongest overall.
3 Experiments
- The Qwen3-1.7B experiment uses GRPO with a 32K-token context window and batch size 512, choosing a small model for constrained deployment and rapid iteration.Its best configuration uses 22,223 training samples, 2,745 validation samples, and completes a 347-step epoch.Source: §3 · p. 3
- Nemotron-3-Nano is fine-tuned with GRPO and rank-128 LoRA adapters; the four-node setup uses stabilization measures to complete 347 steps.These include importance-sampling correction, sequence masking, a KL penalty, and learning-rate adjustments.Source: §3 · p. 3
3.1 Results
- 85.3% for BUSTA-BB-Qwen3-1.7B and 95.2% for BUSTA-BB-Nemotron-3-Nano knowable-register accuracy, gains of 12.1 and 31.9 points over their respective base models.Frontier models remain strongest overall, led by GPT-5.5 at 99.2%; the fine-tuned models close much of the primary-task gap at smaller scale.Source: §3.1 · p. 4, Table 1 · p. 4
4 Conclusion
The conclusion presents targeted GRPO post-training on inverse basic-block execution as evidence that small open-weight models can improve at low-level binary reasoning and close much of the frontier-model gap.
4 Conclusion
- Targeted GRPO post-training substantially improves knowable-register accuracy and closes much of the gap to out-of-the-box frontier models at smaller scale.The authors suggest it may be practical for compute-constrained reverse-engineering environments.Source: §4 · p. 4
5 Limitations and Future Work
The experiments are limited by the dataset’s size and scope, and reward design and task formulation remain areas for further study. The authors propose extending evaluation to longer program contexts and the target deployment environment.
5 Limitations and Future Work
- The dataset covers only 100 Assemblage binaries, and future work should test larger, more diverse sets of unseen binaries.The authors also identify reward-design improvements for edge cases and clearer separation of knowable-register accuracy from broader register classification.Source: §5 · p. 4
- The role of forward prediction remains open: it was removed in favor of inverse-only training, but its potential contribution warrants further analysis.The authors also propose extending beyond individual basic blocks to sequences and whole-program target-state reachability, and evaluating in the actual deployment environment.Source: §5 · p. 4
A Data Curation Details
The curation pipeline filters invalid emulations and traces, reduces each sample to one run, and structures register inputs and outputs explicitly. The resulting valid samples are split into train, validation, and test sets.
A Data Curation Details
- Samples are removed when emulation fails or parsing yields inconsistent traces, and are invalidated when trace addresses do not match the assembly or block.Memory-trace addresses must be a subset of instruction addresses; across three executions, memory traces must either always appear or never appear.Source: A Data Curation Details · p. 7
- Each sample retains only the first of three runs, and per-instruction memory and register traces are omitted before training and evaluation.Those traces expose register-state changes and are generally unavailable in realistic reverse-engineering workflows.Source: A Data Curation Details · p. 7
- 282,741 valid samples are split 80/10/10 into training, validation, and test sets after input and output registers are explicitly labeled.The splits contain 225,993 training, 28,374 validation, and 28,374 test samples.Source: A Data Curation Details · p. 7
B Downsampling
Downsampling stratifies basic blocks by memory-trace presence and instruction count, then preserves rare mnemonics while filling per-stratum caps. Separate caps are used for evaluation, training, and validation.
B Downsampling
- Samples are divided into eight strata by memory-trace presence and instruction-count bins, with greedy retention of samples introducing unseen mnemonics before random filling.The resulting sample is enriched for memory traces, high instruction counts, and rare mnemonics; per-instruction memory traces are removed to avoid leaking register-state information.Source: B Downsampling · p. 7
- The held-out test split is capped at 100 samples per stratum, yielding 635 samples; training caps are 2,000 per stratum in the base configuration and 4,000 in the best configuration.Validation is capped at 500 samples per stratum, independently of the evaluation-set capping stage.Source: B Downsampling · p. 7
C Evaluation Protocol
The evaluation protocol varies seeds and tunes sampling parameters where possible, while separating reasoning from structured output. Responses are normalized and filtered before scoring.
C Evaluation Protocol
- Models are evaluated across three to ten seeds, except Claude and GPT models whose APIs limit or disallow seed selection; temperature and top-p are tuned before evaluation.The tuning aims to capture each model’s best performance.Source: C Evaluation Protocol · p. 7
- A vLLM reasoning parser separates reasoning from structured generation so smaller models such as Qwen3-1.7B can reason without structured-output limits constraining them from the start.Source: C Evaluation Protocol · p. 7
- Responses are normalized and validated after generation, and duplicated registers from base Qwen3-1.7B are removed in their entirety.Register names are normalized to lowercase.Source: C Evaluation Protocol · p. 8
- Table 2 presents evaluation metrics and definitions, including valid response rate, register coverage, knowable-register accuracy, and unknowable classification accuracy.Unknowable value accuracy is included as a supplemental metric.Source: Table 2 · p. 8
D Training Details
The training experiments vary invalid-response penalties, sample sizes, and metric weights, using data subsets to tune parameters efficiently. Nemotron training separates rollout and optimization workers, then adjusts training settings to stabilize late-stage mismatch and complete the run.
D Training Details
- Experiments varied malformed-response penalties, per-stratum sample caps, and metric weights, using 20k–100k-sample subsets for efficient tuning.The penalty values were 0, −0.5, and −1.0; sample caps ranged from a 2k-per-stratum baseline to the full dataset.Source: D Training Details · p. 8
- Nemotron training used separate vLLM rollout and Megatron policy-optimization workers, with importance-sampling correction for their policy mismatch.The four-node non-collocated architecture addressed node-level memory constraints; masking sequences with log-probability error above 1.5 supported stable early learning.Source: D Training Details · p. 8
- Late-training mismatch prompted a reference-policy KL increase from 0.01 to 0.03 and cosine learning-rate decay toward 5 × 10−7.These changes reduced batch starvation from sequence masking and allowed learning to continue through the full 347-step run.Source: D Training Details · p. 8
E Reward Function Details
The reward combines mean register-value and knowability-classification scores after transforming each component. It weights register-value recovery more heavily and assigns malformed responses a fixed penalty.
E Reward Function Details
- The reward uses mean per-register value and classification scores, with value correctness, ground-truth unknowability, and the model’s knowability label defined per register.Classification scores are computed over predicted registers in G ∩ P.Source: E Reward Function Details · p. 8
- Each component’s 2x −1 transform bounds the penalty floor at −0.5 and rewards higher accuracy superlinearly.The two transformed components are then combined in the final reward.Source: E Reward Function Details · p. 8
- The final reward weights the register score at 0.7 and the unknowable-classification score at 0.3; malformed responses receive a default −0.5 penalty instead.Responses that fail JSON or schema validation receive the fixed penalty in place of the final reward.Source: E Reward Function Details · p. 8