PaperlayerView exact revision on arXiv

Source-linked AI summary

Symbol Preference Aware Generative Models for Recovering Variable Names from Stripped Binary

Xiangzhe Xu, Zhuo Zhang, Zian Su, Ziyang Huang, Shiwei Feng, Yapeng Ye, Nan Jiang, Danning Xie, Siyuan Cheng, Lin Tan, Xiangyu Zhang

arXiv:2306.02546v4cs.SE

TL;DR

Recovering variable names from stripped binaries is challenging, and function-only fine-tuning limits contextual knowledge. GENNM teaches generative models to use calling context and mitigates model biases with symbol preference optimization. It improves state-of-the-art precision, including when names are unseen during training.

  • Problem

    Function-only fine-tuning limits models’ contextual knowledge, despite calling context playing a crucial role in understanding binary programs.

    Source: §I · p. 2

  • Method

    GENNM uses context-aware fine-tuning to teach models about calling contexts and symbol preference optimization to mitigate model biases.

    Source: §I · p. 2, §IX · p. 15

  • Results

    GENNM improves state-of-the-art precision by 5.6–11.4 percentage points on two datasets and by 32% (17.3% to 22.8%) when ground-truth names are unseen during training.

    Source: §I · p. 2

  • Takeaways & Limitations

    GENNM is presented as a systematic solution for recovering names from fully stripped binaries.

    Source: §I · p. 2

    VarBERT cannot predict names outside its training vocabulary, and one-function-at-a-time analysis misses calling-context information.

    Source: B. Challenges and Limitations of… · pp. 3–4

Abstract

from arXiv · show

Decompilation aims to recover the source code form of a binary executable. It has many security applications, such as malware analysis, vulnerability detection, and code hardening. A prominent challenge in decompilation is to recover variable names. We propose a novel technique that leverages the strengths of generative models while mitigating model biases. We build a prototype, GenNm, from pre-trained generative models CodeGemma-2B, CodeLlama-7B, and CodeLlama-34B. We finetune GenNm on decompiled functions and teach models to leverage contextual information. GenNm includes names from callers and callees while querying a function, providing rich contextual information within the model's input token limitation. We mitigate model biases by aligning the output distribution of models with symbol preferences of developers. Our results show that GenNm improves the state-of-the-art name recovery precision by 5.6-11.4 percentage points on two commonly used datasets and improves the state-of-the-art by 32% (from 17.3% to 22.8%) in the most challenging setup where ground-truth variable names are not seen in the training dataset.

I. INTRODUCTION

Recovering variable names from stripped binaries is difficult because conventional methods miss descriptive semantics and classification models are limited by vocabulary, bias, and context. GENNM combines generative modeling with context-aware fine-tuning, symbol preference optimization, and iterative context propagation, and reports improved recovery over prior systems, including for unseen names.

I. INTRODUCTION

  • Variable-name recovery requires machine and descriptive semantics, while classification methods are limited to training vocabularies, biased name distributions, and individual-function inputs.For names absent from training, the state-of-the-art precision is reported below 10%; performance for rarely seen names drops from 31.8% to 13.5%.Source: §I · pp. 1–2
  • GENNM’s context-aware fine-tuning uses calling-context information alongside a function body to teach relations between local-variable names and names in those contexts.Typical fine-tuning predicts names from individual binary functions, limiting contextual knowledge.Source: §I · p. 2
  • SymPO guides GENNM toward context-preferred variable names rather than names favored only by dataset frequency.For network programming, packet may be preferred to the more frequent name array for data sent over a network.Source: §I · p. 2
  • GENNM iteratively propagates context through the program call graph from functions to callers and callees, then uses program analysis and majority voting to select final names.The iterative process begins with local-context predictions and is inspired by breadth-first human reverse engineering.Source: §I · p. 2
  • GENNM improves name recovery accuracy over state-of-the-art techniques by 5.6–11.4 percentage points on two datasets and by 32% (17.3% versus 22.8%) for unseen ground-truth names.The latter comparison is the most challenging setup described.Source: §I · p. 2

A. Motivating Example

  • In the motivating example, decompiled names n and v29 fail to preserve source-variable meaning; n corresponds to ip hdrlen, the IP packet-header length.The example illustrates the goal of replacing placeholders or names synthesized from library functions with meaningful names.Source: A. Motivating Example · pp. 2–3

B. Challenges and Limitations of State-of-the-Art

State-of-the-art classifiers struggle with unseen names, imbalanced name frequencies, and missing call-context information. GENNM addresses these challenges with generative fine-tuning, symbol preference optimization, and iterative context-aware inference.

B. Challenges and Limitations of State-of-the-Art

  • 8.5% versus 22.8% precision on unseen names: VarBERT is limited to its training vocabulary, whereas GENNM can compose names not seen during training.The cited comparison reports VarBERT at 8.5% and GENNM at 22.8% for variables whose ground-truth names were unseen in training.Source: B. Challenges and Limitations of… · p. 3
  • More than 50% of variable names appear only once in the training dataset, making its distribution significantly biased; typical classification loss emphasizes frequent names by optimizing ground-truth name prediction.Source: B. Challenges and Limitations of… · p. 3, Figure 3 · p. 4
  • Analyzing functions independently omits call context, leaving models without information from callees that could clarify variable semantics.In the motivating example, VarBERT predicts ip 0 for v29, whose ground-truth name is ip hdr.Source: B. Challenges and Limitations of… · p. 4

C. Our Method

  • GENNM fine-tunes generative models to compose unseen names and uses SymPO to align name probabilities with developers’ context-specific preferences.SymPO uses model mistakes and better-ranked alternatives to construct preference pairs; the examples show packet and ip hdr gaining probability relative to frequent alternatives.Source: C. Our Method · p. 4
  • Context-aware fine-tuning teaches GENNM relations between local-variable names and caller/callee names; iterative call-graph propagation and data-flow validation refine candidates across rounds.Function signatures summarize calling contexts, while data-flow-related names help select semantically consistent candidates.Source: C. Our Method · p. 5

III. PROBLEM DEFINITION

  • The problem is formalized around binary programs, decompiled functions, call-graph edges, and a name map from function and variable identifiers to meaningful names.The dataset consists of binary-program and ground-truth-name-map pairs; the discussion simplifies decompiled code to expressions involving identifiers and function calls.Source: §III · p. 5

A. Overview

  • GENNM’s training pipeline fine-tunes a pretrained code model, mines preference pairs, and applies SymPO; inference iteratively predicts, validates, and propagates names until convergence or a budget limit.Inference collects candidates across rounds, validates them using data flow, then propagates selected names along the call graph for the next round.Source: A. Overview · pp. 5–6

B. Fine-tuning Generative Model

GENNM fine-tunes on both local decompiled functions and functions augmented with call-graph context. Its context propagation renames caller callsites and callee signatures, while causal language modeling trains on query and response tokens.

Fine-tuning Generative Model

  • GENNM fine-tunes on both individual functions and context-augmented functions, using causal language modeling loss on tokens from the query and response.The local dataset supplies function bodies and ground-truth names; the contextual dataset additionally supplies information gathered through the call graph.Source: B. Fine-tuning Generative Model · pp. 6–7
  • Caller names can suggest a function’s higher-level purpose, while callee names can indicate its primitive functionality; appending their full code bodies is described as costly and potentially less effective with longer inputs.The paper instead uses contextual information gathered along the call graph.Source: B. Fine-tuning Generative Model · pp. 6–7
  • Context propagation renames caller callsites with predicted caller-variable names and appends renamed callee signatures to the analyzed function’s query.The name map contains ground-truth names during training and predicted names during inference; the propagation rules specify caller and callee contexts separately.Source: B. Fine-tuning Generative Model · p. 7

C. Symbol Preference Optimization

SymPO addresses training-data bias by teaching GENNM to prefer better variable names over less preferred alternatives. It uses model-generated pairs and a reference model to guide preference updates without changing the model’s distribution too much.

Constructing the SymPO dataset

  • The preference signal is based on string similarity to ground-truth names, not human evaluation.Each sample pairs a query function with a preferred name and a less preferred name sampled from the model’s output.Source: C. Symbol Preference Optimization · p. 7
  • The pairwise dataset uses cases where GENNMCtx errs but one of its top 20 predictions is significantly more similar to the ground truth.The better prediction is preferred; another sampled prediction serves as the worse name, with token-level precision and recall measuring similarity.Source: C. Symbol Preference Optimization · pp. 7–8
  • Rather than use ground-truth names as preferences, SymPO selects the model’s best prediction because ground-truth names may lie too far from its learned distribution.Using ground-truth names underperforms when GENNMCtx has not learned to generate them.Source: C. Symbol Preference Optimization · p. 7
  • Static-code heuristics reduce the SymPO dataset by 60%, while training on the reduced data performs slightly better than training on all samples.The filters exclude functions with mostly unnamed callees and short functions without branches; test functions are not removed.Source: C. Symbol Preference Optimization · p. 8

Loss function

  • SymPO trains GENNM to assign higher probabilities to preferred names than to less preferred names while limiting divergence from its original distribution.During SymPO, GENNMCtx remains frozen as the reference model.Source: C. Symbol Preference Optimization · p. 8

D. Context Augmentation at the Inference Stage

At inference, GENNM iteratively propagates predicted names across a program’s call graph and selects consistent candidates. The accompanying setup evaluates this approach on deduplicated datasets using binary-level splits, several model sizes, and token-based and GPT4 metrics.

D. Context Augmentation at the Inference Stage

  • GENNM updates function contexts iteratively by propagating generated names along the call graph, then uses data-flow analysis and majority voting to select the most consistent candidate.The authors distinguish this customized analysis from conceptual novelty, noting prior work combining program analysis and LLMs.Source: D. Context Augmentation at the… · pp. 8–9

A. Dataset

  • The DIRTY dataset is rebuilt from popular GitHub projects and VarCorpus comes from Gentoo; both use strict deduplication and string-similarity filtering to limit overlap.The processed datasets contain 348k DIRTY functions and 895k VarCorpus functions; tests with similarity above 90 to training samples are filtered out.Source: A. Dataset · p. 9

B. Splits

  • Most experiments split training and test data by binary rather than by function to reduce leakage through shared global-variable placeholders.A separate split-by-function setup follows prior work.Source: B. Splits · p. 9

C. Models

  • Most experiments use GENNM fine-tuned from CodeGemma-2B, with CodeLlama-7B and CodeLlama-34B also evaluated on DIRTY.The 34B model uses LoRA because resources do not support full fine-tuning; VarBERT and ReSym are baselines.Source: C. Models · pp. 9–10

D. Metrics

  • Evaluation combines token-based semantic precision and recall with GPT4 ratings of context relevance and semantics accuracy.GPT4 evaluates each predicted name using a decompiled function, ground-truth names, and a mapping from ground-truth to predicted names.Source: D. Metrics · p. 10

VI. EVALUATION

GENNM outperforms VarBERT and ReSym across the reported datasets and splits, with gains also reported for larger models, optimized programs, rare names, and GPT4 ratings. Its advantage is especially notable for names unseen in training and for project-not-in-train samples.

A. Performance in terms of Semantics Match

  • 3.9–8.6 percentage points: GENNM outperforms both baselines on project-not-in-train samples, and gains more than 10 percentage points over VarBERT on project-in-train samples.Both groups exclude training-data samples, but project-in-train programs come from projects represented in training.Source: A. Performance in terms of Semantics… · p. 10
  • GENNM outperforms VarBERT and ReSym on all datasets/splits; versus ReSym, precision gains are 5.6 points on DIRTY and 9.5 on VarCorpus, while recall gains are 4.6 and 6.2 points.Source: A. Performance in terms of Semantics… · p. 10

B. Performance w.r.t. Different Sizes of Base Models

C. Generalization to Different Compiler Optimizations

  • Across compiler optimization levels -O0 to -O3, GENNM outperforms both baselines, with larger improvements at -O0 and -O1 than at -O2 and -O3.The authors attribute the smaller gains on aggressive optimization levels to longer programs that diverge further from source-code distribution.Source: C. Generalization to Different… · p. 11, Figure 10 · p. 12

D. Generalization to Rare Names

  • 27.1%: GENNM’s performance on variables with name frequency 1–10, compared with 13.5% for VarBERT and 20.4% for ReSym.For names never seen in training, GENNM exceeds 20% precision; 95% of rare names are composed of frequently appearing tokens.Source: D. Generalization to Rare Names · pp. 11–12, Figure 11 · p. 12

E. Performance Evaluated by GPT4Evaluator

F. Performance Compared to Blackbox LLMs

GENNM outperforms the best black-box LLM results on DIRTY, while ablations identify contextual methods as major contributors. Case studies show recovered names can clarify malware behavior and improve binary summaries.

F. Performance Compared to Blackbox LLMs

  • 11.3 percentage points in precision and 6.4 percentage points in recall separate GENNM from the best black-box LLM results on DIRTY.The comparison queries GPT-3.5, GPT-4, and CodeLlama-70B in zero-shot and 3-shot setups, except CodeLlama-70B is evaluated only 3-shot because of zero-shot format errors.Source: F. Performance Compared to Blackbox LLMs · pp. 12–13

G. Ablation Study

  • Context-aware fine-tuning and context propagation contribute most to GENNM’s 5.6% precision improvement over ReSym, and all tested components contribute.A 5-degree context sensitivity empirically works well.Source: Figure 13 · p. 13, G. Ablation Study · p. 13
  • Static feature heuristics reduce the symbol-preference dataset by around 60%, with slightly better overall performance than using the whole dataset; ground-truth preferred names perform significantly worse.The default setup uses the model’s best-predicted names as preferences and static feature heuristics to reduce dataset size and noise.Source: G. Ablation Study · p. 13

VII. CASE STUDIES

  • In a malware case study, GENNM’s names clarify command parsing and dispatch, while a separate study reports that ChatGPT summarizes decompiled functions more accurately with recovered names.The renamed variables expose the sample’s command-and-control handling and suspicious command execution.Source: §VII · p. 13

VIII. RELATED WORK

  • Unlike classification-based DIRTY and VarBERT, which select from a training vocabulary, GENNM’s generative formulation can predict rarely seen names.VarBERT pre-trains on source code before fine-tuning on decompiled code; DIRTY trains on decompiled code [15].Source: §VIII · p. 14
  • GENNM extends ReSym’s name-recovery goal with context-aware fine-tuning and SymPO, while its analysis propagates names rather than inferring types.ReSym fine-tunes on individual decompiled functions and uses data-flow analysis for type and data-structure information.Source: §VIII · p. 14

IX. CONCLUSION

  • GENNM combines generative models with contextual fine-tuning and symbol preference optimization to recover meaningful names from fully stripped binaries.The conclusion characterizes its prototype as significantly improving on the state of the art in challenging setups.Source: §IX · p. 15

APPENDIX

The appendix explains how SymPO’s reference model limits further optimization of learned preferences, describes preprocessing and leakage checks, and tests a name-prediction correlation involving memset.

A. How a Reference Model Prevents the SymPO Model from Diverging too much

  • The reference model reduces updates on examples where the optimized model already prefers better names, limiting further optimization and divergence from the reference.The preference-improving term raises better-name probability and lowers worse-name probability; the reference-dependent term constrains update magnitude.Source: A. How a Reference Model Prevents the… · p. 17

B. Dataset Preprocessing and Statistics

  • The dataset pipeline filters projects below 20 stars, excludes relocatable binaries, and conservatively deduplicates binaries unless over 70% of their function names are new.The authors say deduplication avoids adding wrapper programs after their shared-object logic is already included.Source: B. Dataset Preprocessing and Statistics · p. 17
  • Exact-string matching can miss near-duplicate functions and overestimate model performance; the authors use edit-distance similarity and flag test samples above 90 similarity to training data.For allocate examples, two versions differing only in allocation size are near-identical, while another version also zeroes allocated memory; Table VI reports substantially better model performance on flagged overlaps, which the authors say is not representative.Source: B. Dataset Preprocessing and Statistics · p. 17, Figure 15 · p. 18, Table VI · p. 18

C. Correlation between memset and buffer

1 update ←True

Name validation propagates correlated name candidates across program data flow, then iteratively selects names for semantic consistency.

Name validation

  • Name validation starts with each variable’s top-1 predicted name, gathers candidates from directly data-flow-related variables, and iterates until no names change or a preset budget is reached.Multiple rounds allow propagation across several data-flow edges.Source: D. Program Analysis and Semantics… · p. 19

Correlation extraction

  • The analysis maps each variable to correlated names and tracks each name’s origin to prevent repeated data-flow paths from overweighting it in voting.The origin record identifies the source function, variable, and name.Source: D. Program Analysis and Semantics… · p. 19
  • For an assignment id1 to id0, correlated names of id1 flow to id0, while only id0’s selected name flows back to id1 because id0 is assumed broader.The rule avoids propagating all of id0’s correlated names to the narrower id1.Source: D. Program Analysis and Semantics… · p. 19

Program Analysis and Semantics Voting

  • Semantics voting selects each variable’s candidate by similarity to correlated names and other candidates, using cosine similarity between name embeddings.The overall objective is minimal total semantic distance across correlated names.Source: D. Program Analysis and Semantics… · pp. 19–20

E. Assumptions about Semantics Consistency and Copied Variables

The paper tests assumptions about name semantics under direct copies, notes challenges in long coding contexts, and examines how context-propagation sensitivity affects GENNM.

E. Assumptions about Semantics Consistency and Copied Variables

  • For direct-copy variable pairs, left-hand-side names more often denote broader semantics than right-hand-side names, supporting the paper’s assumptions about correlated but asymmetric names.In the reported pairs, the left side is broader in 50% of correlated cases, compared with 32% for the right side; direct-copy pairs show more semantic correlation than random pairs.Source: E. Assumptions about Semantics… · p. 20, Table VIII · p. 20

F. Reasoning Long Context Remains Challenging for Code Models

  • Code models’ performance drops on longer, more complex code-generation contexts; CodeLlama-70B reports 67.8% pass@1 on HumanEval but under 20% on a longer-context task.The passage cites prior work for the long-context comparison, [8] and HumanEval result [50], [14].Source: F. Reasoning Long Context Remains… · p. 21

G. Effects of Contexts Sensitivity

  • Increasing context sensitivity improves GENNM, but gains become marginal when sensitivity rises from 5 to 10 degrees.The default propagates names only from direct callers and callees, corresponding to one degree of context sensitivity.Source: G. Effects of Contexts Sensitivity · p. 21

H. Prompts Input to ChatGPT

The appendix describes ChatGPT prompt settings and uses examples to show how recovered names can affect binary summaries and naming judgments. It also illustrates GenNm’s use in malware reverse engineering.

H. Prompts Input to ChatGPT

  • The prompt asks a binary-program expert to suggest one meaningful name per queried identifier and return the suggestions as JSON.Identifiers are specified in a Q:[var1,var2,...] format.Source: H. Prompts Input to ChatGPT · pp. 21–22
  • The evaluation uses both zero-shot prompts and three-shot prompts containing three randomly sampled training functions with their expected outputs.In both settings, the query function follows the prompt or examples.Source: H. Prompts Input to ChatGPT · p. 22
  • More than 95% of rarely seen names can be composed from frequently seen tokens, according to the cumulative distribution of their average token frequencies.“Rarely seen” names have training-set frequencies below 10.Source: Figure 18 · p. 22

I. Case Study: Binary Summarization

  • The GPT4Evaluator examples include a prediction scored Context 4 and Semantics 2, and another scored 5 on both dimensions.The first prediction’s context is close to the ground truth, but its semantics are misleading; the second matches both context and semantics.Source: §I · p. 23, Figure 19 · p. 23
  • Renamed code produces a more context-relevant and detail-accurate BMP initialization summary than decompiled code, including the 4-byte alignment detail.The example compares both versions with a source-code summary.Source: §I · pp. 22–23

ARTIFACT APPENDIX

The artifact packages GenNm’s data, checkpoints, and experiment materials, and describes the system and the claims its experiments are intended to test.

A. Description & Requirements

  • GenNm is presented as a technique for recovering meaningful variable names from stripped-binary decompilations whose variables otherwise have placeholder names.The artifact evaluation is for a version before the paper’s major revision.Source: A. Description & Requirements · p. 25

1) How to access:

  • The artifact’s DOI and corresponding GitHub repository are listed as access points for its instructions and resources.Preprocessed datasets, model checkpoints, and intermediate results are uploaded at Zenodo.Source: 1) How to access · p. 25

2) Hardware dependencies:

3) Software dependencies:

4) Benchmarks:

  • The two benchmarks are a GitHub dataset collected using a setup similar to DIRTY and a dataset reused from VarBERT.Source: 4) Benchmarks · p. 25

B. Artifact Installation & Configuration

  • Installation instructions direct users to download and unpack the Zenodo package, create a Conda environment, and install the listed dependencies.The provided commands create a Python 3.10 environment and install requirements.txt.Source: B. Artifact Installation & Configuration · p. 25

C. Major Claims •

  • The artifact claims GenNm exceeds VarBERT in precision and recall, including across projects inside and outside the training data.The claims also cover different decompilers and optimization levels, rarely seen names, GPT-Evaluator judgments, and black-box LLM comparisons.Source: C. Major Claims • · p. 25

D. Evaluation

The artifact’s evaluation instructions cover reproducing Table 1 comparisons and Fig. 13’s comparison across decompilers and optimization levels.

D. Evaluation

  • The evaluation section states that it repeats the instructions in the code repository’s README.Source: D. Evaluation · p. 25

1) Experiment (E1) [10 human-minutes + 10 computeminutes]:

  • Experiment E1 reproduces Table 1 by calculating average precision and recall for GenNm and VarBERT across functions in and out of the training dataset.The instructions map these groups to the in-project and not-in-project columns for the DIRTY and VarCorpus results.Source: 1) Experiment (E1) [10 human-minutes +… · p. 25

2) Experiment (E2) [5 human-minutes + 10 computeminutes]:

  • Experiment E2 reproduces Fig. 13, which compares GenNm and VarBERT across decompilers and optimization levels.The instructions specify IDA-O3 and Ghidra-O0; the figure labels distinguish precision and recall for projects in or out of training.Source: 2) Experiment (E2) [5 human-minutes +… · p. 26

3) Experiment (E3) [5 human-minutes + 5 computeminutes]:

The artifact instructions describe frequency-based, GPT4-based, and black-box LLM comparisons, plus subset inference runs. They report an example rare-name precision comparison and expect GenNm to outperform VarBERT in the subset runs.

3) Experiment (E3) [5 human-minutes + 5 computeminutes]:

4) Experiment (E4) [5 human-minutes + 10 computeminutes]:

  • Experiment E4 reproduces GPT4 evaluation distributions for Context Relevance and Semantics Relevance scores assigned to generated names.The example output reports counts for each score and question.Source: 4) Experiment (E4) [5 human-minutes +… · p. 26

5) Experiment (E5) [5 human-minutes + 10 computeminutes]:

6) Experiment (E6) [5 human-minutes + 240 computeminutes]:

  • For the subset runs, the expected result is that GenNm consistently outperforms VarBERT; each command takes less than two hours on one A6000 GPU.The instructions identify small subsets of DIRTY and VarCorpus.Source: 6) Experiment (E6) [5 human-minutes +… · p. 26
Loading 2306.02546v4…