PaperlayerView exact revision on arXiv

Source-linked AI summary

jTrans: Jump-Aware Transformer for Binary Code Similarity

Hao Wang, Wenjie Qu, Gilad Katz, Wenyu Zhu, Zeyu Gao, Han Qiu, Jianwei Zhuge, Chao Zhang

arXiv:2205.12713v1cs.CRcs.SE

TL;DR

BCSD needs scalable and accurate similarity analysis for applications such as vulnerability discovery, but existing methods omit either instruction semantics or control-flow information and rely on limited benchmarks. jTrans embeds control flow into a Transformer with jump-aware representations and jump task prediction, and BinaryCorp supplies a larger, more diverse benchmark. On 10,000-function pools, jTrans reaches 62.5% Recall@1 versus 32.0% for the best SOTA solution, while vulnerability searching achieves 2X higher recall than SOTA baselines.

  • Problem

    BCSD supports vulnerability discovery and other applications, but existing methods and datasets inadequately capture both binary semantics and control flow under realistic, diverse evaluation conditions.

  • Method

    jTrans embeds control-flow information into a Transformer using jump-aware representations and replaces BERT’s second pre-training task with jump task prediction; BinaryCorp is built from ArchLinux repositories.

  • Results

    62.5% Recall@1 versus 32.0% for the best SOTA solution was achieved on 10,000-function pools, and vulnerability searching produced 2X higher recall than SOTA baselines.

  • Takeaways & Limitations

    jTrans provides a Transformer-based BCSD approach that combines instruction semantics with control-flow information and is evaluated with the diversified BinaryCorp benchmark.

  • Takeaways & Limitations

    The study trains jTrans on one architecture, although the technique is described as applicable to other architectures.

Abstract

from arXiv · show

Binary code similarity detection (BCSD) has important applications in various fields such as vulnerability detection, software component analysis, and reverse engineering. Recent studies have shown that deep neural networks (DNNs) can comprehend instructions or control-flow graphs (CFG) of binary code and support BCSD. In this study, we propose a novel Transformer-based approach, namely jTrans, to learn representations of binary code. It is the first solution that embeds control flow information of binary code into Transformer-based language models, by using a novel jump-aware representation of the analyzed binaries and a newly-designed pre-training task. Additionally, we release to the community a newly-created large dataset of binaries, BinaryCorp, which is the most diverse to date. Evaluation results show that jTrans outperforms state-of-the-art (SOTA) approaches on this more challenging dataset by 30.5% (i.e., from 32.0% to 62.5%). In a real-world task of known vulnerability searching, jTrans achieves a recall that is 2X higher than existing SOTA baselines.

1 INTRODUCTION

BCSD supports applications including vulnerability discovery, malware detection, plagiarism detection, patch analysis, and supply-chain analysis, but existing approaches face limitations in semantics, control-flow modeling, dataset diversity, and realistic evaluation. jTrans addresses these gaps with a jump-aware Transformer and BinaryCorp, achieving stronger results on real-world BCSD tasks.

  • BCSD measures similarity between binary code snippets and supports vulnerability discovery, malware detection, plagiarism detection, patch analysis, and software supply-chain analysis.
  • NLP-based methods model instruction sequences but omit actual control flows, limiting semantic understanding and robustness to compiler-induced code changes.
  • CFG-only methods miss instruction semantics within basic blocks and commonly rely on GNNs that capture structural information but are difficult to train and parallelize.
  • Existing datasets are often small or insufficiently diverse, potentially causing overfitting and overstating performance, while evaluations frequently omit large candidate pools.
  • jTrans fuses instruction semantics and control-flow information directly into a Transformer rather than simply combining NLP-based and GNN-based features.
  • BinaryCorp is a large, diversified dataset built from ArchLinux repositories and varied compiler optimizations to mitigate overfitting and limited diversity.
  • 62.5% Recall@1 versus 32.0% for the best SOTA solution was achieved by jTrans on 10,000-function pools using full-sized BinaryCorp.

2 PROBLEM DEFINITION

BCSD retrieves similar binary functions by comparing their similarity scores, with one-to-one, one-to-many, and many-to-many task formulations. This study focuses on one-to-many retrieval, where candidate functions are ranked for a source function.

  • One-to-one tasks return a score for one source-target pair, one-to-many tasks rank a target pool, and many-to-many tasks group functions by similarity.
  • The study focuses on one-to-many tasks, which reduce to one-to-one when the target pool has size one and can extend to many-to-many through repeated retrieval.
  • Figure 1 contrasts linear-layout assembly containing jump addresses with the corresponding control-flow graph.
  • A binary function is an ordered instruction set compiled from a source function, with specific semantics, internal constraints, and a corresponding control-flow graph.
  • A BCSD task retrieves the top-k functions from a pool according to their similarity scores to a source function.
  • BCSD calculates similarity between two binary functions, with similar or patched source functions expected to receive similar scores.

3 RELATED WORK

Learning-based BCSD maps binary functions into vectors for efficient similarity computation, but existing approaches face limitations in feature coverage, robustness, and dataset diversity.

  • Traditional graph-based and static techniques compare CFGs, categorized instructions, instruction sequences, or expression-tree structure.These approaches include graph-isomorphism and edit-distance methods for measuring binary similarity.
  • Learning-based BCSD embeds binary functions into numerical vectors and estimates similarity using vector distance.This representation supports large-scale detection because vector-distance computation is efficient.
  • NLP-based methods model assembly sequences, whereas other approaches use graph neural networks to learn CFG representations.Some studies combine sequence and graph-based modeling, while Siamese networks commonly require mappings of equivalent binary functions.
  • Unsupervised BCSD methods avoid ground-truth equivalent-function mappings, but representative Asm2Vec performs worse than supervised state-of-the-art methods.
  • Existing datasets are often small and insufficiently diverse, which can encourage overfitting and make evaluations less representative of real-world use cases.Common sources include GNUtils, coreutils, and openssl, whose similar code patterns can produce misleadingly high performance.

4 METHODOLOGY

jTrans adapts a Transformer encoder to binary code by combining instruction semantics with explicit jump relationships and specialized pre-training. Its representation normalizes assembly tokens and links jump sources to targets through shared positional parameters.

  • jTrans modifies a Transformer encoder to learn instruction semantics and control-flow information from binary programs.The model preprocesses assembly code so jump relationships are represented in the input.
  • MLM pre-training masks assembly tokens and requires their reconstruction from neighboring context.This encourages the model to learn assembly syntax and contextual relationships among instructions.
  • Jump Target Prediction requires jTrans to predict a jump instruction’s target, encouraging learning of how jumps are incorporated into code.The paper reports that this auxiliary task contributes significantly to model performance.
  • String literals, constants, and selected function names are normalized into special tokens, while jump sources become JUMP_XXX tokens indexed by target order.Replacing absolute or relative jump addresses reduces the impact of random binary base addresses.
  • Binary code is represented as token and position embeddings, with each final input embedding formed by their sum.The representation addresses binary-code vocabularies and explicit jump relationships.
  • For each direct jump pair, the source token’s embedding shares parameters with the target token’s positional embedding.This gives source and target tokens a strong contextual connection that remains coupled during training.
  • jTrans focuses on direct jumps, while incorporating indirect-jump information is identified as future work because recognizing their targets remains an open challenge.

4.3 Pre-training jTrans

jTrans pre-trains on masked-token reconstruction and Jump Target Prediction (JTP), replacing BERT’s sentence-continuity task to learn instruction context and control flow. JTP masks jump sources and requires predicting their corresponding targets.

  • Pre-training objectives: jTrans preserves masked language modeling (MLM) and replaces BERT’s sentence-consecutive task with Jump Target Prediction (JTP).JTP is designed to improve contextual understanding of jump instructions.
  • Masked language modeling: MLM masks randomly selected tokens using an 80% mask-token, 10% random-token, and 10% unchanged replacement scheme.The task requires reconstructing masked assembly tokens from context.
  • Jump Target Prediction: JTP selects jump-source tokens, replaces them with <LOC>, and trains the model to predict each corresponding target token.The task requires learning control-flow relationships between jump sources and targets.
  • Jump Target Prediction: 92.9% accuracy is achieved on JTP, and the authors report that the task improves jTrans’s ability to learn analyzed functions’ control flow.The JTP example replaces JUMP_20 with <LOC> and predicts its target index.
  • Pre-training objectives: The overall pre-training loss sums the MLM and JTP objective functions.This jointly trains instruction semantics and jump-related contextual understanding.

4.4 Fine-tuning for Binary Similarity Detection

After unsupervised pre-training, jTrans is fine-tuned for supervised binary-function similarity detection. Contrastive learning increases similarity for related pairs and decreases it for unrelated pairs, with cosine similarity used for scoring.

  • Supervised fine-tuning: jTrans is fine-tuned to maximize similarity between similar binary-function pairs and minimize similarity between unrelated pairs.The supervised task uses function embeddings and contrastive learning.
  • Similarity scoring: Cosine similarity between function embeddings provides the similarity score for two binary functions after fine-tuning.The score is computed from the embeddings of f1 and f2.
  • Training objective: Contrastive learning trains on triples containing a query function, a similar function, and an unrelated function.The similar function may be compiled from the same source code.

4.5 Large-Scale Dataset Construction

BinaryCorp is constructed from ArchLinux and Arch User Repository packages through an automated compilation pipeline. The pipeline targets C/C++ projects, varies compiler optimization levels, and preserves function symbols for labeling.

  • Source repositories: BinaryCorp draws projects from ArchLinux’s official repositories and the Arch User Repository, which together provide a large package base.The repositories include diverse software categories and tens of thousands of packages.
  • Project selection: The pipeline selects likely C/C++ projects by inspecting PKGBUILD build commands and dependencies for compiler and language indicators.Projects using cmake, make, gcc, or g++ are retained, while projects indicating Rust, Go, or JVM dependencies are removed.
  • Compilation pipeline: A modified compiler toolchain changes optimization-related command-line parameters to automate compilation at selected optimization levels.The pipeline substitutes modified gcc, g++, clang, and clang++ compilers through CC or CXX.
  • Label collection: The pipeline replaces strip with a modified version that preserves the symbol table, enabling function offsets and labels to be collected.Many real-world projects invoke strip during compilation.

5 EXPERIMENTAL SETUP

The experiments evaluate jTrans and six baselines on BinaryCorp, a large-scale dataset built from automatically compiled ArchLinux and AUR binaries. BinaryCorp spans substantially more functions than commonly used datasets and supports one-to-many retrieval evaluation.

  • BinaryCorp dataset: BinaryCorp is produced by automatically compiling ArchLinux and AUR packages into a large collection of binaries.The dataset includes binaries generated with different optimization levels.
  • Dataset scale: BinaryCorp-26M contains approximately 26 million functions, exceeding GNUtils by more than 160 times and Coreutils by more than 339 times.Table 1 reports dataset statistics for projects, binaries, and functions.
  • Dataset scale: BinaryCorp-3M contains about 3.6 million functions from 10,265 binary programs, approximately 22 times GNUtils and 47 times Coreutils.The smaller dataset addresses scalability limits posed by BinaryCorp-26M.
  • Dataset diversity: Compared with BinKit, BinaryCorp uses five compile options across nearly 10,000 projects and is described as more diverse in developers, project size, coding style, and application scenarios.The authors position this diversity as a basis for more realistic learning and evaluation.
  • Baselines: The evaluation compares jTrans with Genius, Gemini, SAFE, Asm2Vec, GraphEmb, and OrderMatters.These baselines include non-deep-learning, sequence-modeling, and graph-based approaches.
  • Evaluation task: The study focuses on one-to-many retrieval, ranking target functions by similarity to a source function and evaluating retrieval position with defined metrics.One-to-many problems can be reduced to one-to-one or extended toward many-to-many settings.

6 EVALUATION

The evaluation compares jTrans with SOTA baselines across datasets, pool sizes, vulnerability search, jump-aware design, and pre-training. jTrans performs especially strongly in larger, more realistic search pools.

  • Experimental Setup: The evaluation uses BinaryCorp-3M and BinaryCorp-26M with function pools of 32 and 10,000, assigning entire projects to train or test sets.This design addresses varying task difficulty and reduces information leakage from related binaries.
  • BCSD Performance: jTrans outperforms its closest competitor by 0.26 MRR and over 27% recall@1 at poolsize=10,000.At poolsize=32, the corresponding margins are 0.07 MRR and over 10% recall@1.
  • Poolsize Effects: As poolsize increases, baselines decline more rapidly than jTrans, whose performance avoids sharp drops across tested pool sizes.The experiments cover pools from 2 through 10,000, with the x-axis plotted logarithmically.
  • Poolsize Effects: At poolsize=2, jTrans exceeds SAFE and Asm2Vec by approximately 2%, making small pools weak indicators of real-world performance.The paper contrasts these small pools with applications where candidate pools are often orders of magnitude larger.
  • Real-World Vulnerability Search: For CVE-2016-3183, jTrans reaches 100% recall@10 across 3,038 functions, versus 36.9% for Asm2Vec and 28.6% for SAFE.The vulnerability-search evaluation covers eight CVEs and uses each project’s full function set as the search pool.
  • Jump-Aware Design: jTrans outperforms standard BERT by 7.3% on average in Recall@1 with poolsize=10,000.The comparison evaluates BinaryCorp-3M across optimization pairs and supports incorporating control-flow information into sequence modeling.
  • Pre-training: Without fine-tuning, jTrans-zero improves over the closest baseline by 0.1 MRR and 10.6% recall@1 on BinaryCorp-26M at poolsize=10,000.At poolsize=32, SAFE exceeds jTrans-zero by 11.4%, while the larger pool is described as more representative of real-world scenarios.

7 DISCUSSION

The discussion identifies scope and efficiency boundaries for applying jTrans to real-world BCSD. Its training focuses on one architecture, while direct pairwise comparison remains a future trade-off.

  • Scope: The paper trains jTrans on one architecture, such as x86, although the proposed technique may apply to other architectures.The stated architectural coverage is therefore narrower than the technique’s intended applicability.
  • Efficiency Trade-off: jTrans and existing deep-learning methods compare vectors for individual functions, while directly comparing function pairs could better capture inter-function relationships.The paper identifies higher overhead for direct pairwise comparison and leaves balancing accuracy with overhead for future work.

8 CONCLUSION

The conclusion presents jTrans as a jump-aware Transformer solution for binary similarity detection and BinaryCorp as a large, diverse benchmark. Experiments report consistent gains over SOTA approaches.

  • Contributions: jTrans is presented as the first Transformer-based language model solution to embed control-flow information without relying on GNNs.The design uses a jump-aware architecture, and theoretical self-attention analysis is reported as supporting its soundness.
  • Findings: Experimental results show that jTrans consistently outperforms SOTA approaches by a large margin on BCSD tasks.The paper also reports weaknesses in how current SOTA methods are evaluated.
  • Dataset: BinaryCorp is released as a large, diversified benchmark intended to support future BCSD studies.The conclusion describes it as containing the largest amount of diversified binaries to date.

APPENDIX

The appendix derives properties of the modified self-attention mechanism under jump-aware positional and token embeddings. It formalizes attention through projections, matrix products, and an SVD of the jump-sharing matrix.

  • Attention Derivation: The derivation projects each layer embedding E_m into Q_m, K_m, and V_m before computing scaled dot-product attention.The proof abbreviates the layer-specific embeddings and projection parameters for the subsequent derivation.
  • Attention Derivation: The pre-softmax attention matrix A is expressed through pairwise embedding products and trace rearrangement.The displayed relation is given for every token pair i, j.
  • Matrix Decomposition: The proof applies SVD to the jump-sharing matrix J, representing it as U × S × V^T with orthogonal U and V.The diagonal entries of S are nonnegative.
  • Jump-Aware Assumption: The jump-aware assumption ties the token embedding at position i to the position embedding at its jump-connected position j.The appendix denotes the current-layer token and position embeddings as T and P, with E = P + T.
Loading 2205.12713v1…