Source-linked AI summary
SAFE: Self-Attentive Function Embeddings for Binary Similarity
Luca Massarelli, Giuseppe Antonio Di Luna, Fabio Petroni, Leonardo Querzoni, Roberto Baldoni
TL;DR
Binary similarity seeks similar functions in compiled code, amid increasingly complex and widely distributed software infrastructures. SAFE maps disassembled functions to embeddings with a self-attentive network, and reports improved speed and predictive performance alongside 95% semantic classification accuracy for four algorithm classes.
Problem
Binary similarity seeks to find similar functions in compiled code, which is difficult as software infrastructures become more numerous, complex, distributed, and partly inaccessible for inspection.
Method
SAFE maps assembly instructions to vectors with word2vec and computes function embeddings using a bidirectional recurrent self-attentive network that weights contextual instruction summaries.
Results
95% classification accuracy was achieved for four broad algorithm classes, while SAFE also outperformed Gemini on nDCG and vulnerability-function recall tests.
Takeaways & Limitations
SAFE provides a faster and more precise cross-platform function-embedding solution without debug symbols or control-flow graphs, while its semantic clusters support malware-function recognition and further analysis.
Takeaways & Limitations
The study defines similarity for functions compiled from the same source code and includes partial binary fragments among its scope boundaries.
Abstract
from arXiv · showhide
The binary similarity problem consists in determining if two functions are similar by only considering their compiled form. Advanced techniques for binary similarity recently gained momentum as they can be applied in several fields, such as copyright disputes, malware analysis, vulnerability detection, etc., and thus have an immediate practical impact. Current solutions compare functions by first transforming their binary code in multi-dimensional vector representations (embeddings), and then comparing vectors through simple and efficient geometric operations. However, embeddings are usually derived from binary code using manual feature extraction, that may fail in considering important function characteristics, or may consider features that are not important for the binary similarity problem. In this paper we propose SAFE, a novel architecture for the embedding of functions based on a self-attentive neural network. SAFE works directly on disassembled binary functions, does not require manual feature extraction, is computationally more efficient than existing solutions (i.e., it does not incur in the computational overhead of building or manipulating control flow graphs), and is more general as it works on stripped binaries and on multiple architectures. We report the results from a quantitative and qualitative analysis that show how SAFE provides a noticeable performance improvement with respect to previous solutions. Furthermore, we show how clusters of our embedding vectors are closely related to the semantic of the implemented algorithms, paving the way for further interesting applications (e.g. semantic-based binary function search).
1 Introduction
SAFE addresses binary similarity by generating function embeddings directly from disassembled code, targeting faster computation and broader applicability than prior approaches. The paper evaluates its performance and shows that embeddings also support semantic classification and malware analysis.
- Binary similarity seeks similar functions in compiled code, where software scale, complexity, and inaccessible internals make automated analysis important.
- Existing embedding methods can introduce feature-selection bias, incur CFG-processing costs, require call symbols, or support only specific CPU architectures.CFG-derived features impose a 10× speed penalty relative to features extracted from disassembled code.
- The design targets rapid embedding generation and operation on stripped, statically linked binaries across AMD64 and ARM architectures.The stated goals include processing several hundreds of binaries and supporting stripped binaries with statically linked libraries.
- SAFE generates function embeddings from disassembled binaries using a self-attentive neural network that models instruction sequences with recurrent context and attention.Instruction vectors are processed as a sequence, and attention weights the GRU hidden states when forming the final embedding.
- SAFE improves over Gemini by 6% to 29% across several metrics and performs better on vulnerable-function identification.
- SAFE embeddings classify four broad algorithm classes with 95% accuracy, while malware analysis produced only one false positive among 10 functions flagged as Encryption.
2 Related Work
Prior binary-similarity research includes graph matching, execution-based signatures, intermediate representations, and embedding-based methods. SAFE is positioned against cross-platform embedding systems, especially Gemini and approaches with architecture or symbol constraints.
- Binary similarity research divides broadly into embedding-based and non-embedding approaches.
- Works not based on embeddings: Non-embedding methods use CFG matching, execution traces, code strands, runtime-collected features, or intermediate representations.These approaches include graph matching, tracelet and strand comparison, repeated execution, and semantic intermediate representations.
- Cross-Platform solutions: Cross-platform graph-based methods transform binary code into intermediate or conditional representations and compare semantic structure, but some produce signatures whose size depends on function complexity.
- Based on embeddings: Asm2Vec uses word2vec-inspired embeddings over CFG random walks, but requires libc call symbols and is limited to single-platform embeddings.
- Based on embeddings: Gemini computes embeddings from annotated CFGs containing manually selected features and is described as a state-of-the-art cross-platform deep-neural approach without call symbols.
- Based on embeddings: Other embedding-related work includes clustering-based vectors and recurrent neural networks for matching similar CFG blocks.
3 Problem Definition and Solution Overview
SAFE addresses binary similarity by representing disassembled functions as vectors while preserving structural similarity and supporting semantic classification. Its architecture combines instruction embeddings with a self-attentive neural network designed to handle long instruction sequences.
- Binary similarity asks whether functions compiled from the same source code remain similar despite different compilers.
- SAFE represents each function’s instruction sequence as a vector in R^n while preserving structural similarity relations.
- Instruction2vec first maps assembly instructions to real-valued vectors using word2vec trained on a large instruction corpus.
- A bidirectional recurrent network summarizes each instruction using its context, and attention computes the final embedding as a weighted sum of those summaries.
- SAFE uses self-attention because it performs well in natural language processing and reduces classic RNN long-memory difficulties for long binary functions.
4 Details of the SAFE, Function Embedding Network
SAFE transforms instruction sequences into fixed-size function embeddings through bidirectional recurrent summaries and multi-hop attention, then learns similarity with a shared-parameter Siamese architecture.
- SAFE denotes the complete embedding network, which converts disassembled function instructions into learned representations.
- Instruction Embeddings: Instruction embeddings are learned with skip-gram, using assembly instructions as tokens after replacing large immediates and base memory addresses with special symbols.
- Self-Attentive Network: A bidirectional RNN produces context-aware summary vectors for each instruction, with forward and backward states combined.
- Self-Attentive Network: The attention matrix assigns weights to summary vectors; multiple attention hops can emphasize different aspects of a binary function.
- Function Embedding: The attention output has fixed size r × u and is flattened and passed through fully connected layers to produce an n-dimensional function vector.
- Siamese Learning: A Siamese network applies the same embedding network to two functions and compares their vectors using a similarity score.
5 Implementation Details and Training
SAFE was implemented with standard deep-learning and binary-analysis tools, trained separately for ARM and AMD64, and evaluated using compiler- and architecture-diverse datasets with controlled splits.
- Implementation: The prototype uses Python, TensorFlow, ANGR, radare2, and IDA Pro, with Adam optimization, learning rate 0.001, and batch size 250.
- Implementation: The prototype uses GRU cells with u = 100, r = 10, da = 250, e = 2000, and n = 100.
- Implementation: Functions are truncated to m = 150 instructions, covering more than 90% of dataset functions as a trade-off between training time and accuracy.
- Instruction Models: Separate instruction2vec models are trained for ARM and AMD64 using word2vec skip-gram embeddings.
- Training Corpora: The corpora contain 547K unique AMD64 functions and 752K unique ARM functions extracted from 2.52 GB and 3.05 GB of binaries, respectively.
- Training Procedure: Training pairs include same-source similar functions labeled +1 and randomly paired dissimilar functions labeled −1, with similarity-aware dataset partitions.
6 Evaluation
SAFE is evaluated across similarity testing, function search, vulnerability search, and semantic classification. The reported results show improvements over prior systems and strong retrieval and classification performance.
- Task 1 — Platform Models: SAFE improves performance by 6.8% for single-platform models and 4.4% for cross-platform models, operating within 1% of a perfect model.
- Task 2 — Function Search: Function search achieves precision above 80% in the first 15 results and recall of 47% in the first 50 results.
- Task 3 — Vulnerability Search: Vulnerability search reaches 84% recall in the first 10 results across several vulnerabilities.
- Task 4 — Semantic Classification: Semantic classification reaches 95% accuracy on the test dataset and identifies encryption functions in real-world malware.
6.1 Task 1 - Single and Cross Platform tests
SAFE is evaluated on single-platform and cross-platform binary similarity using ROC/AUC, with results compared against Gemini. It achieves near-perfect single-platform performance and higher cross-platform AUC.
- Evaluation protocol: The study performs two disjoint tests covering single-platform and cross-platform settings.The reported figures correspond to the AMD64multipleCompilers and AMD64ARMOpenSSL datasets.
- Evaluation protocol: The evaluation measures predictive performance with ROC curves and their area under the curve, where higher AUC indicates better performance.The tests use validation and test splits for AMD64multipleCompilers and five-fold cross-validation for AMD64ARMOpenSSL.
- Single-platform results: SAFE achieves 0.99 AUC on AMD64multipleCompilers, compared with Gemini’s 0.932 AUC.The reported improvement is 6.8, with SAFE close to the perfect-case AUC of 0.99.
- Cross-platform results: SAFE achieves an average AUC of 0.992 on AMD64ARMOpenSSL, compared with Gemini’s 0.948.Across five cross-validation runs, SAFE has standard deviation 0.002 versus Gemini’s 0.006, yielding a 4.4% average improvement.
6.2 Task 2 - Function Search
SAFE is evaluated for searching similar functions in a large PostgreSQL binary dataset using precision, nDCG, and recall across ranked results. It consistently improves ranking quality and retrieval over Gemini.
- Measures: nDCG evaluates both retrieval relevance and ordering, assigning better scores when similar functions appear earlier in the result list.The measure ranges from 0 to 1 and compares ranked results with an optimal query response.
- Evaluation setup: The function-search evaluation averages precision, nDCG, and recall over 160K queries from the AMD64PostgreSQL dataset.The dataset contains 581640 functions compiled with 12 compilers and four optimization levels.
- Precision: SAFE exceeds 80% precision for k ∈[0, 15] and 60% for k ∈[0, 30], outperforming Gemini by around 10% across the range.At k ∈{10, 20, 30, 40, 50}, SAFE scores {84%, 77%, 61%, 49%, 41%} versus Gemini’s {74%, 66%, 51%, 41%, 34%}.
- nDCG: SAFE’s nDCG exceeds 80% for k ∈[0, 18] and remains above 50%, improving on Gemini by around 10% across the range.At k ∈{10, 20, 30, 40, 50, 100, 200}, SAFE scores {85%, 80%, 69%, 61%, 57%, 59%, 62%} versus Gemini’s {75%, 69%, 59%, 52%, 48%, 50%, 52%}.
- Recall: SAFE reaches 47% recall at k = 50 and 56% at k = 200, compared with Gemini’s 39% and 45%.The reported recall values across the tested k values are higher for SAFE than for Gemini.
6.3 Task 3 - Vulnerability Search
SAFE is evaluated for retrieving vulnerable binary functions across multiple compiled vulnerabilities. It outperforms Gemini, reaching 84% recall among the first 10 results and finding all 13 Heartbleed functions in the first 13 results.
- Dataset and methodology: The vulnerability dataset contains binaries compiled with 11 compilers, covering 810 vulnerabilities and 3,160 disassembled functions.
- Results: SAFE outperforms Gemini for every tested k, with 84% versus 55% recall at k = 10 and 87% versus 58% at k = 15.
- Results: Recall reaches a maximum of 88% for SAFE versus 76% for Gemini, while accuracy declines because each vulnerability has 7.6 similar functions on average.
- Caveat: Some vulnerable functions are lost during disassembly, constraining the evaluated function set.
6.4 Task 4 - Semantic Classification
SAFE embeddings support semantic classification of binary functions across four algorithm categories and show visible class separation in a two-dimensional projection. They also identify encryption-related functions in Windows ransomware despite training on AMD64 ELF executables.
- Dataset and methodology: An SVM classifier is trained and evaluated with 10-fold cross-validation on SAFE embeddings and compared with Gemini embeddings.
- Dataset and methodology: The Semantic Dataset contains 15,158 binary functions from 443 annotated source functions across encryption, sorting, string manipulation, and mathematical algorithms.
- Quantitative results: SAFE embeddings outperform Gemini embeddings in all classes, with improvements ranging from 2% to 10% and a weighted average improvement of around 6%.
- Qualitative analysis: A t-SNE projection shows relatively clear separation between the four algorithm classes in the embedding space.
- Malware use case: In ransomware samples, the classifier flags confirmed encryption-related or helper functions, with one false positive reported for Vipasana.
- Malware use case: The semantic classifier is applied to Windows 32-bit malware although the system was trained on ELF executables for AMD64.
7 Speed considerations.
SAFE avoids control-flow-graph construction and computes embeddings rapidly after disassembly. End-to-end processing is under five minutes for PostgreSQL and under four minutes for OpenSSL, while Gemini takes substantially longer.
- Runtime design: Disassembling a function with radare2 is 10 times faster than computing its control-flow graph.
- Runtime design: An Nvidia K80 computes embeddings for 1,000 functions in approximately 1 second.
- Measured runtimes: Processing 3,432 binaries and 32,592 resulting functions takes 235 seconds for disassembly and preprocessing plus 33.3 seconds for embedding computation.
- Measured runtimes: End-to-end embedding computation takes less than 5 minutes for PostgreSQL and less than 4 minutes for OpenSSL.
- Comparison: Gemini requires 43 minutes for PostgreSQL and 26 minutes for OpenSSL, up to 10 times slower than SAFE.
8 Conclusions and future works
The paper concludes that SAFE produces faster and more precise cross-platform function embeddings without debug symbols or control-flow graphs. Its semantic-classification results motivate richer binary analysis, while future work considers incorporating libc symbols.
- Conclusions: SAFE computes cross-platform function embeddings without requiring debug symbols or control-flow graphs.
- Conclusions: SAFE processes more than 100 functions per second end to end and is described as both faster and more precise than previous solutions.
- Conclusions: Semantic detection experiments are presented as a basis for developing more complex binary classifiers.
- Future work: Future work will retrain the instruction embedding model with libc call symbols to quantify their effect on embedding quality.
- Future work: Using libc symbols could support finer semantic distinctions, such as whether encrypted content is sent through a socket or written to a file.