Source-linked AI summary
DOBF: A Deobfuscation Pre-Training Objective for Programming Languages
Baptiste Roziere, Marie-Anne Lachaux, Marc Szafraniec, Guillaume Lample
TL;DR
Source-code pre-training has mainly reused objectives developed for natural language, raising questions about whether they exploit programming-language structure. The paper introduces DOBF, which trains models to recover identifier names from obfuscated code, and reports stronger performance than existing approaches across multiple downstream tasks, alongside successful full-file deobfuscation and name suggestion.
Problem
It is unclear whether natural-language pre-training objectives such as MLM provide the best pre-training for source code.
Method
DOBF obfuscates class, function, and variable names with shared special tokens and trains a model to recover the original names.
Results
DOBF-based models outperform existing pre-training approaches across multiple downstream tasks, including clone detection, code summarization, natural-language code search, and unsupervised code translation.
Takeaways & Limitations
DOBF supports recovering fully obfuscated code, suggesting relevant identifier names, and pre-training transformer models for programming-language tasks.
Abstract
from arXiv · showhide
Recent advances in self-supervised learning have dramatically improved the state of the art on a wide variety of tasks. However, research in language model pre-training has mostly focused on natural languages, and it is unclear whether models like BERT and its variants provide the best pre-training when applied to other modalities, such as source code. In this paper, we introduce a new pre-training objective, DOBF, that leverages the structural aspect of programming languages and pre-trains a model to recover the original version of obfuscated source code. We show that models pre-trained with DOBF significantly outperform existing approaches on multiple downstream tasks, providing relative improvements of up to 13% in unsupervised code translation, and 24% in natural language code search. Incidentally, we found that our pre-trained model is able to de-obfuscate fully obfuscated source files, and to suggest descriptive variable names.
1 Introduction
Prior pre-training objectives were developed mainly for natural language, leaving open whether they are well suited to source code. The paper introduces DOBF, which uses deobfuscation to provide a more demanding code-specific pre-training signal and improves several programming-language tasks.
- DOBF is a new pre-training objective based on deobfuscation for programming languages.
- DOBF-based pre-training outperforms MLM and existing pre-training methods on code search, code summarization, and unsupervised code translation.
- Models pre-trained with DOBF can deobfuscate fully obfuscated source files and suggest descriptive identifier names.
2 Related work
The related work covers natural-language pre-training objectives, code-oriented pre-training, and identifier deobfuscation. It motivates DOBF as a source-code-specific alternative to objectives whose masking choices were developed for natural language.
- MLM and related objectives train models to recover masked natural-language tokens, while later work also explores spans, sentence relations, and alternative corruption schemes.
- DOBF is motivated by the greater structure of source code, where generic masking can overproduce easy syntax predictions and depend on arbitrary masking choices.
- CodeBERT and GraphCodeBERT adapt natural-language pre-training to source code using token-level objectives and data-flow edge prediction.
- Prior code-deobfuscation research studies identifier-name recovery using statistical and neural models, motivated by the role of naming conventions in code comprehension and maintenance.
3 Model
DOBF replaces code identifiers with shared special tokens and trains a sequence-to-sequence model to recover their original names. This targets code semantics more directly than masking isolated tokens because repeated identifier occurrences cannot simply be copied.
- DOBF replaces class, function, and variable names with special tokens and trains the model to recover the original names.
- All occurrences of a selected identifier receive the same special token, preventing the model from copying an unmasked occurrence.
- The model receives obfuscated code and outputs a dictionary mapping CLASS_i, FUNC_i, and VAR_i tokens to their original names.
- Each identifier is obfuscated with probability p_obf, ranging from one randomly selected identifier at p_obf = 0 to all file-defined identifiers at p_obf = 1.
- Finding informative names requires understanding code semantics, making deobfuscation a more demanding pre-training task than many syntax-focused MLM predictions.
4 Experiments
The experiments evaluate identifier recovery and downstream programming-language tasks using DOBF-pre-trained transformer models. They cover single- and full-file deobfuscation, code understanding and generation benchmarks, and unsupervised translation.
- Deobfuscation: The experiments evaluate single-identifier recovery at p_obf = 0 and recovery of all identifier names at p_obf = 1.
- Deobfuscation: DOBF is used to suggest relevant names for obfuscated identifiers and to revert files whose identifiers have all been replaced with short, uninformative names.
- Deobfuscation: Identifier recovery is measured by exact accuracy and a subtoken precision, recall, and F1 score.
- Downstream tasks: The downstream evaluation includes clone detection, code summarization, natural-language code search, and unsupervised translation across Java, Python, and other supported languages.
- Experimental setup: The models use transformer architectures and are trained on deduplicated Python and Java files selected from the GitHub public dataset.
5 Results
DOBF recovers identifier names from partially and fully obfuscated code and improves downstream performance across code-related tasks. Its strongest configurations outperform established pre-training methods, while qualitative examples show semantically relevant name proposals and deobfuscations.
- 5.1 Deobfuscation: 45.6% of initial identifier names are recovered when fully obfuscated functions are evaluated with DOBF trained at pobf = 1.Training at pobf = 0.5 performs better for identifier name proposal, while MLM initialization improves performance in every configuration.
- 5.1 Deobfuscation: DOBF understands obfuscated code purposes and proposes relevant names for functions implementing matrix operations and diverse Python classes and functions.Examples include fully obfuscated functions and an LSTM cell.
- 5.2 Downstream tasks: DOBF+DAE obtains the best results on every downstream task except clone detection, where CodeBERT scores much higher than the MLM baseline.The combined objective improves over MLM on all downstream tasks, with the largest improvement in natural language code search.
- 5.2 Downstream tasks: +5.3% MRR is achieved over GraphCodeBERT on natural language code search, while Java →Python computational accuracy improves by 4.6% with beam size 10.The latter corresponds to +12.2% correct translations.
- 5.2 Downstream tasks: DOBF-based models obtain state-of-the-art results on all downstream tasks, outperforming GraphCodeBERT, CodeBERT, and MLM.The results indicate that deobfuscation is effective as a pre-training task, and that MLM and DAE provide complementary objectives when combined with DOBF.
6 Conclusion
The paper presents DOBF as a deobfuscation objective for recovering obfuscated code, suggesting identifier names, and pre-training transformer models for programming-language tasks. Across multiple tasks, DOBF-based methods outperform existing pre-training approaches without requiring source code aligned to natural language.
- 6 Conclusion: DOBF supports three uses: recovering fully obfuscated code, suggesting relevant identifier names, and pre-training transformer models.These uses are presented as applications of the deobfuscation objective.
- 6 Conclusion: DOBF-based methods outperform GraphCodeBERT, CodeBERT, and MLM on clone detection, code summarization, natural language code search, and unsupervised code translation.The paper reports these results without requiring parallel source-code and natural-language corpora.
- 6 Conclusion: DOBF leverages source-code structure to create an effective structured-noise pre-training objective.The conclusion also identifies other source-code-adapted surrogate objectives as a direction for future work.
Obfuscated Code Code Deobfuscated using DOBF
The examples show DOBF transforming obfuscated LSTM code into readable code and proposing names that reflect the semantics of fully obfuscated Python functions. The accompanying comparison uses a different TransCoder architecture and reports downstream results for that setting.
- Obfuscated Code Code Deobfuscated using DOBF: The obfuscated LSTM preserves operations such as recurrent-state updates, sigmoid and tanh transformations, and linear layers despite replacing identifiers with VAR and FUNC tokens.The corresponding deobfuscated code restores names such as hidden_size, h1, h2, and init_weights.
- Obfuscated Code Code Deobfuscated using DOBF: DOBF recovers recognizable LSTM components, including the class name LSTM and the full __init__ signature.Even when exact recovery fails, proposed tokens can remain relevant for code readability.
- Obfuscated Code Code Deobfuscated using DOBF: DOBF proposes informative names for fully obfuscated functions computing dot products, downloading HTML, testing uniqueness, taking iterable tails, and summing even elements.These proposals are described as relevant across all shown examples.
- Obfuscated Code Code Deobfuscated using DOBF: The TransCoder architecture uses 6 layers, 1024-dimensional embeddings, 8 activation heads, and 143M parameters instead of 12 layers, 768 dimensions, 12 heads, and 126M parameters.It also uses ReLU rather than GeLU activations.