Source-linked AI summary
Large Language Models for Code Analysis: Do LLMs Really Do Their Job?
Chongzhou Fang, Ning Miao, Shaurya Srivastav, Jialin Liu, Ruoyu Zhang, Ruijie Fang, Asmita, Ryan Tsang, Najmeh Nazari, Han Wang, Houman Homayoun
TL;DR
Systematic evidence on LLM code analysis, particularly for obfuscated code, remains limited. This paper evaluates LLMs on code datasets and real-world cases; larger GPT models perform impressively on code analysis, while obfuscated-code explanation remains more useful than de-obfuscated code generation.
Problem
Systematic evaluation of LLM code analysis is limited, especially for obfuscated code, despite existing code-analysis tools’ dependence on predefined data types.
Source: §1 · p. 1, §2.4 · p. 3Method
The authors construct code and obfuscated-code datasets, systematically evaluate LLM code analysis, and conduct case studies on real-world malicious software.
Source: §1 · p. 2, §5.1 · p. 10Results
Larger GPT-series models perform impressively on code analysis, and their obfuscated-code explanations remain reasonably useful, but they have limitations generating de-obfuscated code.
Source: §8 · p. 14Takeaways & Limitations
The study finds considerable potential for LLMs in code analysis while noting that further performance optimization remains unexplored.
Source: §8 · p. 14On long code bodies, LLMs may fail to retain important information across queries under strict input limits.
Source: §5.3 · p. 13
Abstract
from arXiv · showhide
Large language models (LLMs) have demonstrated significant potential in the realm of natural language understanding and programming code processing tasks. Their capacity to comprehend and generate human-like code has spurred research into harnessing LLMs for code analysis purposes. However, the existing body of literature falls short in delivering a systematic evaluation and assessment of LLMs' effectiveness in code analysis, particularly in the context of obfuscated code. This paper seeks to bridge this gap by offering a comprehensive evaluation of LLMs' capabilities in performing code analysis tasks. Additionally, it presents real-world case studies that employ LLMs for code analysis. Our findings indicate that LLMs can indeed serve as valuable tools for automating code analysis, albeit with certain limitations. Through meticulous exploration, this research contributes to a deeper understanding of the potential and constraints associated with utilizing LLMs in code analysis, paving the way for enhanced applications in this critical domain.
1 Introduction
The paper addresses limited systematic evidence on LLM code analysis, especially for obfuscated code, and evaluates models using code datasets and real-world malicious-software case studies. It contributes datasets, a systematic evaluation, and practical case studies of LLM analysis.
1 Introduction
- Few systematic studies examine LLM code analysis, while existing AST-based tools depend on predefined data types and are unsuitable for general software code analysis.Code analysis includes tasks such as code quality assessment and vulnerability detection, and is used in security-related reverse engineering.Source: §1 · p. 1
- The authors construct two datasets containing code samples from popular programming languages and their obfuscated versions for evaluation.The study also asks whether LLMs understand source code and comprehend obfuscated code.Source: §1 · pp. 1–2
- The paper uses real-world malicious software case studies to examine LLMs’ capabilities and limitations in security-related code analysis.The introduction describes the case studies as testing whether publicly available LLMs can assist with security tasks.Source: §1 · p. 2
- The paper presents the first systematic evaluation of LLMs’ ability to comprehend both source code and obfuscated code.It evaluates publicly available GPT and LLaMA models and uses real-world malicious-software case studies to examine capability and limitations.Source: §1 · p. 2
2 Background
The background reviews LLMs, code analysis, code generation, and obfuscation, motivating evaluation of LLM analysis on transformed code. It describes prior capabilities and approaches while identifying limits in general code analysis and obfuscation’s challenge to protection.
2.1 Large Language Models
- LLMs support natural-language tasks and have shown utility for code explanation and summarization, including reported gains in programmer familiarity with codebases.Prior work reports that generated explanations can be viewed as more accurate and comprehensible than student-written explanations, while project-specific few-shot training improves code summarization.Source: §2.1 · p. 2
2.2 Code Analysis
- Code analysis examines source, byte, or binary code for quality, reliability, and security, but AST-based pattern matching depends on predefined data types.Prior work also extracts source-code features for machine-learning vulnerability detection.Source: §2.2 · p. 2
2.3 Code Generation
- LLM code-generation tools support code completion and repair, but prior work notes possible costs to security and programmer understanding.Studies cover repository-level completion and automated program repair as well as user experiences with Copilot.Source: §2.3 · pp. 2–3
2.4 Code Obfuscation
- Code obfuscation transforms programs to preserve functionality while making them harder to understand, for intellectual-property protection or to hide malicious behavior.The paper investigates JavaScript obfuscators because LLM de-obfuscation may challenge existing protections; the listings illustrate a Hello World example before and after obfuscation.Source: §2.4 · p. 3
3 Experiment Settings
The experiments evaluate five publicly available LLMs using code datasets, obfuscated samples, and manual validation with similarity measures. The setup spans three programming languages and several obfuscation approaches, with an additional de-obfuscation task on contest code.
3 Experiment Settings
- The experiment settings introduce the selected models and datasets used to evaluate LLM code analysis.The study proceeds through model selection, prompts, datasets, obfuscation, and measurement.Source: §3 · p. 3
3.1 LLM Selection
- The authors select five representative popular LLM models for deployment: GPT-3.5-turbo, GPT-4, LLaMA-2-13B, Code-LLaMA-2-13B-Instruct, and StarChat-Beta, with distinct model-specific characteristics.GPT-4 is described as an advanced general-purpose LLM; Code-LLaMAs are fine-tuned on code data, and StarChat-Beta is open-source and trained to assist coding tasks.Source: §3.1 · p. 3
3.2 Prompt Construction
- Prompts use simple code-analysis instructions or empirically supported structures such as role assignment.The paper provides its prompts in the experiment and case-study sections.Source: §3.2 · p. 3
3.3 Non-Obfuscated Code Dataset
- The non-obfuscated dataset covers JavaScript, Python, and C, drawing on benchmarks, online projects, and programming-problem solutions.Comments are removed, and the dataset spans files from a few lines to over 10,000 lines across different sources and use cases.Source: §3.3 · p. 4
3.4 Obfuscated Code Dataset
- JavaScript samples are obfuscated using JavaScript Obfuscator and Wobfuscator, with default, dead-code, control-flow, and string-splitting transformations.The study also incorporates IOCCC code and asks LLMs to generate de-obfuscated code that can compile and run, testing less-standard obfuscation.Source: §3.4 · p. 4
3.5 Measurement Method
- The authors manually validate GPT-4 explanations and use those judged correct as ground truth for comparisons among models.Four graduate or PhD-level students with more than five years of programming experience assess accuracy, with cross-checking to minimize bias.Source: §3.5 · p. 5
- Generated explanations are assessed using cosine similarity, BERT-based semantic similarity, and ChatGPT-based true-or-false evaluation.The stated ranges are 0–1 for cosine similarity and 0–5 for BERT-based similarity; the authors note that manual validation is necessary because quantitative metrics cannot directly determine correctness.Source: §3.5 · p. 5
4 Results
On non-obfuscated code, GPT-4 and GPT-3.5 performed strongly, while smaller models struggled; obfuscation reduced analysis performance, and both GPT models generated functional de-obfuscated code poorly. Results also varied with obfuscation type and code complexity.
4.1 Results on Non-Obfuscated Code Dataset
- 97.4% overall accuracy: GPT-4’s explanations aligned with the content of code samples across C, JavaScript, and Python.Each language exceeded 95% accuracy; Figure 3 presents GPT-4 accuracy.Source: §4.1 · p. 6, Figure 3 · p. 6
- GPT-4 recognized some popular repositories and used identifier names as clues, but it also made incorrect associations.For example, it linked code to the Heron project and inferred a benchmark purpose from an identifier; it also wrongly inferred pandas in a Python sample.Source: §4.1 · p. 6
- On non-obfuscated code, GPT-3.5 performed similarly to GPT-4, whereas LLaMA-series models and StarChat-Beta performed significantly worse.The smaller models often produced short, inconsistent statements or repeated input code instead of explanations, and even manually extracting natural-language output did not close the gap with GPT-3.5.Source: §4.1 · pp. 6–7, Figure 4 · p. 7
4.2 Results on Obfuscated Code Dataset
- GPT-4 reached 87% analysis accuracy on obfuscated code; under obfuscation, both models’ similarity scores degraded, while GPT-4’s scores remained consistently higher than GPT-3.5’s.Source: §4.2 · p. 7
- Basic obfuscation only slightly affected GPT models, but Wobfuscator’s inserted WebAssembly hindered understanding, and longer, more complicated code increased errors.The smaller models did not produce meaningful explanations on obfuscated code; the paper also reports GPT-4 could explain and de-obfuscate a reformatted, variable-substituted sample.Source: §4.2 · pp. 8–9
- GPT-3.5 outperformed GPT-4 at generating, compiling, and correctly running de-obfuscated code, yet both models produced functional code infrequently.GPT-3.5 generated code for all targets, with around 20% compilable and 8% of generated samples correct; GPT-4 generated code for 76%, with 19% of all samples compilable and 4% correct. Figure 6 presents success rates.Source: §4.2 · p. 9, Figure 6 · p. 9
- GPT-4 identified 22 IOCCC samples, versus 2 for GPT-3.5, but gave incorrect details about specific samples; it also refused some de-obfuscation tasks.The paper links refusals to complicated logic and specially formatted code, while reporting that text-level obfuscation did not affect de-obfuscation ability.Source: §4.2 · pp. 9–10
- Despite worse de-obfuscation success rates, GPT-4-generated code was more readable than GPT-3.5-generated code.The paper defines readability as improved formatting and more meaningful identifier names.Source: §4.2 · pp. 9–10
5 Case Studies
The case studies test LLMs on repository code, a decompiled Android virus, and WannaCry; they show useful analysis alongside failures that depend on context and code complexity.
5.1 Github Repository Analysis
- GPT-4 explained functions and identified malicious activity in KratosKnife, correctly classifying files that contained malicious behavior.It recognized VM-environment checks as malware-analysis countermeasures.Source: §5.1 · p. 11
- GPT-4 falsely classified independent encryption utilities as malicious and accused them of replacing files, despite its explanation saying encrypted code was stored separately.The discrepancy led the authors to call for thorough examination of LLM classifications.Source: §5.1 · p. 11
5.2 Mobile Platform Virus Analysis
- Both models deciphered the Android virus's behavior, but neither detected malicious activity from the three steps separately; only GPT-4 did when the information was combined.The authors stress that defensive analysis requires enough context.Source: §5.2 · pp. 11–12
5 Case Studies
- Across the case studies, LLMs surfaced malicious behavior, but missed threats when evidence was fragmented and made other errors on complex code.The authors conclude that LLMs can assist defensive analysis but cannot replace expert humans.Source: §5.2 · p. 12, §5.3 · p. 13
5.3 WannaCry Ransomware Analysis
- GPT-3.5 extracted WannaCry behaviors including port 445 connectivity, service creation, and an “alive” URL check, partially matching public analysis.It incorrectly described the service's purpose as disguise rather than exploitation of the SMB vulnerability.Source: §5.3 · p. 13
- On long WannaCry functions split across query rounds, GPT-3.5 missed labels already supplied, a weakness the authors link to input limits and complicated context.The code was submitted interactively in chunks of 200–500 lines.Source: §5.3 · pp. 12–13
6 Discussion
The discussion finds a gap in LLM performance across model sizes and obfuscation settings, and identifies dataset construction, memorization, and evaluation metrics as future-work areas.
6.1 Using LLMs for Code Analysis
- Larger models such as GPT-3.5 and GPT-4 have higher explanation success rates, while smaller models—including coding-fine-tuned ones—often perform poorly, and even larger models can be unreliable on complicated obfuscation.The authors consider larger LLMs the safer choice for code analysis at present.Source: §6.1 · p. 13
6.2 Future Work
- The authors identify a shortage of obfuscated-code datasets and call for studying LLM memorization, whose underlying mechanisms remain unexplored.They note that most existing work focuses on normal code.Source: §6.2 · p. 13
- Neither n-gram nor ChatGPT-based similarity metrics reliably capture code-analysis results in their experiments; the latter also required manual calibration, especially for mixed code-and-language inputs.The authors suggest developing a more sophisticated metric as a research direction.Source: §6.2 · pp. 13–14
7 Related Work
Prior work covers code analysis and LLM evaluation, but the closest cited study [88] does not include obfuscated-code comprehension or de-obfuscated code generation.
7 Related Work
- Prior studies address code structure, vulnerability detection, decompilation, and LLM code capabilities, but the closest work [88] omits obfuscated-code comprehension and de-obfuscated code generation.Other LLM evaluations cited focus on code generation or broader capability and security assessments.Source: §7 · p. 14
8 Conclusion
The paper concludes that GPT-series models perform impressively on code analysis, while evaluated LLaMA models perform unsatisfactorily. On obfuscated code, GPT models remain useful for explanations but struggle to generate de-obfuscated code.
8 Conclusion
- GPT-series models show impressive code-analysis performance, whereas the evaluated smaller LLaMA models perform unsatisfactorily.The conclusion also notes LLM memorization phenomena.Source: §8 · p. 14
- For obfuscated code, GPT models produce reasonably useful explanations but have limitations generating de-obfuscated code.The authors describe broader optimization opportunities as unexplored.Source: §8 · p. 14