PaperlayerView exact revision on arXiv

Source-linked AI summary

Robustness and Trade-offs for Code LLMs on Protected Code

Jin Wen, Yuejun Guo, Yujie Ma, Qiang Hu, Maxime Cordy

arXiv:2609.04220v1cs.SE

TL;DR

Protected-code pipelines lack systematic evidence about whether deobfuscation improves code LLM performance across models and transformations. The paper evaluates plain, obfuscated, and same-model deobfuscated inference on executable translation and completion benchmarks, finding that direct inference often matches or exceeds restoration and that higher-capability models retain about 90% Pass@1 on obfuscated translation inputs. These results support model-aware routing, while the study’s conclusions remain bounded by function-level benchmarks and coupled same-model restoration.

  • Problem

    Protected-code pipelines lack systematic evidence about whether deobfuscation improves code LLM performance across models, languages, and protection methods.

  • Method

    The study compares plain, obfuscated, and deobfuscated inference across seven models, four source languages, five protection methods, and executable translation and completion tasks.

  • Results

    Direct inference often matches or exceeds restored inference, while GPT-4.1 and Qwen3-Coder-30B reach 90.0% and 90.5% Pass@1 under obfuscated inference.

  • Takeaways & Limitations

    The findings support model-aware pipeline design using direct inference first and restoration conditionally when verification fails or confidence is low.

  • Takeaways & Limitations

    The study is limited by HumanEval-style function-level tasks and same-model restoration, which couple restoration quality with downstream robustness.

Abstract

from arXiv · show

Code large language models (LLMs) are increasingly used on software artifacts that may be intentionally obfuscated for intellectual-property protection, reverse-engineering resistance, or controlled access. In reverse engineering and security analysis, deobfuscation is commonly treated as the preprocessing step before downstream analysis or inference, yet its utility for code LLM pipelines has not been systematically validated across models and protection methods. We present an execution-based study of seven code LLMs on protected-code translation and completion, spanning source programs from C++, Go, Java, and JavaScript, five obfuscation methods, and three inference protocols: plain, obfuscated, and deobfuscated. Our results show that direct inference on obfuscated code often matches or exceeds inference on restored code. In this controlled benchmark, higher-capability models such as GPT-4.1 and Qwen3-Coder-30B retain about 90% Pass@1 on obfuscated translation inputs, indicating that explicit restoration is often unnecessary. Same-model restoration does recover some obfuscation-induced failures, but it also degrades many cases that already succeed, with lower-capability models showing the largest net losses. Across settings, model capability is the primary factor, while source language and obfuscation method have secondary but consistent effects. Overall, our findings support model-aware pipeline design and indicate that protected-code workflows should be evaluated primarily with execution-based metrics rather than static similarity alone.

I. INTRODUCTION

Protected code is designed to preserve execution while restricting inspection, but it is unclear whether code LLMs can reliably perform downstream tasks on it. This study addresses that gap by systematically comparing direct and restored inference across models and protection settings.

  • Motivation: Obfuscation preserves program behavior while increasing the difficulty of understanding, modifying, and analyzing code.The paper distinguishes protected code as a pipeline-level term from obfuscated code as the concrete transformed artifact.
  • Operational setting: Protected-code workflows target authorized maintenance, migration, auditing, and incident response rather than adversarial intellectual-property circumvention.The central reliability question is whether restoration improves correct downstream behavior.
  • Research gap: Code LLMs may infer program behavior directly from surface representations, motivating the question of whether obfuscation prevents meaningful downstream software engineering.The paper contrasts this capability with traditional static analysis, which relies on explicit structural analysis.
  • Research gap: Prior work examines obfuscation, LLM deobfuscation, and robustness separately, but lacks a comprehensive execution-based comparison across models and protection methods.The study specifically compares plain, obfuscated, and deobfuscated inference across seven models, four source languages, and five protection methods.
  • Study contribution: The paper reframes deobfuscation as a rescue–degradation trade-off rather than assuming restoration is uniformly beneficial.This trade-off is operationalized through conditional metrics grounded in executable outcomes.

D. Robustness and Semantics-Preserving Transformations

The study asks when protected-code transformations help or harm code LLM performance and how model, language, and protection choices shape that trade-off. It evaluates translation primarily, with completion as supporting evidence, using controlled executable benchmarks.

  • Research motivation: Obfuscation can both harm and improve LLM performance by changing the surface cues available to downstream reasoning.This motivates evaluating transformation effects through task outcomes rather than assuming uniformly negative impact.
  • Research questions: The paper addresses whether direct inference or deobfuscation is preferable, which cases are rescued or degraded, and how boundary conditions affect pipeline choices.The research questions cover model capability, source language, protection method, and practical decision rules.
  • Tasks and data: Translation is the primary evaluation task because it requires processing a full source program and producing functionally equivalent output, while completion provides supporting evidence.Both tasks use HumanEval-style problems with executable tests.
  • Tasks and data: The benchmark uses 164 HumanEval-style instances per dataset setting across C++, Go, Java, and JavaScript source programs translated into Python.The translation output must be a Python function with the required entry point.
  • Scope: The controlled function-level benchmark does not represent repository-scale dependencies, build systems, or long-context maintenance tasks.This scope limits direct generalization to broader software-engineering settings.

B. Models

The study evaluates seven code LLMs spanning model families, parameter scales, and deployment types, alongside five protection methods that alter identifiers, structure, formatting, or model-facing representations.

  • Models: The benchmark includes seven models: CodeLlama-7B, CodeLlama-70B, DeepSeek-Coder-V2, DeepSeek-R1-Qwen-14B, Qwen2.5-Coder-14B, Qwen3-Coder-30B, and GPT-4.1.The selection spans open-source and proprietary models, instruction-tuned and reasoning-oriented variants, and multiple parameter scales.
  • Protection methods: The protection suite contains five methods: Identifier Rename, Dead Branch Injection, Remove Symbols, Random, and CodeCipher.Together they cover surface- and structure-level transformations, including an LLM-aware method.
  • Protection methods: Identifier transformations replace meaningful names with deterministic or seed-controlled meaningless identifiers while preserving program structure.Identifier Rename uses names such as v0 and v1, whereas Random uses unique seed-controlled alphanumeric names.
  • Protection methods: Dead Branch Injection adds unreachable junk code, Remove Symbols strips formatting elements, and CodeCipher learns transformations intended to reduce LLM code-understanding effectiveness.These methods preserve the intended program semantics while changing structural, surface, or model-facing features.
  • Restoration design: Restoration uses the same model as downstream inference, so the experiment measures a coupled same-model restoration pipeline rather than isolated deobfuscator quality.The restored code is passed to the downstream task under the same code-only output constraint.

D. Inference Protocols

Each configuration compares plain, obfuscated, and deobfuscated inference under controlled zero-shot generation. Functional correctness is measured primarily by execution-based Pass@1, with restoration analyzed through rescue and degradation rates.

  • Inference protocols: Plain-Inference uses original code, Obfuscated-Inference uses protected code, and Deobfuscated-Inference uses restored code produced from protected code.The three protocols isolate the effect of adding restoration to downstream inference.
  • Generation setup: The evaluation uses fixed zero-shot prompts, code-only outputs, 2,000-token input truncation, 2,500-token generation caps, and primarily greedy decoding.Prompts require the specified entry-point signature and outputs wrapped in triple backticks.
  • Evaluation: Pass@1 is computed from compilation, syntax, and execution checks against each task’s test harness.Execution-based testing is treated as the primary measure of functional correctness.
  • Conditional metrics: Rescue Rate measures restored success among cases that pass plainly but fail under obfuscation, while Degradation Rate measures restored failures among cases robust to obfuscation.The rates are defined over the passing-instance sets S_plain, S_obf, and S_deobf.
  • Supplementary metrics: Compilation rate, syntax validity, CodeBLEU, edit distance, AST distance, Halstead length, and CodeBERT similarity supplement execution outcomes.These signals characterize surface, structural, and representation changes but are not the primary reliability measure.

G. Statistical Analysis

Across models, direct inference on obfuscated code often remains close to plain inference, while same-model restoration more often reduces accuracy than improves it. The rescue–degradation trade-off varies substantially by model capability, with restoration losses especially pronounced for weaker models.

  • Evaluation design: The evaluation compares Plain, Obfuscated, and Deobfuscated inference using execution-based Pass@1 and paired protocol differences.Aggregate statistical tests are treated as supplementary because the practical focus is consistent changes in executable outcomes.
  • Rescue versus degradation: Rescue measures obfuscated failures that become successful after restoration, whereas degradation measures obfuscated successes that become failures afterward.Both conditional rates are defined among cases that succeed under plain inference.
  • Model-level comparison: 90.0% and 90.5% Pass@1 were achieved by GPT-4.1 and Qwen3-Coder-30B under obfuscated inference, respectively.These results were close to their plain-code performance, indicating that stronger models can often process protected inputs directly.
  • Model-level comparison: Six of seven models lost accuracy after same-model restoration, while only Qwen2.5-Coder-14B improved, gaining 2.2 percentage points from 59.6% to 61.8%.CodeLlama-70B had the largest decline, falling from 15.2% to 3.8%, a 74.8% relative reduction.
  • Item-level comparison: 78% of model–obfuscator points fell below the Plain–Deobfuscated diagonal, indicating reduced accuracy after restoration in most cases.The gap was smallest for GPT-4.1 and Qwen3-Coder-30B and largest for CodeLlama-70B.

2) Qualitative Analysis: Why Does Deobfuscation Harm Lower-Capability Models?:

Deobfuscation can rescue failures but also damage successful cases, especially for lower-capability models. Restoration artifacts and disrupted coping strategies help explain why degradation often outweighs rescue for weaker systems.

  • Restoration can introduce semantic distortion even when the resulting code remains syntactically valid.An example changes a loop bound from arr.length to arr.length - 1, causing an off-by-one error.
  • The same restoration failures occur less frequently and have smaller aggregate effects in GPT-4.1 and Qwen3-Coder-30B.The paper links this pattern to more stable internal representations under input perturbation.
  • Pipeline guidance therefore favors direct protected-code inference for strong models and cautious or avoided restoration for weaker models.The paper recommends adaptive restoration in the middle capability range and generally avoiding it below 50% plain accuracy.
  • 69.8% rescue for GPT-4.1 and 64.0% for Qwen3-Coder-30B contrast with only 6.8% for CodeLlama-70B.Rescue rates range from 32.2% to 69.8%, with lower-capability models recovering substantially fewer obfuscation-induced failures.
  • 5.7% and 8.4% degradation for Qwen3-Coder-30B and GPT-4.1 contrast with 95.2% for CodeLlama-70B.DeepSeek-R1-Qwen-14B also reaches 32.6% degradation, indicating that reasoning-oriented training alone does not ensure robustness to restoration artifacts.
  • +61.4 percentage points net effect for GPT-4.1 contrasts with −88.4 points for CodeLlama-70B.The net effect is defined as rescue minus degradation, so the balance between recovered failures and collateral damage differs sharply by model.
  • Only three of seven models have positive net effects, and all three have plain pass rates above 70%.The positive-net models are GPT-4.1, Qwen3-Coder-30B, and Qwen2.5-Coder-14B.

2) Case Study: When Rescue Succeeds vs. Fails:

The case studies show that restoration helps when it recovers semantic cues but harms when it disrupts a model’s learned ability to handle protected representations. Across analyses, capability remains the strongest determinant, while restoration outcomes vary by language and protection method.

  • When Rescue Succeeds: GPT-4.1 correctly solves an obfuscated Java binary-search method after deobfuscation restores meaningful variable names.The model initially sets the loop condition incorrectly, but succeeds once low, high, and mid are restored.
  • When Rescue Fails: CodeLlama-70B solves protected Python factorial code before restoration but produces an incorrect recursive solution afterward.Dead-code removal simplifies the structure yet disrupts the model’s prior coping behavior.
  • Interpretation: Rescue succeeds when restoration restores lost semantics, whereas degradation occurs when it disrupts learned coping strategies.These two mechanisms explain the contrasting case-study outcomes.
  • Model Capability: 92.9% plain pass rate for GPT-4.1 and 88.3% for Qwen3-Coder-30B accompany low degradation and positive net effects.By contrast, CodeLlama-70B’s 16.8% plain pass rate accompanies the largest degradation.
  • Model Capability: Higher-capability models maintain 82–92% and 82–94% deobfuscation pass rates across languages, while CodeLlama-70B remains at 2–5%.This consistency supports model capability as the primary factor rather than language-specific effects.
  • Metric Robustness: The supplementary robustness ranking preserves the model-capability ordering across static, vector, and execution signals.This supports restoration sensitivity as a model property rather than an artifact of a single metric family.

2) Language Effects are Secondary but Consistent:

Language and protection method affect restoration outcomes, but their effects are secondary to model capability. These factors matter most as conditional routing signals because behavior varies across model–language and model–method combinations.

  • Language Effects: 41–48% rescue rates are relatively consistent across languages, while degradation varies from 24% to 34%.Go has the highest degradation at 34.4% and the lowest deobfuscated pass rate at 47.4%; Java has the lowest degradation at 24.5%.
  • Language Effects: Go appears more sensitive to restoration artifacts, whereas Java shows the most stable post-restoration behavior.The paper suggests rigid syntax, type discipline, and idiomatic constructs may make Go artifacts more consequential.
  • Conditional Effects: Language effects are heterogeneous across models: strong models remain consistent, while mid-tier models show opposing language-specific weaknesses.DeepSeek-Coder-V2 drops on Go and JavaScript, whereas Qwen2.5-Coder-14B is less stable on JavaScript but remains strong on Java.
  • Routing Implications: Source language is therefore better treated as a conditional routing feature than as a global policy variable.Language becomes operationally important when paired with model capability.
  • Protection Methods: 31.2% rescue for CodeCipher on Java contrasts with 51.7% for Random and 50.7% for Identifier.CodeCipher also shows moderate degradation at 24.6%, illustrating that protection methods alter the rescue–degradation balance.
  • Protection Methods: CodeCipher-style learned protection appears more effective for preventing LLM-based deobfuscation, while simpler methods may better preserve utility when restoration is needed.The paper still identifies model capability as the dominant factor, with language and method as secondary boundary conditions.

D. RQ4: Practical Guidance

The paper recommends model-aware, obfuscated-first pipelines that invoke restoration selectively and validate outputs with execution-based checks. Pass@1 should drive decisions because static and vector metrics can diverge from functional correctness.

  • Decision Rules: Models above roughly 80% plain pass rate should usually process protected inputs directly.GPT-4.1 and Qwen3-Coder-30B exemplify this group because restoration adds little benefit while retaining degradation risk.
  • Decision Rules: Models in the 50–80% range should attempt obfuscated inference first and reserve restoration for evidence of an unreliable first pass.Useful triggers include failed execution, low confidence, or disagreement with a downstream verifier.
  • Decision Rules: Models below 50% plain pass rate should generally avoid deobfuscation because degradation risk exceeds rescue potential.CodeLlama-70B falls from 15.2% to 3.8%, a 75% relative drop, after restoration.
  • Implementation: An obfuscated-first deployment should test direct outputs, invoke restoration only after failure or low confidence, and escalate when neither path passes.The same execution or lightweight verifier should check both paths.
  • Metric Selection: 19.1% of translation cases show favorable Pass@1 changes, compared with 42.6% showing favorable CodeBLEU changes.This divergence demonstrates why static metrics alone can overstate certainty.
  • Metric Selection: Pass@1 should be the primary decision criterion, with compilation rate as a secondary syntax-level check.CodeBLEU, EditDist, and vector similarity remain descriptive context rather than interchangeable measures of execution success.
  • Output Characteristics: Deobfuscated outputs are slightly longer but have lower estimated AST-based cyclomatic complexity than plain-inference outputs.For translation, mean NLOC changes from 7.60 to 8.50 while complexity changes from 3.31 to 3.10; completion shows similar trends.

3) Cost–Benefit Analysis:

Deobfuscation has mixed operational value: it can recover failures but also disrupt cases that already succeed. The trade-off depends strongly on model capability and should be judged by downstream execution reliability.

  • Cost and accuracy: 30–50% latency savings are available for higher-capability models when deobfuscation is skipped, with little accuracy loss.Lower-capability models incur extra compute and reduced accuracy when attempting restoration.
  • Model-dependent effects: GPT-4.1 and Qwen3-Coder-30B preserve more than 90% of protected-code successes while recovering a substantial fraction of failures.Restoration is more favorable for stronger models across both preservation and recovery conditions.
  • Difficulty effects: 88.8% and 90.5% after-deobfuscation pass rates remain high for GPT-4.1 and Qwen3-Coder-30B on medium-difficulty items.The corresponding rates fall to 41.9% for CodeLlama-7B and 3.9% for CodeLlama-70B; hard items remain near zero for all models.
  • Error trade-off: Restoration redistributes errors across contested cases rather than creating a broadly easier regime.This supports verifier-backed or confidence-aware routing instead of a single static preprocessing policy.
  • Evaluation criterion: Execution-based evaluation is necessary because restoration can improve representation-level similarity without improving translated-program reliability.The relevant criterion is measurable reduction in downstream failure risk, not readability or textual similarity alone.

C. Implications for Operational Deployment

Operational deployment should treat restoration as a monitored, model-aware decision rather than a fixed preprocessing step. The benchmark’s scope and implementation choices limit how broadly this guidance can be applied.

  • Monitoring: Production pipelines should retain slice-level telemetry comparing direct and restored paths across recovery, collateral-failure, and persistent-failure cases.These patterns distinguish when restoration justifies its cost from when model choice, prompting, or verification is the more relevant remedy.
  • Adaptive routing: Restoration should be triggered by evidence such as execution failure, low-confidence self-checks, or verifier disagreement rather than applied universally.The proposed policy treats deobfuscation as one candidate action in an observable decision system.
  • Protection design: CodeCipher offers strong protection with moderate degradation, whereas identifier renaming offers weaker protection but preserves more utility under authorized deobfuscation.The comparison frames protection strength and downstream utility as a design trade-off.
  • Reversibility: Explicit reversibility constraints may improve deobfuscation reliability when protected code must support debugging, migration, or maintenance.The paper suggests that CodeCipher appears to optimize reversibility implicitly.
  • Scope boundaries: The benchmark uses function-level HumanEval-style tasks rather than repository-scale code with dependencies, build systems, long contexts, and broader idiomatic variation.Repository-scale studies are needed before making broader deployment claims.
  • Task coverage: The analysis is primarily a translation study, while completion results serve mainly as a comparison point.Completion is not developed to the same depth as translation in the main text.
  • Transformation coverage: Only five obfuscation methods are evaluated, so the conclusions should not be treated as universal across transformations such as control-flow flattening and virtualization.The selected methods cover several major categories but not the full design space.
  • Inference setting: Zero-shot inference is used throughout, leaving fine-tuned, few-shot, and obfuscation-trained models outside the evaluation.Those variants may produce different robustness profiles.

VIII. THREATS TO VALIDITY

The study’s validity is constrained by its functional-correctness construct, benchmark and task scope, model and transformation choices, and implementation assumptions. Its strongest conclusions are descriptive and should not be overextended beyond the evaluated settings.

  • Construct validity: Pass@1 measures functional correctness through test execution but does not capture all aspects of code quality, including efficiency and readability.CodeBLEU and EditDist provide additional context but also have limitations.
  • Scoring protocol: Code-only scoring can disadvantage models that emit multiple alternatives or long explanations when a correct solution appears later in the response.The paper therefore treats output-style explanations as hypotheses rather than causal claims.
  • Metric interpretation: Rescue and degradation rates are conditional probabilities based on Plain-Inference success, so alternative conditioning choices would yield different numerical values.The relative ordering of models is expected to remain preserved under alternative definitions.
  • Internal validity: Model capability is confounded with factors such as training data and architecture despite comparisons across multiple model families.The descriptive association between stronger plain performance, lower degradation, and more favorable restoration outcomes cannot fully establish capability as the sole cause.
  • Implementation validity: Bugs or suboptimal obfuscation and deobfuscation implementations could affect results, and semantic equivalence is not independently proven for every restored source.The study uses deterministic transformations and syntax, compilation, and execution checks to reduce these risks.
  • External validity: Results focus on translation from C++, Go, Java, and JavaScript to Python and may not generalize to other language pairs or tasks.Surface-level tasks may show different effects from tasks requiring deep semantic understanding.
  • Model currency: The evaluated models were released between 2023 and 2025, so newer models may exhibit different robustness characteristics.The paper expects the rescue–degradation trade-off to persist when restoration introduces perturbations, but this remains a qualified expectation.
  • Claim strength: The strongest claims are descriptive, while finer-grained conclusions about failure taxonomies and adaptive routing remain suggestive rather than definitive.Large numbers of benchmark instances reduce noise but do not eliminate uncertainty in those analyses.
Loading 2609.04220v1…