TRACED: Execution-aware Pre-training for Source Code

• Static code models are limited for tasks requiring program behavior, including semantic-clone detection and vulnerability analysis.These tasks involve behavior under inputs, executable locations, and value flows.Source: §1 · p. 1
• In the motivating example, CodeX fails to predict branch coverage despite added data-flow hints, and fine-tuned UnixCoder predicts neither branch will be covered.The example is presented as evidence of difficulty reasoning about simple execution logic.Source: §1 · pp. 1–2
• TRACED jointly pre-trains on source code, program states, and execution coverage to learn static and dynamic code properties.Program states are defined as the values of variables in the current scope; execution coverage records which lines run.Source: §1 · p. 2
• TRACED maps concrete runtime values into 30 discrete categories to reduce data complexity and sparsity, while retaining execution patterns as learning targets.The introduction identifies noise, outliers, and irregularities in concrete values as challenges for learning.Source: §1 · p. 2
• TRACED improves statically pre-trained code models by 12.4% for execution path prediction and 25.2% for runtime variable value predictions.Source: §1 · p. 2
• TRACED reports 91.2% MAP@R on CodeXGLUE-POJ104, 50.4% F1 on ReVeal, and 65.9% accuracy on CodeXGLUE-defect-detection.The paper reports these as state-of-the-art results in code understanding tasks.Source: §1 · p. 2

Stage-1: Tracing & Feature Engineering

• Stage 1 executes source programs on executable inputs, records runtime variable values and line coverage, quantizes values, and prepares state and coverage labels.The resulting samples pair source code and program input with execution-trace labels.Source: §2 · p. 2, Figure 2 · p. 3

Stage-2: Execution-aware Pre-training with Traces

• Stage 2 uses supervised pre-training with objectives to reconstruct source code and predict program states and execution coverage.The state objective targets data flows and side effects; the coverage objective targets dynamic control flow and how states are reached and evolve.Source: §2 · p. 3

Stage-3: Task-specific Fine-tuning

• Stage 3 loads TRACED’s pre-trained weights and fine-tunes for downstream tasks without executing programs during fine-tuning.The tasks include static execution estimation, clone retrieval, and vulnerability detection.Source: §2 · p. 3

3.1 Representing Program States

• Program states are snapshots of variable values after source-code lines execute, modeled as debugger-like mappings from variables to current values.The state mapping is updated until execution terminates.Source: §3.1 · p. 3
• For loops and recursion, TRACED retains each line’s last occurring state, trading execution detail for lower model complexity and computational requirements.The authors argue final values often capture what callers or later execution use.Source: §3.1 · pp. 3–4

3.2 Quantized Variable Values

• 30 quantized-value categories address sparse, noisy concrete values while representing common variable types and runtime-value patterns.The paper cites uninitialized C variables as cases where concrete values may be meaningless.Source: §3.2 · p. 4, Table 1 · p. 4

3.3 Building Learnable Labels for Code Models

• Program-state labels pair each quantizable variable’s data type, value type, and quantized value, which models predict from source-code representations.For example, a positive-large integer is labeled (Basic, Integer, Positive Large).Source: §3.3 · p. 4
• Execution-coverage labels mark variables on executed lines “Yes” and those on unexecuted lines “No,” assigning the latter an “Unknown” quantized value.Data-type and value-type labels are retained for unexecuted variables.Source: §3.3 · p. 4

4 MODEL

• TRACED uses a 12-layer Transformer backbone with task-specific prediction heads during pre-training, then replaces those heads for downstream fine-tuning.The pre-training heads predict masked tokens, program-state labels, and execution-coverage labels.Source: §4 · pp. 4–5

4.1.1 Model Input of Pre-training.

• Each pre-training sample concatenates an executable input and source code, while execution traces serve as ground-truth labels rather than model inputs.The input is separately truncated to at most 64 executable-input tokens and 960 source-code tokens.Source: §4.1.1 · p. 5

4.1.2 Learning Execution-aware Code Representations with Traces.

• Three summed objectives train TRACED to reconstruct masked code tokens, predict traced-variable program states, and predict variable coverage.Program-state labels combine data type, value type, and quantized value; coverage labels indicate whether the variable’s line was covered.Source: §4.1.2 · pp. 5–6

4.2 Task-specific Fine-tuning

• Fine-tuning targets static coverage and runtime-value estimation, semantic-clone retrieval, and vulnerable-versus-non-vulnerable function classification.For execution estimation, the model predicts branch coverage and quantized variable values from source code and program input.Source: §4.2 · p. 6

5.1 Trace Collection

• Trace collection compiles C programs with gcc at -O0, then steps through user-defined code with GDB to record in-scope variables and function parameters.The -g option preserves debug information; standard-library functions are skipped.Source: §5.1 · p. 6

5.2.1 Pre-training Dataset.

• 121,319 execution traces from 1,805 CodeNet C challenges form the pre-training set, while 95 challenges are held out by problem to reduce leakage.Up to 200 traces are sampled per challenge.Source: §5.2.1 · p. 7

5.2.2 Downstream tasks.

• Evaluation uses held-out CodeNet challenges for execution estimation, POJ104 for clone retrieval, and REVEAL, D2A, and Devign for vulnerability detection.The setup reports path and branch metrics plus variable-value accuracy; clone retrieval uses MAP@R, while vulnerability metrics vary by dataset balance.Source: §5.2.2 · p. 7

5.3 Model Configuration

• TRACED uses a UnixCoder-initialized RoBERTaBASE backbone, is further pre-trained for 10 epochs, and uses a fine-tuning learning rate of 8e-6.The model has 12 Transformer layers, and the reported pre-training learning rates are 5e-5 and 2e-5.Source: §5.3 · p. 7

6.1 RQ1. Effectiveness of TRACED in Static Estimation of Execution

• 71.6% of held-out samples have correctly predicted complete execution paths, and all variable values are accurately predicted in 49.2% of executions; the improvements over UnixCoder are 12.4% and 25.2%, respectively.The comparison covers static execution coverage and dynamic variable values.Source: Table 3 · p. 8, §6.1 · p. 8

6.2 RQ2. Effectiveness of TRACED’s Pre-training Objectives

• Removing any one of TRACED’s three pre-training objectives noticeably hurts static execution estimation, supporting joint learning of static and dynamic code properties.Variants were pre-trained with one objective removed and then fine-tuned on static execution estimation.Source: §6.2 · p. 9

6.3 RQ3. Effectiveness of TRACED’s Quantized Variable Values

• TRACED’s quantized variable values reduce concrete-value sparsity while retaining categories detailed enough to distinguish values for execution-path reasoning.Concrete values perform poorly against value abstractions; Figure 7 reports that both LExecutor abstractions perform slightly worse than TRACED.Source: §6.3 · p. 9, Figure 7 · p. 9

6.4 RQ4. TRACED’s Performance in Code Understanding Tasks

• TRACED outperforms statically pre-trained models in clone retrieval and vulnerability detection, with the authors suggesting that general execution estimation helps capture code semantics.The section also notes that potential anomalies in vulnerable code could be directly identified in some cases.Source: Table 4 · p. 10, §6.4 · p. 10

Pre-trained Models for Source Code

• Source-code pre-training includes encoder-only, decoder-only, and encoder-decoder models, which use different objectives and emphasize code understanding, generation, or both.The cited objectives include masked language modeling, left-to-right token prediction, denoising, missing-identifier prediction, and method-name recovery.Source: §7 · p. 10
• Existing source-code models primarily learn static code properties, while runtime behavior and complex program states remain difficult to infer.Source: §7 · p. 10

Modeling Program Execution

• Binary-execution models use concrete register values [38–40], whereas TRACED addresses more complex source-level variable types with quantized values to reduce data complexity and sparsity.Source: §7 · p. 10
• LExecutor predicts missing execution values [45], while TRACED jointly learns code text and dynamic execution, including execution coverage, as a broader pre-training strategy.The paper reports that LExecutor’s abstractions are not perfectly aligned with TRACED’s scope.Source: §7 · p. 10
• Prior work either annotates possible executions using static information [34] or uses dynamic traces as input [19] [37]; TRACED aims to estimate dynamic semantics from static information alone.Source: §7 · p. 10

Internal Validity

• The current quantized-value design does not cover all variables, given the complexity of data structures, value ranges, and memory allocation.Source: §8 · p. 10
• Tracing uses valid, executable inputs that do not terminate the program or throw errors, which may limit the model’s ability to capture those behaviors.Source: §8 · p. 10

External Validity

• TRACED currently supports only C because it relies on a tracer; extending it to other languages requires suitable tracers that capture execution information consistently.Source: §8 · p. 10

9 CONCLUSION

• TRACED jointly learns static and dynamic code properties, improving static execution estimation and transferring execution awareness to code-understanding tasks.Source: §9 · p. 10