Beyond Accuracy: Evaluating Self-Consistency of Code Large Language Models with IdentityChain

1 INTRODUCTION

• Conventional evaluation focuses on individual-task accuracy, while trustworthy models should preserve semantics when translating between code and specifications they generated themselves.The paper calls this property self-consistency.Source: §1 · p. 1
• Existing code-generation and code-summarization evaluations overlook self-consistency, while prior LLM probes rely on closed-domain QA and do not generalize to open-ended generation.Existing NL-to-PL assessments use tests, while PL-to-NL assessments commonly use token-based metrics.Source: §1 · p. 2
• IdentityChain formalizes self-consistency and evaluates it using Test Output Match and greedy decoding.Its chain alternates model-generated programs and specifications, checking whether successive programs have matching test outputs.Source: §1 · p. 2, Figure 1 · p. 2
• Eleven current Code LLMs, including GPT-4, are not always self-consistent, and greater accuracy does not necessarily imply greater self-consistency.The authors identify self-consistency as a distinct aspect from conventional accuracy.Source: §1 · p. 2
• The authors report TOM as effective for PL-to-NL evaluation, completing a framework that assesses NL-to-PL accuracy, PL-to-NL accuracy, and self-consistency and can expose model weaknesses.They present IdentityChain as a potential debugging tool for model developers.Source: §1 · p. 2

Evaluating Code Large Language Models

• NL-to-PL metrics range from token-based measures to execution-based Pass@K, but existing metrics assess one-time accuracy rather than consistency across a model’s own outputs.Static code similarity misses dynamic semantics, motivating execution-based measures.Source: §2 · p. 3
• PL-to-NL evaluation commonly uses BLEU and also considers ROGUE, chrF, and BERTScore, but these static metrics require references and do not separate semantics from syntax.The paper proposes TOM as a dynamic, reference-free metric for PL-to-NL evaluation.Source: §2 · p. 3

Evaluating Self-Consistency of Large Language Models

• Earlier LLM self-consistency studies use semantically related inputs, but their fixed-ground-truth tasks do not generalize to open-ended generation.The cited settings include arithmetic reasoning and cross-language question answering; this paper studies NL-to-PL and PL-to-NL generation.Source: §2 · p. 3

3.1 SELF-CONSISTENCY DEFINITION

• The formulation defines NL-to-PL and PL-to-NL functions over valid programs and valid, unambiguous specifications, with experiments using English specifications for Python programs.The natural-language domain excludes nonsensical or ambiguous specifications.Source: §3.1 · p. 3
• Self-consistency requires the program, its model-generated specification, and the next generated program to share semantics across repeated translations.The intended property can hold through infinitely many translation steps.Source: §3.1 · p. 3
• Accuracy measures one-step translation, whereas a model can be self-consistent while preserving an error, or accurate yet weakly self-consistent.The definitions focus on semantic behavior rather than requiring the initial program to match the original specification.Source: §3.1 · pp. 3–4
• The paper’s scope is program input-output semantics, not pragmatic alignment such as complexity, security, or readability.These pragmatic aspects are noted as separate from the semantics considered in the paper.Source: §3.1 · p. 3
• A semantics space and interpretation function map each valid program or specification to a unique semantic element.The paper does not assume a particular definition of that semantics space.Source: §3.1 · p. 4
• Strong self-consistency additionally requires the initial generated program to match the original specification semantically.Self-consistency and strong self-consistency are defined with respect to a starting specification.Source: §3.1 · p. 4

3.2 SELF-CONSISTENCY EVALUATION

• Because an infinite chain and exhaustive specification set are infeasible, evaluation uses a fixed chain length and a finite input set, aggregating binary consistency outcomes into scores.The chain alternates PL-to-NL and NL-to-PL transformations, corresponding to identity transformations in the semantics space.Source: §3.2 · p. 4

3.1 SELF-CONSISTENCY DEFINITION

• Consistency over all specifications does not establish consistency over all programs, and the converse implication also fails.The paper gives a case where the NL-to-PL function maps all specifications to the same program.Source: §3.2 · p. 4, §3.1 · p. 4

4.1 EFFECTIVE SELF-CONSISTENCY EVALUATION

• Because generated specifications and programs lack ground-truth references, IdentityChain approximates self-consistency by comparing the original and back-translated programs.The paper reports a positive correlation between this program equality and the target semantic equalities.Source: §4.1 · p. 5
• TOM better correlates with human-judged semantic equality than Pass/Fail Match, supporting its use for self-consistency evaluation.For syntax and runtime errors, TOM compares full error messages rather than only error types.Source: §4.1 · p. 5

4.2 EFFICIENT SELF-CONSISTENCY EVALUATION

• Greedy decoding lets IdentityChain stop a chain when consecutive programs or specifications match exactly, determining self-consistency relative to that output.The framework uses deterministic generation to identify such cases early.Source: §4.2 · p. 6

4.3 HOLISTIC EVALUATION OF CODE LLMS

• IdentityChain combines NL-to-PL Pass@1, PL-to-NL SC1 based on TOM, and SSCn, which accounts for both accuracy and self-consistency.Developers can inspect SCn, Pass@1, and SC1 alongside SSCn to distinguish accuracy shortfalls from consistency shortfalls.Source: §4.3 · p. 6
• IdentityChain can pinpoint self-inconsistent cases to expose subtle model weaknesses for developers to examine.The framework is presented as supporting model debugging.Source: §4.3 · p. 6

• HumanEvalPlus-Mini-v0.1.6 provides 164 HumanEval problems with 16.5 test cases on average, while sanitized MBPP uses 257 verified problems from a 974-problem corpus.The MBPP corpus averages 3.0 test cases per problem.Source: §5 · p. 6
• Meaningful function names are replaced with “func” after the initial step so evaluation relies on code or docstring semantics rather than name-based shortcuts.The authors say an incorrect initial function body can conflict with a meaningful name and weaken evaluation soundness.Source: §5 · p. 6
• The evaluated models include StarCoderBase and Code Llama foundation models, their instruction-tuned versions, Gemini-1.0-Pro-0016, and OpenAI GPT models.The passage identifies GPT-3.5-Turbo-0613, GPT-4-0613, and GPT-4-0125-Preview.Source: §5 · pp. 6–7
• All models use one-shot prompting on both benchmarks, with customized prompt templates; instruction-tuned models receive chat-formatted prompts.Google and OpenAI models use parameter-frozen snapshots for reproducibility.Source: §5 · p. 7

6.1 SELF-CONSISTENCY OF CODE LLMS

• SSC5 declines by up to 78.0% versus initial Pass@1, while SC5 declines by up to 43.8% versus SC1, showing models fail to preserve consistency across steps.The reported declines occur across all models as the number of iteration steps increases.Source: §6.1 · p. 7, Table 1 · p. 7
• Higher conventional accuracy does not guarantee higher self-consistency: StarChat-Beta-15B beats Code Llama-Instruct-7B on Pass@1 but trails it on SC5 across both benchmarks.StarCoderBase-7B also outperforms its 15B version in SSC5 on both benchmarks despite lower Pass@1.Source: §6.1 · p. 7, Table 1 · p. 7

6.2 EFFECTIVENESS OF TOM SCORE

• TOM correlates with human judgments better than EM, CodeBLEU, and P/FM, verifying the tested conjectures about dynamic and fine-grained metrics.Table 2 reports Pearson, Spearman, and Kendall-Tau correlations against human judgments.Source: §6.2 · p. 8, Table 2 · p. 8
• TOM outperforms BLEU, ROUGE-L, chrF, and BERTScore on the 117 problems with ground-truth references and also works on 46 without them.The NL-space metric comparison is limited to cases where the original program passes all test cases and a reference is available.Source: §6.2 · p. 8, Table 2 · p. 8

6.3 EFFICIENCY OF GREEDY DECODING

• Greedy decoding is more efficient and appropriate for self-consistency evaluation; exact repetition of a program or specification lets IdentityChain stop that chain.Under deterministic greedy decoding, an exact repeat means the same program and specification will recur.Source: §6.3 · p. 9

6.4 IDENTITYCHAIN AS A MODEL DEBUGGING TOOL

• Models often overlook data types, and inaccurate interpretations can lead to erroneous code use, including memory-management, performance, or execution problems.IdentityChain is presented as a way to expose weaknesses that accuracy-oriented evaluations do not capture.Source: §6.4 · p. 9
• Models may summarize explicit checks while missing implicit logic such as an else-branch, producing misleading or ambiguous documentation.The paper identifies this failure to capture implicit code semantics as a major root cause of non-self-consistency.Source: §6.4 · p. 9
• When summarizing code, models can produce correct specifications but incorrect input-output examples, revealing weak prediction of execution outcomes.The paper notes this weakness is concerning for generating test cases in test-driven development.Source: §6.4 · p. 9

• IdentityChain evaluates self-consistency alongside conventional accuracy, addressing a gap in current Code LLM evaluations.The paper also presents it as a tool for studying model weaknesses and potentially inspiring future improvements.Source: §7 · p. 9
• IdentityChain can help model developers study Code LLM weaknesses and potentially inspire future improvements.Source: §7 · p. 9
• The paper points to future directions that could extend self-consistency evaluation or improve Code LLMs using IdentityChain.Source: §7 · p. 9

Introducing PL-to-PL and NL-to-NL Generation

• IdentityChain could be extended to support additional programming and natural languages, including code and machine translation tasks.Source: A FUTURE WORK · p. 13

Studying Weaknesses of Code LLMs

• Future work could identify more subtle Code LLM weaknesses and investigate their relationship to training data.The paper suggests these weaknesses may be barely addressed by current training paradigms.Source: A FUTURE WORK · p. 13

Fine-tuning Code LLMs for Better Self-Consistency

• The authors say it remains unclear how to improve self-consistency and propose testing fine-tuning strategies for different accuracy profiles.For balanced accuracy, they suggest considering training pairs generated by the model itself.Source: A FUTURE WORK · p. 13

• GPT-3.5 and GPT-4 use temperature zero to minimize randomness because their APIs do not expose greedy decoding.Source: B EXPERIMENT CONFIGURATIONS · p. 13
• All models use greedy decoding in the main experiment; temperature zero is the stated approximation for the closed-source models.Source: B EXPERIMENT CONFIGURATIONS · p. 13
• Open-source experiments use NVIDIA RTX A6000 GPUs and FP16 inference, with prompt and generation lengths capped at 1,024 and 512 tokens.Source: B EXPERIMENT CONFIGURATIONS · p. 13

• With the original function name, GPT-3.5 generates code that conflicts with that name and later summarizes by relying on the name rather than the code.Source: Figure 4 · p. 14
• Replacing the original name with “func” lets GPT-3.5 summarize the code semantics correctly against the original docstring.Source: Figure 4 · p. 14
• IdentityChain uses the generic name “func” when generating nl_i and pl_i for i ≥1 to avoid a false self-consistency failure.Source: Figure 4 · p. 14

D GENERALIZABILITY OF GREEDY DECODING

• SSC5 model rankings mostly remain stable across temperatures, supporting the generalizability of greedy-decoding results.Source: Figure 5 · p. 15

E EXAMPLES OF CODE LLMS’ WEAKNESSES

• A code check requiring three int inputs is summarized only as requiring three “numbers,” losing data-type information.Source: Figure 6 · p. 15
• Summaries miss implicit else branches that return −1 when no qualifying number is found.Source: Figure 7 · p. 16
• Some input-output pairs in the summaries are wrong, indicating failures to predict code execution.Source: Figure 8 · p. 16