Source-linked AI summary
Evaluating Language Models on Cross-Language Code Functional Equivalence
Hui Sun, Anderson Uchôa, Rohit Gheyi, Wesley K. G. Assunção
TL;DR
Existing evaluations may overstate LLM semantic code understanding because they emphasize single-language or synthetic programs. This paper introduces PolyHuman, evaluates equivalence judgments within and across languages, and finds unreliable, difficulty-dependent behavior with language- and model-specific failures. The authors conclude that current LLMs do not reliably capture functional equivalence.
Problem
Existing functional-equivalence evaluations focus largely on single-language or synthetic code, motivating tests on human-written programs across languages.
Method
The paper introduces PolyHuman, evaluates open-weight and proprietary LLMs on intra- and inter-language equivalence, and analyzes model disagreements and failure categories.
Results
LLMs show difficulty-dependent misclassification, language- and model-dependent biases, partial reliance on similarity signals, and unreliable functional-equivalence judgments.
Takeaways & Limitations
Functional-equivalence evaluation should use human-written, cross-language programs because current LLM judgments are not reliably semantic.
Takeaways & Limitations
PolyHuman labels solutions that pass all online-judge tests as equivalent, although unseen edge cases may still reveal behavioral differences.
Abstract
from arXiv · showhide
Background: Large Language Models (LLMs) have demonstrated strong performance across a variety of code-understanding tasks, leading many to believe that they can reason about program semantics. However, existing evaluations primarily focus on single-language settings or rely on synthetically generated code, raising concerns about whether current results reflect true semantic understanding. Aims: We investigate whether LLMs can accurately judge functional equivalence across different programming languages in human-written code, a setting that requires deeper reasoning beyond superficial similarity. Method: We introduce PolyHuman, a dataset of human-written programs in CPP, Java, and Python. Using this dataset, we evaluate intra- and inter-language equivalence detection across open-weight and proprietary LLMs, selecting GPT-o4-mini as a representative model to assess stability. We then manually analyze 81 cases of systematic disagreement in which models incorrectly judge functional equivalence, examining the code logic and the generated Chain-of-Thought reasoning. Finally, we categorize these failures and compare them across GPT-o4-mini, Claude-Opus-4.7, and Gemini-3-Flash to determine whether they reflect model-specific issues or broader limitations of state-of-the-art LLMs. Results: We identify a difficulty-dependent breakdown in equivalence judgment (harder problems make the model increasingly prone to misclassifying non-equivalent code as equivalent), a model-specific sensitivity to programming language for the best-performing model (particularly a more conservative behavior on Python), and a partial reliance on similarity-based cues. GPT-o4-mini also shows substantial run-to-run instability under identical settings, indicating inconsistent rather than absent capability. Conclusions: Current LLMs do not reliably capture functional equivalence within or across languages.
1 Introduction
The paper tests whether LLMs can judge functional equivalence in human-written programs across languages, where superficial similarity is unreliable. It introduces PolyHuman and reports systematic, difficulty- and model-dependent failures that challenge claims of robust semantic code understanding.
- Research gap and benchmark: Existing evaluations often use intra-language or synthetically transformed programs, allowing lexical or structural cues to substitute for semantic reasoning.Cross-language human-written programs can differ substantially in syntax, structure, and idioms despite equivalent behavior.
- Research gap and benchmark: PolyHuman contains human-written CPP, Java, and Python programs for evaluating functional equivalence within and across languages.The benchmark is designed to capture realistic implementation variation rather than synthetic transformations.
- Study scope: The study evaluates intra-language and inter-language equivalence assessment and examines complexity, similarity signals, language characteristics, and recurring error patterns.The research questions compare cross-language performance with intra-language baselines and investigate systematic prediction biases.
- Main findings: Models show systematic, model-dependent biases, with cross-language settings particularly prone to conservative non-equivalent judgments or over-acceptance of equivalence.Human-written programs expose a substantial gap between synthetic and real-world evaluation scenarios.
- Main findings: The best-performing model is partially influenced by similarity signals, indicating incomplete and inconsistent reasoning mechanisms.Overall, the findings suggest that current LLMs do not reliably reason about functional equivalence, especially across languages.
2 Methodology
The methodology constructs PolyHuman from human-written competitive-programming solutions and evaluates LLM equivalence judgments on intra- and inter-language pairs alongside established benchmarks. The setup varies languages, pair types, and benchmark construction while using controlled prompting and deterministic decoding.
- Study design: The study evaluates functional-equivalence detection in intra-language and inter-language settings using a Goal-Question-Metric framework.The design compares same-language judgments with cross-language judgments to assess generalization beyond language-specific syntax.
- PolyHuman construction: PolyHuman builds on CodeContests, whose human-written solutions cover CPP, Java, and Python and vary in code length, correctness, and implementation characteristics.These properties support analyses of problem complexity and domain effects on equivalence assessment.
- PolyHuman construction: Solutions passing all online-judge tests are treated as correct, while other outcomes are treated as incorrect when constructing code pairs.Correct–incorrect pairs are therefore treated as non-equivalent based on observed test-suite behavior.
- PolyHuman construction: The final PolyHuman dataset contains 5,035 problems after filtering and excluding problems with unknown difficulty.The pair-construction process combines sampled solutions into intra-language and inter-language categories while limiting over-representation of solution patterns.
- Baseline benchmarks: EquiBench and SeqCoBench provide Python-only comparison benchmarks built from independent programs or semantic-preserving and semantic-altering transformations.The study uses these benchmarks to contrast human-written and transformation-based evaluation settings.
- Experiment setup: The evaluation samples 500 SeqCoBench instances, uses the first 500 PolyHuman instances, and evaluates all 800 EquiBench instances.The PolyHuman subset is not sorted by difficulty, model performance, or another known property.
- Experiment setup: Prompts use zero-shot instructions focused on program behavior and constrain responses to a binary Yes-or-No decision.The study combines quantitative error distributions with qualitative analysis of incorrect predictions.
3 Results and Discussion
LLM equivalence judgments are shaped by systematic biases and superficial cues, while difficulty and code realism expose inconsistent semantic reasoning. GPT-o4-mini generally over-rejects equivalence, but increasing difficulty, code length, and similarity can increase false-equivalence judgments.
- Synthetic benchmarks can create an accuracy illusion because balanced labels allow near-constant Yes or No predictions to appear reasonable.Models show systematic false-positive or false-negative tendencies rather than symmetric errors.
- GPT-o4-mini’s accuracy declines from 0.97 on SeqCoBench and 0.91 on OJ-V to 0.82 on OJ-A and 0.87 on PolyHuman Python.The decline accompanies a shift from synthetic or renaming-based pairs toward independently written implementations.
- GPT-o4-mini detects Python Pass vs. Fail pairs at 0.93 accuracy but Python Pass vs. Pass pairs at 0.80 accuracy.This pattern indicates difficulty recognizing equivalence between structurally diverse implementations.
- GPT-o4-mini over-rejects equivalence across languages, whereas Qwen2.5-Coder-14B-Instruct has the most balanced decision gaps.Qwen’s gaps are 0.013 for CPP, 0.005 for Java, and 0.073 for Python, while model-specific differences exceed the common language effect in the evaluated set.
- Larger overall code inputs increase false-equivalence judgments on NEQ pairs, while EQ Yes rates remain high and stable.The effect is stronger for min_LOC, max_LOC, and sum_LOC than for size-imbalance measures.
- Similarity correlates with correct EQ Yes judgments but also creates shortcut bias on NEQ pairs, increasing false equivalences.For EQ pairs, CodeBLEU has ρ = 0.87 and UniXcoder ρ = 0.84; for NEQ pairs, CodeBERT has ρ = 0.76.
- As difficulty increases, GPT-o4-mini misses more true equivalences and produces more false equivalence judgments, reducing discrimination between EQ and NEQ pairs.The reported NEQ false-equivalence rate rises from 5.5% to 33.3%.
4 Actionable Implications
The analysis identifies structured but incomplete model reasoning and recommends stronger evaluation and human oversight for high-risk uses.
- GPT-o4-mini’s predictions combine ground-truth labels with code-similarity signals, yielding Pseudo R2 = 0.429.Ground-truth equivalence dominates code similarity, but both contribute to predictions.
- Manual inspection of targeted disagreement cases provides only a lower bound on dataset label noise.
- Specific abstraction-level gaps substantially diminish LLM self-correction, making human-in-the-loop validation valuable in high-risk settings.
- Claude-Opus-4.7 performs better on abstraction and intermediate reasoning, whereas Gemini-3-Flash is comparatively stronger on domain- or language-specific factual knowledge.
5 Threats to Validity
PolyHuman’s labels, sampling context, and model coverage constrain interpretation, although observed labeling issues have limited overall impact and contamination would bias results upward.
- 17 of 900 instances (1.9%) exposed potential label or validation issues because passing test suites may miss unseen edge cases.PolyHuman treats two solutions passing all online-judge tests as functionally equivalent, despite incomplete input coverage.
- CodeContests may overlap with model training data, which would overestimate performance and make poor equivalence judgments a conservative finding.
- Competitive-programming programs may not fully represent real-world code, limiting external validity.
- The evaluation primarily covers open-source models with 3B–20B parameters plus GPT-o4-mini, restricting model-selection breadth.
6 Related Work
Prior work established strong but inconsistent LLM code-understanding performance and evaluated functional equivalence mainly with single-language or synthetically altered programs.
- LLMs for Code Understanding: LLMs perform strongly on code generation, summarization, and translation, but empirical software-engineering studies find results inconsistent across tasks and settings.
- Functional Equivalence and Clone Detection: Functional equivalence supports refactoring, migration, and clone detection, motivating its evaluation as a code-understanding capability.
- Functional Equivalence and Clone Detection: Existing equivalence benchmarks such as EquiBench and SeqCoBench provide relevant evaluations but construct non-equivalent pairs through synthetic transformations.
7 Conclusion
The paper introduces PolyHuman to test functional-equivalence judgments on human-written CPP, Java, and Python programs across intra- and inter-language settings. Results show sharp difficulty-related degradation, model-dependent biases, partial similarity reliance, and a need for more stable and realistic evaluation.
- PolyHuman is a dataset of human-written programs in CPP, Java, and Python for cross-language functional-equivalence evaluation.
- Performance declines sharply on human-written code, while decision biases appear more model-dependent than uniformly language-driven.
- GPT-o4-mini increasingly misclassifies non-equivalent code as equivalent on complex problems, showing difficulty-conditional errors.
- Future benchmarks should use diverse real-world programs, reduce superficial-similarity reliance, and improve reasoning stability across runs.