Source-linked AI summary
Correctness, Convergence, and AI-Generated Code Detection: A Longitudinal Study of Student and Large Language Model Code in Introductory Programming
Runlong Ye, Jing Fan, Angela Zavaleta Bernuy, Oscar Karnalim, Paul Denny, Juho Leinonen, Michael Liut
TL;DR
Generated-reference matches may reflect either AI use or independent convergence on assignments with few natural implementations, leaving their individual-attribution meaning uncertain. Across three years of CS1 Python labs, we graded LLM attempts with hidden instructor tests and compared overlap and exact code forms; sensitivity analyses used full-pass student files. Later cohorts matched generated references more often, and four constrained functions used fewer exact forms, while open-ended functions remained more diverse.
Problem
Generated-reference matches may reflect either AI use or independent convergence on assignments with few natural implementations, leaving their individual-attribution meaning uncertain.
Source: §1 · pp. 1–2Method
Across three years of CS1 Python labs, we graded LLM attempts with hidden instructor tests and compared overlap and exact code forms; sensitivity analyses used full-pass student files.
Source: §1 · p. 2, §3.3 · p. 4Results
Later cohorts matched generated references more often, and four constrained functions used fewer exact forms, while open-ended functions remained more diverse.
Source: §7 · p. 10Takeaways & Limitations
Generated reference banks can help instructors identify convergence or ambiguity before release and guide assignments toward visible reasoning, testing, and intermediate work.
Source: §7 · p. 10The retrospective models do not reproduce tools available to earlier cohorts, and the observational comparison cannot validate individual AI use or explain code changes.
Source: §6 · p. 10
Abstract
from arXiv · showhide
Large language models can generate plausible solutions to programming assignments, making it tempting to detect their use by matching student code against a reference bank of generated solutions. Yet similar code can also arise when an assignment admits only a few natural implementations, which leaves open what a match actually shows. We investigate generated-reference matching using 29,970 student submissions from ten Python labs offered in 2021, 2023, and 2025, together with 90,000 solution attempts generated retrospectively by three frontier LLMs. We validate the generated solutions using hidden instructor tests, compare code with MOSS after excluding the starter code, and examine the exact abstract syntax tree (AST) forms of selected functions. The models usually produced correct solutions and, across most assignments, converged on similar implementations. Student submissions matched the generated references more often in later cohorts, including among submissions that passed every hidden test. On tightly specified functions, the models converged on a few exact abstract-syntax-tree forms, and the number of distinct student forms also declined across cohorts, whereas open-ended functions remained diverse in both sources. Most reported overlaps were short, making the minimum match length an important choice when reviewing students' code. Finally, we discuss how instructors can build a reference bank of generated solutions before releasing an assignment to identify tasks on which generated solutions converge, decide how much review a match warrants, and redesign tasks to elicit tests, reasoning, and intermediate work. These findings support tracking population-level changes in submitted code, while attributing AI use to an individual submission would require additional evidence about how it was produced, such as prompts, revisions, intermediate code, and student disclosures.
1 Introduction
Generated-reference matching offers an alternative to classifiers whose reliability depends on similarity between checked code and training data. This study examines what such matches can reveal across assignments and cohorts, while distinguishing population-level patterns from individual attribution.
1 Introduction
- Classifier reliability depends on how closely checked code resembles training data, motivating generated-reference matching as an alternative workflow.The workflow compares student code with an LLM-generated reference bank using a similarity detector, then returns snippets for inspection; a minimum match length controls which matches are retained.Source: §1 · pp. 1–2
- Independent convergence, incomplete reference sampling, and minimum-match-length choices mean a match does not prove AI use.The study therefore evaluates matching as a population-level measurement rather than individual attribution.Source: §1 · p. 2
- The study asks how correct generated solutions are, how overlap and exact forms vary across models, tasks, and cohorts, and how match length affects coverage and interpretation.It grades attempts with hidden instructor tests, compares overlap and exact code forms, and examines matching thresholds.Source: §1 · p. 2
- The analysis covers ten Python labs, three course offerings, 29,970 student submissions, and 90,000 retrospectively generated attempts from three frontier LLMs.The same 2026 models and generation protocol are applied across cohorts; student code is compared with MOSS after starter code is excluded, and five functions receive exact-AST analysis.Source: §1 · p. 2
- The study characterizes generated-reference matching and relates its interpretation to reference correctness, assignment structure, and matching criteria.Its stated contribution supports population-level monitoring and assignment review while identifying the process evidence needed for individual attribution.Source: §1 · p. 2
2 Related Work
Prior work shows that generated code can be correct while correctness alone says little about how it was produced. Similarity and detection evidence is also conditional on task structure, evaluation context, and the claims being made.
2.1 Correctness and Revisions in Generated Code
- Prior studies found that LLMs can solve introductory programming exercises, but syntactic and functional correctness alone provide little evidence about how a solution was constructed [7] [11] [45].Studies also report that interaction strategy matters and that better submitted code does not directly establish stronger underlying knowledge [23] [24] [36] [46].Source: §2.1 · p. 2
2.2 Code Similarity Across LLMs, Tasks, and Cohorts
- Short tasks can constrain independent solutions to similar forms, while evidence across cohorts remains sparse and does not identify individual AI use [55].Required signatures, starter files, restrictions, and common idioms can narrow available forms; more open tasks may admit many implementations.Source: §2.2 · pp. 2–3
2.3 Code Similarity for Retrieval and Authorship Claims
- Similarity systems retrieve overlap, but authorship remains a separate judgment; detection performance is conditional and can degrade under domain shifts or rewriting [28] [40] [44] [50] [53].The literature motivates examining reference correctness, generated–student overlap, and matching criteria as population-level evidence rather than treating in-domain accuracy as sufficient for educational use.Source: §2.3 · p. 3
3 Method
The study retrospectively compares final student submissions with generated solutions across ten Python labs and three course offerings. Its design uses year-specific materials and tests, while describing cohort differences and restricting conclusions to aggregate statistics.
3 Method
- The study pairs final student submissions with solutions generated for the corresponding year’s assignments, grading files with year-specific instructor tests and excluding starter code from MOSS comparisons.The three research questions address correctness and revisions, MOSS overlap and exact AST forms, and coverage at increasing minimum match lengths.Source: §3 · p. 3
3.1 Course and Student Submissions
- The dataset comprises ten weekly Python labs from a 12-week CS1 course, covering topics from arithmetic and Boolean expressions to file I/O and regular expressions.The labs contain one to five functions, and task comparisons are descriptive.Source: §3.1 · p. 3
- The analysis uses 29,970 final submissions from 2021, 2023, and 2025, while course staffing, cohort composition, policies, and some assignment details varied.Labs 2–10 retained the same required function set and core specifications; each submission was evaluated against its own year’s materials and tests.Source: §3.1 · p. 3
- Submissions were de-identified, results are reported only in aggregate, and no similarity result is used to make or imply a misconduct determination.The study received university Research Ethics Board approval (#38966).Source: §3.1 · p. 4
3.2 Generating LLM Solutions
The study generates reference solutions from three frontier LLMs using a consistent retrospective protocol across cohorts. Attempts can receive checker-guided revisions, then are evaluated with hidden tests and analyzed for correctness and revision behavior.
3.2 Generating LLM Solutions
- Three frontier proprietary models generated 90,000 attempts across ten labs and three cohort years, using the same model set and generation settings for each cohort.The study selected the most capable coding model then accessible within each of three provider families; the retrospective set does not reproduce the models available to students in earlier years.Source: §3.2.1 · p. 4
3.2.1 LLMs and samples.
- Each attempt received its year-specific handout and starter file, could receive at most two revision prompts after failing the visible checker, and was graded with hidden tests only afterward.Hidden instructor tests were not included in generation prompts.Source: §3.2.2 · p. 4
3.2.2 Prompts and revisions.
- For correctness and revision analysis, the study records first-prompt visible-checker success, revision-prompt counts, and whether attempts remain unresolved after the final revision..Source: §3.2.2 · p. 4
3.3 Generated-Solution Correctness and Revisions
- Each student file and LLM attempt was run against its cohort-year hidden instructor suite, with test scores and full-suite passage reported per file.A full-pass file passes every hidden instructor test; sensitivity analyses restrict student files to full-pass submissions while retaining all generated attempts in the reference bank.Source: §3.3 · p. 4
3.4 Measuring Code Similarity with MOSS
The study uses two MOSS designs to compare generated solutions and student submissions, then tests how cohort comparisons and minimum match length affect reported matching. Starter code is excluded, but retained documentation and the analysis choices constrain how matches should be interpreted.
3.4 Measuring Code Similarity with MOSS
- For each cohort and lab, within-LLM and cross-LLM match rates use the number of possible file pairs as denominators, with within-LLM rates providing a baseline.MOSS’s maximum-occurrence setting is placed above the 3,000-file pool so frequent shared code is not discarded.Source: §3.4.1 · p. 4
3.4.1 Similarity among LLM solutions.
- The LLM–LLM design excludes starter code and docstrings; removing similar conventional doctests makes implementation comparisons more conservative.Its rates are not directly comparable with student–LLM rates, which retain docstrings.Source: §3.4.1 · p. 4, §3.4 · p. 4
3.4.2 Student–LLM matching across cohorts.
- Student–LLM matching reports pair-level incidence, student-file coverage, LLM-attempt coverage, and matched-line counts, with cohort comparisons repeated among full-pass submissions.The cohort comparison uses 2021 as the historical reference and retains lab-specific results.Source: §3.4.2 · p. 4
- Student–LLM matching retains docstrings, so a reported snippet can include nearby documentation and commonly written code, not only distinctive implementation lines.The matched-line count is therefore interpreted as the size of the MOSS-reported snippet.Source: §3.4.2 · p. 4
3.4.3 Varying the minimum match length.
- Minimum match length is varied by post-filtering the same returned pairs, measuring how the share of matched student files and LLM attempts changes with the threshold.The MOSS matching algorithm remains unchanged.Source: §3.4.3 · pp. 4–5
3.5 Counting Exact Code Forms
The exact-form analysis compares serialized ASTs for five selected functions across students, LLM attempts, and cohorts. It uses rarefaction for unequal pool sizes and includes full-pass sensitivity analyses.
3.5 Counting Exact Code Forms
- Two function bodies count as the same form only when their serialized ASTs are equal, with identifiers and body syntax retained after a leading docstring is removed.Four functions channel correct solutions toward intuitive implementations, while loopy_madness allows more implementation choices.Source: §3.5 · p. 5
- For the 2021 student–LLM comparison, LLM attempts are analytically rarefied to student-pool size, and distinct-form counts are reported per 1,000 files.The LLM value is the exact expected number of forms from the full frequency distribution; student counts use the full eligible pool.Source: §3.5 · p. 5
- Across 2021, 2023, and 2025, cohort comparisons use full eligible pools and descriptive intervals, then rarefy full-pass files to the smallest eligible pool as a correctness sensitivity analysis.The sensitivity analysis separates distinct-form counts from differences in the number of correct submissions.Source: §3.5 · p. 5
3.6 Statistical Analysis and Scope of Claims
The analysis reports uncertainty and comparison procedures suited to each outcome, while treating some results as descriptive because observations are dependent. Detector accuracy and individual AI-use attribution are outside the study’s scope.
3.6 Statistical Analysis and Scope of Claims
- Mean instructor-test scores use two-sided 95% t intervals, while MOSS incidence and student-file coverage use Poisson and Wilson intervals, respectively.Cohort annotations compare specified year pairs with rate-ratio or two-proportion tests and Benjamini–Hochberg correction.Source: §3.6 · p. 5
- Because files can appear in multiple matches and students contribute across labs, interval and test precision may be overstated; matched-line and threshold results receive no significance tests.Verified student source labels are unavailable, leaving detector accuracy and individual AI-use attribution outside the study’s scope.Source: §3.6 · p. 5
- Exact-form distributions are tested with Pearson’s χ2 test after pooling forms with fewer than five observations, with false-discovery control and Cramér’s V.These tests assess association between source or cohort and form within the five selected functions.Source: §3.6 · p. 5
4 Results
Student–LLM matching increased across cohorts, including among full-pass submissions, while exact forms declined on four constrained functions. Generated solutions also converged across most tasks, but short matches limit individual attribution.
4.1 RQ1: Generated-Solution Correctness and Revisions
- LLM mean instructor-test scores ranged from 98.93% to 99.99%, compared with 76.37%, 80.76%, and 88.03% for students in 2021, 2023, and 2025.Visible-checker success could still diverge sharply from hidden-suite full passes: on Lab 4 in 2021, Gemini and Claude achieved only 0.4% and 0.0% full-pass rates.Source: Table 1 · p. 5, §4.1 · pp. 5–6
4.2.1 Cross-LLM and Task-Level Similarity.
- Cross-LLM MOSS matches covered 87.79–88.14% of pairs, only 3.40–3.61 percentage points below within-LLM rates; Lab 10 cross-LLM rates were 13.18–22.11%.Lab 10's short regex implementation and separately submitted tests left less shared code for MOSS to match.Source: Table 3 · p. 6, §4.2.1 · p. 6
- For the four tightly specified functions, pooled LLMs yielded 1–14 expected exact AST forms per 1,000 files, versus 151–804 among students; loopy_madness yielded 846 and 929.All five functions showed an association between source and exact form (BH-adjusted q < .001).Source: Figure 1 · p. 6, §4.2.1 · p. 6
4.2.2 Changes Across Student Cohorts.
- Among full-pass files, 2025/2021 match-incidence ratios were 1.59 for GPT-5.5, 2.18 for Gemini, and 2.34 for Claude; student–LLM matching also rose across cohorts.Overall incidence was 1.06–1.19 times its 2021 rate in 2023 and 1.76–2.12 times that rate in 2025.Source: §4.2.2 · pp. 6–7, Figure 2 · p. 7
- By 2025, exact-form rates per 1,000 files fell 51.1% for my_and, 25.4% for exists_triangle, 22.9% for sum_string, and 18.4% for is_square; loopy_madness rose 5.1%.The four constrained-function declines remained 14.4–32.1% among full-pass files rarefied to common sample sizes; loopy_madness changed by +2.3%.Source: Figure 3 · p. 7, §4.2.2 · pp. 7–8
4.3 RQ3: Detection Claims at Different Minimum Match Lengths
- Across cohorts, median MOSS overlaps were 8–9 lines, and 97.0–99.4% of cross-matches contained fewer than 30 lines.At a 30-line minimum, 2.9–3.5% of 2021 student files and 10.8–16.7% of 2025 LLM attempts retained matches; a match alone does not identify AI-written files.Source: §4.3 · p. 8
5 Discussion
Task structure shaped model convergence, while later student cohorts showed more reference matches and fewer exact forms on constrained functions. The findings support pre-release assignment review and population-level monitoring, but do not identify why code changed or establish individual AI use.
5.1 Task Structure and Cross-Model Convergence
- Across ten labs, cross-model matches were about 88%, only 3.40–3.61 percentage points below within-model rates; for Labs 1–9, cross-model rates exceeded 95%.For four constrained functions, the rarefied LLM pool yielded 1–14 expected forms per 1,000 files, compared with 151–804 among 2021 student submissions.Source: §5.1 · p. 8
- Tightly constrained tasks channel solutions toward a small set of natural forms, so similarity to generated code can also reflect independent convergence on a canonical solution.The paper gives my_and as an example of a task with few natural solutions.Source: §5.1 · p. 8
- loopy_madness remained diverse in both sources, while Lab 10’s compact regular expressions show that syntax differences can leave little shared code to match.Interpret matches per task: low overlap on a short artifact may reflect surface-form differences rather than a broader solution space.Source: §5.1 · p. 8
5.2 Student Code Moved Toward the Models, and Toward Each Other
- By 2025, student–LLM match incidence was 1.76–2.12 times its 2021 level, while four constrained functions had fewer exact forms even among full-pass submissions.The relatively open-ended loopy_madness function remained diverse.Source: §5.2 · p. 9
- The cohort timing does not identify a cause: match incidence rose only slightly in 2023, and teaching, cohort, support, and specification differences could also affect submitted code.Increased LLM use is consistent with the pattern, but explaining the change requires further evidence about student work.Source: §5.2 · p. 9
- Higher lab correctness coincided with lower final-exam scores, although different exam questions limit cohort comparability and the contrast does not establish how lab success relates to learning.The paper connects this question to evidence distinguishing immediate performance from later independent performance.Source: §5.2 · p. 9
5.3 What Instructors Can Do Before Releasing an Assignment
- Before release, instructors can generate and test solutions from the exact assignment materials, then inspect task-level convergence and recurring failures to guide revisions.Regenerating the bank after a revision can show whether convergence changed under the same protocol.Source: §5.3 · pp. 9–10
- A 100-attempt pilot cost about US$5, but the study does not establish that this sample is sufficient; staged work, tests, explanations, and process records can make reasoning more visible.Prompts, revisions, intermediate code, and protected disclosures could help establish how submissions were produced.Source: §5.3 · pp. 9–10
6 Limitations and Future Work
The cohort comparison does not establish individual AI-use detection, estimate LLM-use prevalence, or explain changes in submitted code. Its retrospective reference bank, single-course corpus, surface-overlap measure, and selected-function analysis bound what can be inferred.
6 Limitations and Future Work
- Without verified submission-level source labels or contemporaneous disclosures, the observational cohort comparison cannot validate individual detection, estimate prevalence, or causally explain code changes.Course and cohort differences also prevent causal attribution of the observed shift.Source: §6 · p. 10
- The reference bank measures retrospective resemblance to three 2026 models under one protocol, not the tools available to students in 2023 or 2025; similarity rates may also change with models, sampling, and preprocessing.The corpus consists of final submissions from one Python CS1 course, and MOSS measures surface overlap.Source: §6 · p. 10
- Exact-form estimates cover five purposively selected functions and remain syntax-sensitive; neither hidden-test performance nor final-code similarity measures student understanding or effort.The paper calls for held-out code with verified process histories and replication across courses, languages, and task structures.Source: §6 · p. 10
7 Conclusion
The study compares student submissions across three course offerings with retrospectively generated LLM references, finding that task constraints shape convergence and that student–LLM resemblance increased in later cohorts. The findings support population-level tracking and pre-release assignment review, but individual attribution requires direct evidence about how code was produced.
7 Conclusion
- The study examined 29,970 student files from ten CS1 labs across 2021, 2023, and 2025, comparing them with 90,000 reference attempts generated retrospectively by three frontier LLMs.The LLMs usually produced correct solutions and often converged on the same implementations.Source: §7 · p. 10
- Later cohorts more often matched the generated references, and four constrained functions showed fewer exact forms, including among full-pass submissions; open-ended tasks retained more variation.The findings document population-level changes in resemblance and solution diversity.Source: §7 · p. 10
- Before assignment release, generated reference banks can reveal convergence or ambiguity and guide revisions that make reasoning, testing, and intermediate work visible.Attributing an individual submission’s origin remains a separate question requiring disclosures, verified process histories, or other direct evidence.Source: §7 · p. 10
A Initial Generation Prompt
The prompt directs the models to complete an introductory Python lab using its handout and starter file. It constrains output to the completed file and asks models to preserve starter structure and satisfy required tests and examples.
A Initial Generation Prompt
- The prompt requests only the complete final contents of the starter file, with no Markdown fences or explanations.It frames the task as solving an introductory Python lab.Source: A Initial Generation Prompt · p. 10
- The instructions require preserving public names, signatures, imports, constants, and surrounding starter structure while editing only incomplete or required parts.They also direct models to check the handout for required tests, examples, and doctests, and verify expected outputs.Source: A Initial Generation Prompt · p. 10
- Models receive the lab handout and starter-file contents as inputs to the prompt.The prompt names both materials explicitly as the basis for solving the assignment.Source: A Initial Generation Prompt · p. 10