A Risk-Adaptive and Evidence-Constrained Framework for Generative AI Feedback in Programming Education

1. Introduction

• Programming feedback research has examined hints and error messages, but trigger timing and the use of learner history to regulate content and specificity remain connected design questions.Prior evaluations identify correctness, completeness, comprehensibility, and repair accuracy as important quality dimensions.Source: §1 · p. 2
• Feedback must coordinate intervention timing, evidence, and assistance level because each failed submission may warrant a different response based on recent learner history.The study frames prediction and generation as parts of one educational process rather than optimizing risk scores or message fluency separately.Source: §1 · pp. 1–2
• The framework links calibrated risk to intervention timing and assistance intensity, while bounded task and code evidence grounds diagnostic statements and supports source verification.Its progressive policy moves from self-checks to targeted hints and more explicit guidance.Source: §1 · pp. 2–3
• The study analyzes 2,993 failed-submission states from 215 students, comparing student-partitioned prediction models and testing four contextual feedback conditions across 136 cases.The design also evaluates fixed-capacity and sequential intervention policies, with structural and source checks for generated feedback.Source: §1 · p. 2

2.1. Feedback as a Personalized and Timely Learning Process

• Adaptive feedback must coordinate timing and assistance intensity with observed learner need, progressing from self-checks toward localized hints as difficulty persists.Premature intervention can disrupt productive struggle, while more explicit guidance is reserved for sustained difficulty.Source: §2.1 · p. 3

2.2. Learning Analytics, Learner Models, and Self-Regulated Learning

• Trace indicators describe observable behavior, not internal states, so interpretation requires defined temporal windows and evidence beyond risk scores for claims about misconceptions.Unchanged resubmission and later task performance operationalize specific behaviors or outcomes, not complete self-regulation or knowledge mastery.Source: §2.2 · p. 4

2.3. Sequential Modeling of Learning Processes

• Ordered programming histories can distinguish routes to identical failures; recurrent and attention-based models test whether those histories improve prediction beyond structured-data baselines.Student-disjoint partitions, training-only preprocessing, prefix-restricted inputs, and validation-fitted calibration are identified as safeguards for model comparison and capacity-aware decisions.Source: §2.3 · pp. 4–5

2.4. Generative AI Feedback in Programming Education

• Generative programming feedback can provide explanations and hints, but quality evaluation must address correctness, grounding, and learner agency rather than fluency alone.Bounded learner records, source traceability, paired context comparisons, structural checks, and human review are proposed to assess grounded feedback.Source: §2.4 · p. 5

2.5. Synthesis and Research Gap

• The research gap is that timing, analytics, sequential risk, and generative feedback are typically evaluated separately, while deployable systems must connect them with auditable evidence and decisions.The framework integrates calibrated capacity-aware policies, bounded generation contexts, and source checks, while distinguishing predictive validity, policy value, and feedback quality as evaluation endpoints.Source: §2.5 · pp. 5–6

3.1. Research Design and Scope

• The retrospective design links learner-state construction, risk prediction, capacity-limited decisions, bounded feedback generation, and separate evaluation endpoints.Prediction, paired feedback generation, and replayed policy decisions address complementary components of the framework.Source: §3.1 · p. 6

3.2. Dataset, Participants, and Analytical Cohort

• The source dataset contained 17,385 graded function–test records from 215 students, aggregated into 16,365 function-level submission states across 6693 submissions.Student code was parsed for structural features but never executed.Source: §3.2 · p. 6
• The study analyzed 2993 candidate failed-submission states from 215 students, retaining each student wholly in training, validation, or test partitions.The reproducible split was 60%/20%/20%; supervised estimates used 2755 states with observed outcomes, while 238 endpoint-censored states remained in deployment-oriented records.Source: §3.2 · p. 7, Table 1 · p. 7
• Four recorded feedback conditions covered 136 matched cases and 544 messages, with a separate generation experiment producing 544 messages for structural and source checks.Recorded messages received human-reviewed overall ratings on a 0–100 scale.Source: §3.2 · p. 7

3.3. Outcome Construction and Measurement Boundaries

• Persistent failure was coded only when both of the next two observed attempts on the same student-task failed; a pass on either made the outcome zero.With fewer than two subsequent attempts and no observed pass, the outcome remained missing; future states were used only to construct outcomes.Source: §3.3 · p. 7
• An unmodified retry meant the next same-task submission had the same canonical AST, with comment-and-whitespace-normalized code compared when either version could not be parsed.This measure captures observable revision behavior related to monitoring and strategy change, not internal states directly.Source: §3.3 · p. 7
• Related-task performance measured success on the first future distinct task in the same broad content group, within seven days or after seven and within 28 days.No qualifying future task yielded a missing label; the operational definitions and coverage are summarized in Table 2.Source: §3.3 · p. 8, Table 2 · p. 8

• Static predictors captured attempt and score history, failures, timing, code-edit characteristics, AST structure, and categorical task identifiers.Student identifiers were used for partitioning and clustered evaluation, never as model inputs.Source: §3.4 · p. 8
• Preprocessing fitted transformations and categorical vocabularies on training data, while representing missing values and unseen categories explicitly.AST features came from parsing only; no student or autograder program was executed during preprocessing.Source: §3.4 · p. 8
• The primary sequence used up to 20 states from the same student, laboratory, source file, and function, ending at the decision state.Each timestep included eight score, failure, and timing values plus eight missingness indicators; shorter sequences were padded with lengths or masks supplied to the model.Source: §3.4 · pp. 8–9
• A follow-up representation used each student's latest 20 completed across-task states, restricted to timestamps no later than the current state.This was reported as a validation-informed extension of the primary representation.Source: §3.4 · p. 9

• Eight configurations compared logistic regression and gradient boosting with an MLP, GRU variants, a Transformer, and a multitask fusion GRU.The GRU and Transformer comparisons reflect sequence-modeling approaches; the target was observed future failure rather than latent mastery.Source: §3.5 · p. 9
• Fusion models concatenated 32-dimensional static and sequential representations before a prediction head, while the MLP and sequence-only GRU omitted one branch.The Transformer used a single encoder layer with four attention heads.Source: §3.5 · p. 9
• Deep models used AdamW, validation-based PR-AUC early stopping, five random seeds, and averaged predicted probabilities.The multitask model weighted the primary outcome more heavily and masked missing secondary labels from their losses.Source: §3.5 · p. 9
• Hyperparameters and model selection used validation data only, with the test set reserved for final performance estimation.Platt scaling and decision thresholds were fitted on validation predictions and fixed before test evaluation.Source: §3.5 · p. 9

3.6. Predictive Evaluation and Interpretation

• PR-AUC was the primary discrimination metric, with calibration and other performance measures assessed alongside student-level bootstrap confidence intervals.Interpretability analyses supported model audit, while learner-facing feedback was grounded in observable task or code evidence linked to the next action.Source: §3.6 · p. 10

3.7. Paired Feedback Generation and Progressive Assistance

• Four feedback conditions used the same 136 cases, varying task evidence, recent history, complete-module context, calibrated risk, and assistance-level instructions.The same language model and context limits supported within-case comparisons.Source: §3.7 · p. 10
• F4 assigned self-checks, concept reminders, or localized hints by validation-risk quartiles, with ensemble disagreement prompting a verifiable check.The disagreement rule was an operational review heuristic.Source: §3.7 · p. 10
• All 544 outputs used the same model and deterministic decoding, with evidence-based observations, feasible actions, self-checks, and optional exact code excerpts.Instructions prohibited complete solutions and claims unsupported by supplied evidence.Source: §3.7 · p. 10
• Automated checks assessed completeness, actionability, self-check content, and code references, followed by one standardized repair attempt for flagged outputs.Unverifiable references were removed and unresolved structural cases flagged for human review.Source: §3.7 · p. 10

3.8. Intervention Timing and Capacity Evaluation

• The fixed-capacity analysis ranked the same labeled test pool by calibrated model risk, alternative model risk, consecutive failures, or attempt count.Budgets ranged from 5% to 100% of eligible states, with random-alert baselines estimated from 1000 draws at each budget.Source: §3.8 · pp. 10–11
• A sequential trigger used the 80th percentile of calibrated validation risk, fixed before test evaluation and compared with failure-count triggers.Evaluation included alert coverage, precision, persistent-failure capture, and observed time to the next attempt.Source: §3.8 · p. 11
• Endpoint-censored states received predictions, but policy-performance denominators included only observed outcomes.The analysis therefore measured intervention prioritization across recorded trajectories.Source: §3.8 · p. 11

3.9. Additional Analysis

• The additional analysis modeled associations between course feedback conditions and persistent failure, and summarized exit-survey responses descriptively.The model used task and learner-state covariates with student-clustered standard errors.Source: §3.9 · p. 11

4.2. Prediction of Near-Term Persistent Failure

• 0.550 test PR-AUC and 0.681 ROC-AUC: validation-selected logistic regression led the held-out benchmark, with MLP at 0.525 test PR-AUC and 0.670 ROC-AUC.The test partition included 555 labeled states from 42 students; 222 (40.0%) were persistent failures.Source: §4.2 · p. 11, §4.1 · p. 11, Table 3 · p. 12

4.3. Feature Groups, Calibration, and Task Heterogeneity

• Task identifiers drove the largest gradient-boosting permutation decrease (0.052 PR-AUC); logistic-model coefficients and subgroup ROC-AUCs also varied by task context.Timing and attempt features had a mean PR-AUC change of -0.052, reflecting overlap and interaction with other feature groups.Source: §4.3 · p. 12, Figure 3 · p. 13

4.4. Secondary Outcomes and Across-Task Student Histories

• 0.659 PR-AUC and 0.776 ROC-AUC: history-summary gradient boosting predicted unmodified retry, compared with 0.599 test PR-AUC and 0.735 ROC-AUC for dedicated gradient boosting.The outcome-specific student GRU with across-task history reached PR-AUC 0.631 and ROC-AUC 0.764.Source: §4.4 · pp. 12–13, Table 4 · p. 13
• Near-term related-task performance was hardest (dedicated fusion GRU test ROC-AUC 0.303), whereas history-summary logistic achieved 0.710 ROC-AUC for 7–28-day outcomes; knowledge targets need richer measurement.The 7–28-day estimates came from 67 test cases across 25 students.Source: §4.4 · p. 13

4.5. Recorded Feedback Scores and Newly Generated Feedback

• Human-rated condition scores rose from A (63.17) to D (86.13); after repair and source gating, 519/544 generated messages contained all required components.The gate removed 484 unverifiable source references, retaining only exact matches.Source: §4.5 · p. 14, Table 5 · p. 14

4.6. Intervention Capacity and Timing

• At the planned 20% alert budget, consecutive-failure alerts captured 75 failures (0.676 precision), versus 66 (0.595) for logistic regression and 62 (0.559) for student history.The student-history policy had capture recall 0.279 at this budget.Source: Figure 6 · p. 15, §4.6 · p. 15
• A validation-fixed 0.500 student-history threshold selected 17.8% of test states (precision 0.566; capture recall 0.252); validation-only calibration supported test-time threshold selection.Alerting after every failure covered all states at base precision 0.400.Source: §4.6 · pp. 15–16, Figure 7 · p. 16

4.7. Historical Course Feedback and Survey Context

• In the randomized-course historical subset, natural-language feedback was associated with lower persistent-failure odds (OR 0.234); 19/37 survey respondents rated feedback helpfulness 4–5.The adjusted association used task fixed effects; survey ratings describe all 37 respondents.Source: §4.7 · p. 16, Figure 8 · p. 16

5.1. Principal Findings

• The validation-selected logistic model achieved test PR-AUC 0.550 and ROC-AUC 0.681, while the MLP performed similarly on the primary outcome.Recurrent models added value for observable resubmission behavior; the discussion recommends matching model complexity to stable predictive value.Source: §5.1 · p. 17
• Standardized repair produced compact, mostly complete feedback, with source verification removing unsupported code references and routing unresolved outputs for review.The matched four-condition design supports within-case comparison of contextual evidence and assistance instructions.Source: §5.1 · p. 17
• 17.8% of eligible test states were selected and 25.2% of observed persistent failures were captured by the sequential policy.The discussion supports thresholds that reflect instructional capacity and the relative costs of missed and unnecessary support.Source: §5.1 · p. 17

5.2. Predictive Modeling, Measurement, and the Value of Learning Histories

• Broader histories were particularly informative for resubmission, while task identity and current performance carried substantial signal for persistent failure.Later related-task performance depends on learning opportunities and time, so the discussion argues that one learner representation may not serve all outcomes equally.Source: §5.2 · p. 18
• Student-disjoint partitions and decision-time prefixes reduce identity and temporal leakage, and validation-based selection preserves an independent final test.Calibration and capacity curves translate predicted risk into alert volume, precision, and failure capture.Source: §5.2 · p. 18

5.3. From Risk Scores to Evidence-Gated Progressive Assistance

• The four-gate strategy uses calibrated risk and capacity for timing, time-valid observations for content, risk and recent behavior for specificity, and verification before delivery.Support progresses from test inspection or output prediction to concept reminders and localized debugging hints after repeated unsuccessful revisions.Source: §5.3 · p. 18
• Learner explanations should connect an observable issue to an actionable, verifiable next step, while instructors receive supporting traces, risk, uncertainty, and repair history.Timing can preserve productive debugging through a short learner-controlled interval, with stronger support after continuing difficulty and capacity thresholds limiting intervention volume.Source: §5.3 · pp. 18–19

5.4. Contributions to Learning Analytics and Generative Feedback Research

• The framework separates evidence for prediction, prioritization, message quality, and learning, while paired messages and capacity curves support assessment and threshold choice.The discussion proposes blinded ratings before learner-facing deployment, followed by classroom measures aligned to knowledge, self-regulation, transfer, and retention.Source: §5.4 · p. 19

6. Research Scope and Future Research

• Validation is course-scale; external evaluation across cohorts, institutions, programming languages, and assessment designs remains the next step.The authors propose freezing feature definitions, calibration procedures, and decision thresholds before transfer to enable direct portability evaluation.Source: §6 · p. 19
• A prospective comparison of usual feedback, fixed evidence-based feedback, and evidence-gated progressive assistance could assess immediate learning, transfer, delayed retention, and self-regulated learning.The proposed evaluation also includes blinded ratings of feedback and monitoring of subgroup calibration, accessibility, alert exposure, and learner autonomy.Source: §6 · pp. 19–20
• Student-level or micro-randomized designs with preregistered models, thresholds, prompts, outcomes, and moderation analyses could connect the framework to direct educational measures.The proposed agenda examines knowledge, transfer, self-regulation, and retention, including which timing, evidence, and assistance combinations support learning.Source: §6 · p. 20

7. Conclusions

• The framework was evaluated on 2993 failed programming states from 215 students, with four matched feedback forms and alert policies under limited capacity.Its central principle is to keep prediction, decision, generation, and learning outcomes tied to distinct forms of evidence.Source: §7 · p. 20
• Current performance and prior behavior provided predictive value; broader-history models contributed outcome-specific information, especially for observable resubmission behavior.The feedback pipeline produced predominantly complete structures after repair, and source gating supplied a transparent quality-control mechanism.Source: §7 · p. 20
• The proposed evidence-gated assistance strategy uses calibrated risk to trigger support, time-valid evidence to constrain content, and escalating help from self-checks to localized guidance.Selected outputs receive focused review.Source: §7 · p. 20
• The study provides a technical and methodological foundation, but prospective learner deployment is needed to assess knowledge, self-regulated learning, transfer, and long-term retention.The proposed evaluation includes validated measures, blinded feedback assessment, and delayed testing.Source: §7 · p. 20