PaperlayerView exact revision on arXiv

Source-linked AI summary

Revision-Aware Success Prediction from Multi-Attempt Programming Trajectories

Md Faizul Ibne Amin, Yutaka Watanobe, Daniel M. Muepu, Kenta Nanaumi, Haruto Suzuki, Md. Shahajada Mia, Md Mostafizer Rahman

arXiv:2608.26169v1cs.CYcs.SE

TL;DR

Programming success prediction from submission trajectories is difficult because failures, revisions, and future-horizon availability vary. The study compares three prediction horizons, three input regimes, and ML, DL, and PTM models under a unified protocol. Current-only inputs and feature-based ML are the most reliable overall, while Task 2 is hardest and complex models provide task-specific value.

  • Problem

    Programming success prediction is difficult because attempts contain heterogeneous error states, short-term revisions, and uneven future-horizon availability, despite submission traces supporting educational analytics.

  • Method

    The study evaluates current-attempt, next-attempt, and three-attempt recovery prediction under current-only, pairwise, and multi-step regimes using ML, DL, and PTM models.

  • Results

    Current-only inputs and feature-based ML models are the most reliable overall; Task 2 is hardest, while DL and PTMs provide task-specific value, especially on Task 3.

  • Takeaways & Limitations

    Feature-based ML remains highly effective for programming success prediction, while near-future recovery prediction offers a supported target for submission-aware educational analytics.

  • Takeaways & Limitations

    The study uses a small AI-permitted contest-style setting, and several revision-history and sensitivity subsets are small, making generalization and isolated high metrics require caution.

Abstract

from arXiv · show

Programming outcome prediction plays a central role in data-driven programming education, supporting learner modeling, timely intervention, and adaptive assistance. Yet predicting submission success is difficult due to heterogeneous error states, short-term revisions, and uneven future-horizon availability in programming trajectories. This study examines three prediction tasks under a unified formulation: whether the current attempt is accepted (Task~1), whether the next attempt is accepted (Task~2), and whether acceptance is reached within a three-attempt recovery window (Task~3). Each task is evaluated across current-only, pairwise, and multi-step input regimes using ML, DL, and transformer-based pretrained models (PTM), represented by LinearSVM, XGBoost, BiGRU, BiLSTM, GraphCodeBERT, and CodeT5+. Results show a consistent pattern: the current-only regime is the most reliable, while pairwise and multi-step history provide no consistent gain. ML models are the strongest and most stable overall, particularly in Tasks~1 and~3, and Task~2 is the hardest across all model families. DL and PTMs perform well on Task~3 but are more task-dependent. In the Task~3 current-only setting, XGBoost achieves AP/PR-AUC of 99.09% and MCC of 0.6325, while GraphCodeBERT and CodeT5+ reach F1 scores of 80.00% and 73.68%, respectively. A sensitivity analysis confirms that Task~3 conclusions hold most robustly for ML models under stricter future-horizon control. Across all settings, ML models remain highly effective for programming success prediction, while complex models offer value in specific settings. This work provides a systematic comparison across predictive formulations and offers robust modeling guidance for submission-aware analytics in programming education, where near-future success prediction can inform timely intervention in online judge platforms and adaptive programming support systems.

I. INTRODUCTION

Programming submission traces support educational prediction, but heterogeneous failures and revisions make attempt-level success difficult to model. This study addresses that challenge through three horizons, matched input regimes, and a unified comparison of ML, DL, and PTM models.

  • Submission traces can support learner modeling, timely intervention, adaptive assistance, and responsive programming support systems.
  • Heterogeneous error states and multiple revisions make programming success prediction challenging, while revision history may signal debugging direction and proximity to success.
  • LinearSVM, XGBoost, BiGRU, BiLSTM, GraphCodeBERT, and CodeT5+ are evaluated under a unified protocol spanning ML, DL, and PTM families.
  • The benchmark uses judge-system trajectories from an AI-permitted human-AI co-creation contest, preserving verdict feedback, time pressure, and AI tool use.
  • The study compares current-attempt, next-attempt, and near-future recovery prediction under matched current-only, pairwise, and multi-step representations.

B. Knowledge Tracing and Trajectory Modeling

The paper situates attempt-level trajectory prediction within programming education, knowledge tracing, and AI-assisted coding research. It defines each trajectory as ordered user submissions for one problem and constructs labels from individual attempt states.

  • Prior work uses programming behavior, submission records, learning trajectories, knowledge tracing, and code representations for educational prediction.
  • AI-assisted programming provides a human-oversight context in which users inspect and revise AI-supported outputs, while this study analyzes judge-system submissions.
  • Most prior studies target course-level performance, pass/fail outcomes, or latent knowledge states rather than matched attempt-level prediction from short revision histories.
  • A programming trajectory is an ordered sequence of submission attempts by one user for one problem, pairing each submitted code state with its verdict.
  • Each attempt defines a trajectory state from which task labels and input representations are constructed.

1) Task 1: Current-Attempt Acceptance:

Task 1 evaluates the most direct prediction target: whether the current submission is accepted. Its input regimes test current code alone, an immediate revision transition, or a short local history.

  • Task 1 uses the current code state to predict current-attempt acceptance, making it the most direct prediction setting.
  • Input regimes: The current-only regime supplies only the current submission and tests whether that snapshot is sufficient for success prediction.
  • Input regimes: The pairwise regime supplies previous and current code states to capture the immediate revision transition.

3) Multi-Step Revision Representation:

The multi-step regime adds the previous two submissions to the current code state, testing whether short local revision history improves prediction. Dataset construction and evaluation enforce availability constraints and reserve test data for final reporting.

  • 3) Multi-Step Revision Representation:: The multi-step regime uses the previous two and current code states to test whether short local history improves prediction.
  • 3) Multi-Step Revision Representation:: Multi-step inputs require at least two prior attempts, reducing eligible samples and increasing input complexity.
  • 3) Multi-Step Revision Representation:: Task-regime datasets enforce required current, prior, and future code-verdict availability, so current-only settings retain the most eligible states.
  • B. Submission Parsing and Trajectory Reconstruction: The workflow parses submission records, reconstructs participant-problem trajectories, filters revision validity, builds datasets, and evaluates ML, DL, and PTMs.
  • B. Submission Parsing and Trajectory Reconstruction: Validation data selects checkpoints and tunes thresholds, while the test split is reserved for final performance reporting.

C. Language-Consistent Filtering and Trajectory Truncation

The study retains language-consistent participant-problem trajectories, truncates successful trajectories at first acceptance, and constructs eligible task-regime instances from retained states.

  • Language-Consistent Filtering: Trajectories are retained only when revisions remain language-consistent, preventing language changes from being mixed with revision effects.
  • Trajectory Truncation: Successful trajectories are truncated at the first accepted submission, while trajectories without acceptance remain complete.
  • Trajectory Truncation: Truncation prevents post-success attempts from influencing recovery-oriented prediction tasks.
  • Future-Horizon Construction: The retained trajectory length determines each state’s available future horizon for future-window construction and Task 3 sensitivity analysis.
  • Instance Construction: Task 2 and Task 3 use non-accepted states, whereas current-only, pairwise, and multi-step regimes progressively require more historical context.

E. Input Representation Construction

Inputs preserve temporal distinctions among code states and are transformed into family-specific representations for ML, DL, and pretrained models.

  • Temporal Input Construction: State markers M0, M1, and M2 separate code snapshots while preserving each state’s temporal role.
  • Family-Specific Features: ML models use sparse lexical features, DL models use token-index sequences, and PTMs use tokenizer-based truncated and padded inputs.
  • Machine Learning: The ML pipeline applies TF-IDF to each constructed representation using a vectorizer fitted on the training split.
  • Machine Learning: LinearSVM supplies a sparse linear baseline, while XGBoost supplies a nonlinear tree-based alternative.
  • Deep Learning: BiGRU and BiLSTM encode token sequences bidirectionally by concatenating forward and backward summaries.
  • Deep Learning: Class-weighted binary cross-entropy is used to address class imbalance in the deep-learning pipeline.

H. Transformer-Based Pretrained Code Model Pipeline

The pretrained-model pipeline evaluates GraphCodeBERT and CodeT5+ using contextual encoder representations, with state markers distinguishing historical and current code.

  • Pretrained Encoders: GraphCodeBERT and CodeT5+ are evaluated as pretrained code models producing contextual hidden states.
  • GraphCodeBERT: GraphCodeBERT uses the first-token hidden state as its sequence-level representation.
  • CodeT5+: CodeT5+ uses masked mean pooling over encoder hidden states.
  • Temporal Inputs: Pairwise and multi-step inputs include state markers so models distinguish previous code states from the current state.

I. Training, Selection, and Evaluation Protocol

The protocol separates training, validation, and testing at the trajectory level, uses validation-based model selection and thresholding, and documents restricted Task 3 sensitivity subsets.

  • Selection Protocol: DL and PTM checkpoints are selected using validation AP/PR-AUC, while ML thresholds are selected from validation scores.
  • Evaluation Protocol: The selected threshold τ* is fixed before application to the test split, which is reserved for final performance reporting.
  • Dataset: The dataset uses judge-system submissions from 15 participants attempting 13 contest problems in an AI-permitted contest setting.
  • Dataset Splits: Nine main task-regime datasets are split into training, validation, and test sets using participant-problem trajectory partitions.
  • Sensitivity Analysis: S1 remains suitable for binary evaluation, whereas S3 becomes single-class after stricter Task 3 horizon filtering.
  • Metrics: The evaluation reports complementary metrics for class-imbalanced datasets with validation-selected decision thresholds.
  • Sensitivity Analysis: Sensitivity analysis represents the model families with XGBoost, BiLSTM, and GraphCodeBERT selected from the main Task 3 results.

E. Device Configuration

Across the three tasks, matched evaluation shows current-only inputs and feature-based ML models are generally the most reliable, while revision-aware regimes and complex models provide selective rather than consistent benefits.

  • Task 1: XGBoost is the strongest and most stable model for Task 1, reaching AP/PR-AUC 0.8775, MCC 0.5484, and F1 0.7907 in the current-only regime.Pairwise XGBoost also reaches MCC 0.7746, F1 0.8571, and BA 0.9, but this is a regime-specific result.
  • Cross-task findings: Current-only inputs are generally the most stable, whereas pairwise and multi-step histories provide no consistent advantage across model families.Task 1 recurrent models specifically show no reliable benefit from revision history, and Task 2 multi-step results are highly subset-sensitive.
  • DL and PTM behavior: Pretrained models provide useful but task- and threshold-dependent performance, including GraphCodeBERT’s Task 1 F1 0.6667 and mixed Task 2 regime behavior.GraphCodeBERT can collapse to all-negative predictions, while CodeT5+ can fail to identify positive cases under particular regimes.
  • Task 2: Task 2 recurrent models are most reliable with current-only inputs, while pairwise and multi-step settings show weak or unstable thresholded performance.BiGRU reaches AP/PR-AUC 0.6002, and BiLSTM reaches MCC 0.2582 and F1 0.5714 in current-only evaluation.
  • Task 2: Task 2 is the most difficult setting, with pairwise XGBoost providing the strongest practical result at MCC 0.6325, F1 0.75, and BA 0.8333.Perfect multi-step ML scores arise from only five test instances and one positive case, so they should not be generalized.

C. Task 3: Near-Future Success Prediction

Task 3 predicts acceptance within the next three available attempts from an unresolved state. It provides the clearest evidence for short-term recovery prediction, with current-only inputs most stable across model families and longer histories remaining subset-sensitive.

  • Task definition: Task 3 predicts whether acceptance occurs within the next three available attempts from an unresolved state.The task uses ranking, thresholded-decision, and class-specific or calibration metrics.
  • ML results: XGBoost achieves AP/PR-AUC 0.9909, ROC-AUC 0.9875, MCC 0.6325, and BA 0.8 in the current-only regime.These results make current-only XGBoost the strongest stable reference across ranking, thresholded, and probabilistic metrics on the largest Task 3 subset.
  • ML results: Pairwise LinearSVM reaches MCC 0.8018, F1 0.8571, and BA 0.8750, but its smaller subset makes this a model-specific result rather than evidence for revision-history superiority.Multi-step LinearSVM also reports perfect scores from only six test instances and one positive case.
  • DL results: Current-only DL performance is most reliable, with BiGRU and BiLSTM both reaching MCC 0.4842, F1 0.7059, and BA 0.7375.BiLSTM additionally reaches AP/PR-AUC 0.9263 and ROC-AUC 0.8875, with better calibration than BiGRU.
  • PTM results: GraphCodeBERT reaches F1 0.8 and MCC 0.55, while CodeT5+ reaches F1 0.7368 and MCC 0.4472 in current-only evaluation.The PTMs perform better on Task 3 than on earlier tasks, but pairwise and multi-step results remain less stable or subset-sensitive.
  • Interpretation: Task 3 offers the strongest evidence that trajectories carry useful signal for short-term recovery, without supporting a broad claim that longer history consistently improves prediction.Feature-based ML models and selected PTMs are especially useful for near-future success prediction.

D. Task 3 Sensitivity Analysis

Task 3 conclusions are most robust for ML under stricter future-horizon control. Across the broader comparison, current-only inputs remain most stable, while longer histories and non-ML families show more sensitivity to data availability and task conditions.

  • Task 3 sensitivity analysis: S3 becomes single-class after filtering, preventing meaningful binary evaluation and eliminating additional model experiments.All instances are negative across training, validation, and test, so thresholded metrics are non-informative.
  • Task difficulty: Task 2 is the most difficult setting, whereas Task 1 is clearly predictable and Task 3 shows the strongest current-only ranking and decision quality.Figure 3 compares the strongest current-only AP/PR-AUC and MCC for each task.
  • Input-regime behavior: Pairwise and multi-step inputs produce selected gains but no consistent advantage over current-only representations, with smaller subsets making some results fragile.Multi-step results for Tasks 2 and 3 are particularly sensitive to very small test sets.
  • Model-family behavior: ML provides the strongest and most stable overall behavior, while DL and PTMs offer narrower, task-dependent value, particularly for near-future prediction.In Task 3 current-only, PTMs achieve the highest F1, mainly through GraphCodeBERT’s positive-case detection.
  • Cross-task findings: Task 3 current-only reaches AP/PR-AUC of 0.9909 and MCC of 0.6325, compared with Task 1’s MCC of 0.5484.Near-future prediction is described as informative because unresolved code states can signal proximity to acceptance.
  • Task 3 sensitivity analysis: XGBoost remains strongly discriminative under S1, reaching MCC of 0.8165, while BiLSTM reaches MCC of 0 and GraphCodeBERT reaches MCC of 0.25.S1 uses 76 training, 13 validation, and 10 test instances; GraphCodeBERT retains signal but with weaker decision quality.

B. Implications for Programming Learning Analytics

The study connects attempt-level prediction to practical monitoring signals in programming learning analytics. Task 1 supports real-time assessment, Task 3 supports recovery alerts, and Task 2 warrants more cautious use because subsequent revision behavior matters.

  • Intervention points: Task 1 provides a real-time code-assessment signal at submission, while Task 3 can flag learners unlikely to reach acceptance within three attempts.These signals align prediction with distinct intervention points in online judges and adaptive programming systems.
  • Intervention points: Task 2 should be used more cautiously because next-attempt success depends on subsequent revision strategy and possible external assistance.The next-attempt outcome is less directly tied to the current code state.
  • Broader analytics implications: Revision-aware prediction is most useful for estimating whether an ongoing revision process is nearing success, rather than evaluating one code state in isolation.The implication extends beyond programming education to software-engineering analytics involving repeated patches or debugging cycles.
  • Scope and limitations: The study’s conclusions require caution because it uses a small AI-permitted contest-style setting, outcome correctness does not measure broader learning quality, and some subsets are small.The authors emphasize consistent cross-task patterns over isolated high scores from small subsets.
  • Future directions: Future work can examine larger course-scale and online-judge datasets, richer revision features, human-AI traces, and interpretable feedback generation.The proposed extensions combine prediction with explanation or feedback rather than treating predictive signals as deployment claims.
Loading 2608.26169v1…