PaperlayerView exact revision on arXiv

Source-linked AI summary

A Dataset for Modeling Iterative Problem-Solving

Fagun Patel, Sang T. Truong, Duc Q. Nguyen, Kazunori Fukuhara, Benjamin W. Domingue, Sanmi Koyejo, Nick Haber

arXiv:2609.00940v1cs.CL

TL;DR

Iterative problem-solving research lacks large-scale evidence and shared comparisons of models predicting how solvers revise solutions over time. The paper introduces CodeInsight and benchmarks parametric, sequential, and generative approaches on these trajectories. RSSM is strongest on three of four courses, while the LLM generates analyzable submissions but is not a faithful simulator of solver behavior.

  • Problem

    Existing work lacks large-scale evidence and a shared framework for comparing models of iterative solver trajectories, including persistent errors and strategy shifts.

  • Method

    The paper curates CodeInsight and evaluates parametric, sequential, RSSM, and LLM-based predictors under a shared calibration-and-scoring protocol.

  • Results

    RSSM achieves the strongest next-attempt prediction accuracy on three of four courses, while the LLM generates full submissions but primarily acts as a context-conditioned generative solver.

  • Takeaways & Limitations

    Modeling iterative refinement beyond binary correctness supports analysis of systematic failure modes and informs human-learning analytics and autonomous-agent evaluation.

  • Takeaways & Limitations

    The findings are limited to C++ courses at one Vietnamese university and one grading platform, so generalization remains open.

Abstract

from arXiv · show

Solving problems through repeated attempts is a sequential modeling task: at each step, the solver receives feedback and decides how to revise their solutions. Predicting whether performance improves, plateaus, or regresses across attempts is central to understanding any iterative problem-solving process in both human learners and autonomous agents. Beyond outcomes, modeling what errors persist and how strategies shift across attempts provides deeper insight into the mechanics of sequential learning. Studying these dynamics requires observing many solvers as they attempt, receive feedback, and revise. Programming courses with automated grading provide this setting, as students iteratively submit code to test suites and receive feedback on every attempt. We therefore curate CodeInsight, a large-scale dataset of over 3 million submissions from 3,286 undergraduates across 2 introductory C++ courses in 2 academic years, with test-case-level outcomes, timestamps, and source code. On this dataset, we build a benchmark that evaluates models spanning parametric, sequential, and generative traditions under a shared calibration-and-scoring protocol, including a Recurrent State Space Model (RSSM) adapted to track solver characteristics through discrete latent variables and an LLM-based predictor that generates explicit solutions. The adapted RSSM achieves the strongest predictive accuracy on three of the four courses. The LLM predictor is less accurate but produces full submissions at each attempt, enabling direct analysis of failure modes. We find that the model's coding proficiency is inversely related to predictive performance in this setting, with the LLM better understood as a generative solver conditioned on context rather than a faithful predictor of solver behavior. We publicly release our code and the dataset on request to facilitate future research.

1 Introduction

Iterative problem-solving produces sequential trajectories in which solvers use feedback to revise solutions, improve, plateau, or regress. CodeInsight enables shared evaluation of models that predict these dynamics and characterize persistent errors and strategy shifts.

  • Dataset and motivation: Over 3 million submissions from 3,286 C++ undergraduates provide test-case outcomes, timestamps, and source code for studying iterative problem-solving.The dataset covers 394 problems at a Vietnamese university across 2022–2024.
  • Results: The adapted RSSM achieves the strongest predictive accuracy on three of four courses by modeling solver characteristics with discrete latent variables.Its predictions are trained directly on response patterns.
  • Results: The LLM predictor falls below trained models while generating complete submissions that enable direct analysis of failure modes.Its generated solutions support examination of errors and strategies beyond outcome prediction.
  • Benchmark: The benchmark compares parametric, sequential, and generative models under a shared calibration-and-scoring protocol.It is designed to address the lack of comparisons across these modeling traditions under one evaluation framework.
  • Results: Predictive signal is driven more by problem difficulty than solver-specific behavior.This finding qualifies how the benchmark’s predictive models capture solver trajectories.

2 Related Works

Prior work models iterative learning, programming trajectories, and LLM-simulated solvers through varied approaches. CodeInsight addresses the absence of a shared comparison spanning code-aware sequential, parametric, and generative models.

  • Programming trajectory datasets: Programming datasets differ in granularity, coverage, and labels, with some lacking per-test-case outcomes, assignment structure, or correctness labels.The related datasets include CodeWorkout and Blackbox, among others.
  • Iterative problem-solving: Research on iterative problem-solving includes forgetting curves, repetitive practice, feedback-driven strategy adjustment, and temporal modeling with recurrent networks.These traditions study how performance changes across repetitions and feedback.
  • Research gap: No existing study compares code-aware sequential models, parametric models, and generative approaches under a shared evaluation framework.The paper positions this comparison as its central related-work gap.
  • LLMs as simulated problem-solvers: LLMs have been used to simulate learner behavior, constrain realistic errors, train teaching assistants, and generate student code from submission histories.These systems represent several ways of conditioning generation on learner behavior.

3 Dynamic Measurement Models

The paper evaluates next-step prediction from submission histories using a shared temporal protocol across baseline and adapted models. Its RSSM tracks latent solver characteristics, while the LLM generates context-conditioned next submissions without fine-tuning.

  • Evaluation setup: Future performance is predicted from prior submission history using parametric and sequential baselines alongside an RSSM and an LLM-based predictor.All approaches are evaluated under the same train/test and temporal setup.
  • Baseline models: Baseline models span IRT and CIRT parametric models, BKT and DKT sequential models, code-aware DKT, and TIKTOC.TIKTOC jointly predicts per-test-case outcomes and generates the next submission.
  • Recurrent State Space Model (RSSM): The RSSM embeds submitted code and problem descriptions, updates a recurrent history state, and infers discrete latent variables from history and current responses.A posterior uses the current response, while a prior predicts latent variables from history alone.
  • LLM-as-Predictor: The LLM predictor conditions on behavioral profiles, training-set problem statistics, and similar calibration examples to generate code for future attempts.Inputs include submission pacing, precheck usage, topic-level pass rates, and code complexity.
  • LLM-as-Predictor: At attempt t, the LLM receives the trajectory through attempt t−1 and generates the next submission without fine-tuning.This deliberately makes it a zero-training reference for prediction from context alone.

4 A Dataset of Problem-Solving Trajectories

CodeInsight records fine-grained, longitudinal C++ problem-solving behavior from undergraduate courses, including source code, timestamps, and test-case outcomes. The trajectories show rapid iterative work, early gains with diminishing returns, and strong variation driven by problem difficulty.

  • Dataset scope: CodeInsight contains 3,286 undergraduates from two C++ courses offered across the 2022 and 2023 academic years.The courses are Programming Fundamentals and Data Structures and Algorithms at VNU-HCM University of Technology.
  • Data granularity: Each problem is evaluated through multiple hidden test cases, while each submission represents one iterative attempt.The dataset also includes anonymized identities, source code, timestamps, admission-track labels, and coding-interface actions.
  • Data collection: The platform records problem statements, code editing, compiler errors, failed test cases, and feedback for incorrect and correct submissions.These interface elements expose the feedback loop underlying iterative refinement.
  • Temporal patterns: Over 3 million submissions and a 59-second median interval between consecutive submissions capture rapid iterative development cycles.Students primarily work from 8 PM to 11 PM, with activity peaking on Saturdays.
  • Trajectory patterns: Early attempts yield the largest score improvements, followed by diminishing returns across both courses.Figure 3 compares learning curves across the 2022 and 2023 cohorts.
  • Trajectory patterns: Chronological final scores show no consistent upward trend because student ability and problem difficulty vary independently.The curves are descriptive evidence of within-problem improvement rather than causal estimates of learning.
  • Problem-level distributions: Most problems require fewer than 10 attempts, but a long tail extends beyond 20, while final scores are concentrated near zero with a higher-score tail.These distributions reflect substantial variation in problem difficulty and student outcomes.

5 Results

Across the DSA courses, RSSM provides the strongest later-attempt prediction, while LLM performance is driven mainly by problem difficulty rather than student-specific behavior. Repeated attempts show expanding correctness, and LLM prompt context improves its predictions while enabling code-level failure analysis.

  • Experimental Setup: The correctness heatmap shows blue passed-test regions expanding from attempt 1 to attempt 10 as students master additional test cases through repeated practice.Rows represent students sorted by overall pass rate, while columns represent chronologically ordered test cases.
  • Model Comparison: RSSM achieves the strongest balanced accuracy and AUC on three of four courses, with its advantage over parametric baselines concentrated at later attempts.On DSA HK231, RSSM reaches 0.71–0.73 at later attempts versus 0.69–0.71 for IRT and CIRT; on DSA HK221, it reaches 0.66–0.69 versus 0.62–0.68.
  • Model Comparison: Balanced accuracy for parametric models and RSSM rises early, peaks around attempts two to five, and declines thereafter, with RSSM declining least.The later-attempt advantage is strongest because parametric models decline more steeply.
  • Model Comparison: Qwen3-14B has the highest balanced accuracy among the evaluated LLMs, and the full prompt outperforms ablated variants.Removing the persona causes the largest drop, followed closely by removing trajectory context; retrieved examples also contribute to the full pipeline.
  • Model Comparison: LLMs produce analyzable code but their predictive signal is driven more by problem-level difficulty than by student-specific modeling.After subtracting each problem’s mean score, per-student Kendall τ distributions move toward zero across all three LLMs.

6 Conclusion

The paper formalizes iterative problem-solving with CodeInsight and benchmarks diverse next-attempt predictors under a shared framework. RSSM performs best on three of four courses, whereas LLMs generate analyzable submissions without faithfully simulating human behavioral flaws.

  • Conclusion: CodeInsight is introduced as a large-scale dataset for studying how solvers refine code over time.
  • Conclusion: RSSM with discrete latent variables achieves the strongest next-attempt prediction accuracy on three of four courses, especially at later attempts.
  • Conclusion: LLMs generate analyzable code submissions but function primarily as context-conditioned generative solvers rather than faithful simulators of human behavioral flaws.
  • Conclusion: Modeling persistent errors and shifting strategies could support personalized feedback and adaptive instruction, while motivating fault-guided training for more realistic student-specific LLM errors.

Limitations

The findings are bounded by a single university, C++ courses, one grading platform, frozen LLMs, and prediction-only evaluation. Generalization, behavioral fidelity, intervention effects, and diagnostic usefulness therefore remain unresolved.

  • The dataset covers one Vietnamese university and only C++ courses, limiting whether the observed patterns generalize to other settings.The grading platform’s feedback design also shapes the trajectories observed.
  • The LLM evaluation used only three models with frozen parameters and no fine-tuning, so other models or training conditions may behave differently.The authors suggest error-rich student code as a possible route to improved behavioral fidelity but did not test it here.
  • The evaluation measures prediction accuracy but does not test whether forecasts improve outcomes when surfaced to instructors or solvers.Such effects would require controlled deployment in a live course.
  • Score prediction does not diagnose the underlying misconception or identify the help that would resolve repeated failure.
  • Future work should test additional models, languages, retrieval or decoding strategies, and interpretable RSSM latent states.These extensions would help distinguish setting-specific findings from broader properties of generative sequential prediction.

Ethical Considerations

The study uses post-course permission and anonymization to curate student submissions, while restricting dataset release to educational use. The course setting includes automated feedback and saved iterative submissions.

  • The researchers obtained Department permission because the data were curated and used after the courses ended.
  • Student identities were anonymized with numeric indices, and names and unique personal identifiers were removed.
  • Submissions were screened for embedded emails and student identification numbers before distribution.
  • The dataset remains university-department intellectual property and is released on request under a license restricted to educational purposes.
  • Students submit code repeatedly, receive test-case feedback and scores, and the system saves each submission for later analysis.Prechecks expose public test-case outcomes, whereas private submissions return only a numerical evaluation score.

A.1 Student Behavior Patterns

Student behavior reflects rapid, repeated code submission and varied revision patterns. More submissions are associated with higher final scores, but submission volume alone does not reliably predict success.

  • More submissions per problem are associated with higher final scores, while the median student makes 455 submissions overall.The mean is 610 submissions across all problems.
  • Pause duration has a nuanced relationship with performance because successful students use a range of pacing strategies.
  • 59 seconds is the median pause between consecutive submissions, indicating frequent short-cycle testing and revision.The mean pause is 2.7 minutes, and the authors interpret the pattern as trial-and-error guided by automated feedback.
  • High-volume submitters have varied outcomes: some reach perfect or near-perfect scores, while others continue struggling.Revision quality and the ability to interpret and act on feedback matter alongside submission count.
  • Most students cluster around moderate edit distances and submission counts, with outliers showing very large edits or long inter-submission intervals.The comparison concerns average time between submissions, average Levenshtein edit distance, and total submissions.

A.2 Course Characteristics and Trajectory Quality

Course trajectories differ sharply: PF is dominated by always-fail and flat patterns, whereas DSA shows improvement and stronger sequential signal. DSA HK221 provides the clearest recurrent-model context.

  • PF courses contain 58–63% always-fail multi-attempt trajectories, while 69–72% show no meaningful change between trajectory halves.
  • Removing survivorship correction makes PF pass rates decline by 2–3 percentage points from attempt 1 to 10.Students persisting to later attempts are predominantly stuck rather than improving.
  • DSA courses show genuine improvement, but DSA HK221 has the strongest sequential signal and the only positive trend across all four courses.Its pass rate rises 2.5 percentage points from attempt 1 to 10.
  • DSA HK221 starts at a 27.8% first-attempt pass rate versus 39.1% for DSA HK231, leaving more room for improvement.
  • DSA HK221 has longer sequences and more problems with 5+ attempts, giving recurrent models more sequential context.Its mean attempts per problem are 3.79 versus 3.18, and 32% versus 25% of problems have 5+ attempts.
  • The RSSM leads parametric models on both DSA courses, with its advantage concentrated at later attempts.The trajectory-quality gradient motivates focusing the main results on DSA courses.

B Examples of Student Learning Trajectories

The examples show iterative coding trajectories in which students revise solutions after feedback, with outcomes ranging from incomplete and partially correct to correct. Across trajectories, behavior includes rapid iteration, incremental edits, strategy changes, repeated submissions, and varied debugging approaches.

  • Students iteratively debug and refine code in response to automated test feedback, producing trajectories that expose changing solutions and outcomes.The dataset examples include incomplete, partially correct, incorrect, and correct attempts.
  • One trajectory improves from an incomplete first attempt to a correct fourth attempt through successive code revisions.The intermediate submissions receive partially correct marks of 0.40 before the final submission receives 1.00.
  • Across the courses, trajectory patterns range from always-fail and flat behavior to more diverse outcomes and gradual improvement.The course-level breakdown distinguishes always-fail, mixed, and always-pass trajectories alongside improving, flat, and declining patterns.
  • Another trajectory moves from a rapid, likely pre-prepared submission to a correct result after several attempts.The student submits a complete solution 9 seconds after starting, and the trajectory later records a correct outcome.

C.8 Evaluation Data Filtering

The evaluation filters problem instances by final-submission pass rate and student coverage, then applies a shared capped-attempt protocol and common metrics across models. Sensitivity checks indicate that the model ordering remains stable across alternative filtering and attempt-cap configurations.

  • Evaluation Data Filtering: Problems are retained when final-submission pass rates lie in [10%, 90%] and more than 25% of students attempted them.These criteria remove problems that nearly everyone passes or fails and problems with insufficient student coverage.
  • Evaluation Data Filtering: Changing filtering thresholds or the attempt cap leaves the RSSM ranked first in every tested configuration on DSA HK231.The sensitivity analysis varies pass-rate, coverage, and attempt-cap settings, retraining each model for every evaluation universe.
  • Evaluation Data Filtering: Models predict next-attempt pass probabilities independently for each test case under a shared student split and evaluation protocol.The reported metrics include balanced accuracy, AUC, log loss, and Brier score; the LLM reports only balanced accuracy because it outputs binary outcomes.
  • Evaluation Data Filtering: The trajectory examples illustrate the behavioral variation that the benchmark evaluates, including syntax errors, logical errors, repeated code, and successful strategy changes.Examples include both successful recursive pivots and submissions that retain flawed logic across attempts.

D.7 Prompt Ablation

The prompt ablation tests the student persona, attempt trajectory, and retrieved examples separately. The full pipeline performs best, while persona and trajectory removal hurt most; problem-level correlations indicate that the LLM remains driven largely by its own coding ability.

  • D.7 Prompt Ablation: The ablation compares a full prompt with variants removing the persona, trajectory context, or retrieved examples, plus a recency-based retrieval condition.The full pipeline combines student persona, prior attempt trajectory, and RAG examples from five similar problems.
  • D.7 Prompt Ablation: The full pipeline achieves the highest balanced accuracy among the tested Qwen3-14B conditions.The comparison is reported in Figure 6’s bottom panel.
  • D.7 Prompt Ablation: Removing the persona causes the largest drop, with trajectory removal producing a similar but smaller decline.These results identify the student profile and attempt history as the strongest conditioning components.
  • D.7 Prompt Ablation: Removing RAG examples causes a smaller decrease, while recency-based retrieval performs comparably to removing retrieval entirely.The result favors content similarity over simple recency as the more informative retrieval signal, within this ablation.
  • D.7 Prompt Ablation: The persona can calibrate predicted failure frequency through aggregate pass-rate statistics without requiring student-specific dynamics.This interpretation connects the ablation result to the difficulty-driven pattern observed in Figure 20.
  • D.7 Prompt Ablation: The LLM’s problem-level pass rates correlate more strongly with its own direct-solve outputs (r = 0.80) than with real student pass rates (r = 0.40).The comparison suggests that problem performance is driven primarily by the LLM’s coding ability rather than the student profile.
  • D.7 Prompt Ablation: The paper reports supervised language-refinement and code-debugging assistance from GPT-family and Claude 3.5 tools, under author supervision.The authors retain responsibility for the conceptual contributions, analyses, and interpretations.
Loading 2609.00940v1…