Offline Apprenticeship Learning for Human-Centric Environments

DEDICATION

• The author dedicates the dissertation to his parents, brother, and wife for supporting and encouraging him during his Ph.D. journey.The dedication names Mohammad Rofiqul Islam, Rehena Parveen, Md Raisul Islam, and Tasfia Tafannum.Source: DEDICATION · p. 5

BIOGRAPHY

• After earning a BUET computer science degree and pursuing a Ph.D. at NC State, Mirajul developed offline learning frameworks for human-centered education and healthcare.His work addressed learning from mixed-quality demonstrations where online trial-and-error is infeasible or unsafe.Source: BIOGRAPHY · p. 6
• During internships at MarginEdge, Under Armour, and Amazon, Mirajul worked on invoice data processing, athlete performance modeling, and hierarchical multi-agent content generation.His Amazon internship led to a return offer.Source: BIOGRAPHY · p. 6
• After defending his Ph.D. in May 2026, Mirajul began working as an Applied Scientist at Amazon Web Services.Source: BIOGRAPHY · p. 6

Across chapters

• Dr. Min Chi provided conceptualization, research guidance, supervision, and manuscript revision across every chapter.Source: AUTHORSHIP STATEMENT · p. 14

Chapters 1, 2, and 8

• Islam was sole author of Chapters 1, 2, and 8.Source: AUTHORSHIP STATEMENT · p. 14

Chapter 3: Modeling Heterogeneous Student Pedagogical Strategies

• For Chapter 3, Islam led research, experiments, and writing; Yang handled methodology, while Hostetter and Saha collected data.Source: AUTHORSHIP STATEMENT · p. 14

Chapter 4: Modeling Evolving Reward Functions

• In Chapter 4’s AIED 2025 study, Islam led research, experiments, and writing; in its KDD 2025 study, Yang led research, methodology, and writing.Debnath and Saha collected data for the AIED 2025 study; Islam conducted experiments for the KDD 2025 study, and Gao helped with writing.Source: AUTHORSHIP STATEMENT · p. 14

Chapters 5 and 6

• For Chapters 5 and 6, Islam led research, methodology, experiments, and writing; Debnath and Saha collected data for Chapter 5, and Guo ran additional Chapter 6 experiments.Guo used code and an experimental setup provided by Islam.Source: AUTHORSHIP STATEMENT · p. 14

Chapter 7 and AI disclosure

• Islam led research, methodology, experiments, and writing for Chapter 7; generative AI was used only for formatting and accessibility remediation, not research content, findings, or substantive writing.Source: AUTHORSHIP STATEMENT · p. 14

1.1 Overview

• Human-facing RL adoption is limited by costly, ethically constrained interaction data and the difficulty of designing accurate rewards for complex, latent, context-dependent goals.In sensitive settings, live policy experimentation raises ethical concerns and can make exploratory data collection infeasible.Source: §1.1 · pp. 15–16

1.1.1 Apprenticeship Learning as an Alternative

• Apprenticeship Learning infers reward functions from expert demonstrations and can induce policies without explicit reward specification, but classical online methods require interaction; strictly offline AL learns only from pre-collected logs.The dissertation develops frameworks to relax assumptions routinely violated by real-world human demonstration data.Source: §1.1.1 · p. 16

1.1.2 Reward Modeling: Heterogeneous and Evolving Reward Functions

• EM-EDM clusters trajectories by latent reward and induces cluster-specific policies, while THEMES segments trajectories into temporally coherent sub-trajectories; both outperform relevant AL baselines in their reported settings.EM-EDM outperforms four AL and two DRL baselines with as few as 24 expert demonstrations; THEMES outperforms flat and hierarchical AL baselines across one educational and three clinical datasets.Source: §1.1.2 · pp. 16–17

1.1.3 Demonstration Quality: Imperfect and Negative Demonstrations

• HALIDE integrates continuous quality signals into hierarchical policy induction, enabling ranked reuse of mixed-quality trajectories and consistent gains in cross-semester generalization.Its account distinguishes ranking demonstrations by influence from using reward functions to define optimal behavior.Source: §1.1.3 · p. 17, §1.3 · p. 20
• CCHAL assigns trajectory quality to individual decisions, scores decision criticality by contrasting expert and non-expert behavior, and uses failure trajectories in a Bradley-Terry contrastive IRL objective.Across three clinical EHR datasets and an ITS dataset, it outperforms flat and hierarchical AL baselines, including HALIDE.Source: §1.1.3 · p. 18, §1.3 · p. 20

1.1.4 Real-World Classroom Validation

• StuEDM significantly improves posttest performance and normalized learning gain over the Expert baseline, and pretest-adjusted posttest score over StuCQL, while overriding students less than half as often.In a controlled classroom study with 104 students, StuEDM was trained on a substantially smaller, agency-driven demonstration corpus than the CQL policy.Source: §1.1.4 · p. 18, §1.3 · pp. 20–21

1.2 Research Questions

• The five research questions ask how offline AL can model heterogeneous and evolving rewards, learn from imperfect demonstrations at trajectory and decision levels, and improve outcomes in a live classroom.Source: §1.2 · pp. 18–19

1.3 Contributions

• The dissertation’s five contributions progress from heterogeneous and evolving reward modeling to quality-aware and contrastive learning, culminating in real-world classroom validation.Together, they establish applications across education and healthcare.Source: §1.3 · pp. 20–21

2.1 Sequential Decision-Making under Uncertainty

• Human-centric RL faces sample inefficiency, hard-to-specify and non-stationary rewards, and ethical barriers to live policy experimentation, motivating apprenticeship learning.The passage notes that DQN may require millions of interactions while real-world settings may provide only a few thousand trajectories.Source: §2.1 · p. 23

2.2 Apprenticeship Learning (AL)

• Offline AL infers latent rewards from expert demonstrations to learn policies without hand-crafted rewards or environment interaction, but must address distribution shift, policy validation, and reward identifiability.Batch IRL removes rollouts but can rely on restrictive reward parameterizations or off-policy evaluation, leading to unstable or suboptimal solutions.Source: §2.2 · pp. 23–24

2.3 AL with Heterogeneous Reward Functions

• This dissertation adapts an EM framework to energy-based distribution matching, enabling heterogeneous reward modeling at scale in continuous spaces.Prior EM-IRL approaches clustered demonstrations and learned cluster-specific rewards but were limited to discrete states and feature sets.Source: §2.3 · p. 24

2.4 AL with Evolving Reward Functions

• Because rewards can evolve within a trajectory, THEMES automatically partitions demonstrations into temporally coherent sub-trajectories, induces cluster-specific policies, and refines segmentation with a hierarchical reward regulator offline.The framework is used to model evolving pedagogical and clinical decision-making strategies across educational and healthcare datasets.Source: §2.4 · pp. 24–25

2.5 AL from Imperfect and Mixed-Quality Demonstrations

• HALIDE addresses scarce expert data and mixed-quality demonstrations by integrating continuous quality signals from Normalized Learning Gain into hierarchical policy induction while preserving temporal and strategic structure.This targets limitations of coarse trajectory-level weighting, which may propagate suboptimal decisions and miss within-trajectory quality variation.Source: §2.5 · pp. 25–26

2.6 Contrastive AL from Negative Demonstrations

• CHALIDE localizes failure-inducing divergence in negative trajectories, applying temporally weighted anti-imitation at detected points and cluster-level reward ordering to retain informative correct segments.This avoids uniform anti-imitation, which can discard strategy-consistent behavior and introduce noisy gradients.Source: §2.6 · p. 26

2.7 RL and AL in Intelligent Tutoring Systems

• StuEDM, the first live classroom deployment of an EDM-induced student-choice-with-override policy, outperformed CQL trained on a large historical corpus across multiple normalized learning gain measures without added study time.The policy intervenes only when its recommendation disagrees with the student’s autonomous choice.Source: §2.7 · p. 27

3.1 Overview

• EM-EDM handles over 140 continuous features, identifies distinct student reward functions with as few as 24 demonstrations, and is also evaluated on three clinical EHR datasets.The clinical validation concerns septic shock prevention; the supplied passages do not report its results.Source: §3.1 · pp. 30–31

Modeling Heterogeneous Student Pedagogical Strategies

• EM-EDM outperforms four AL baselines across all performance metrics and two DRL baselines on both tasks.The framework models complex student pedagogical decisions in a large, continuous state space and adapts to heterogeneous reward functions with few demonstrations.Source: Modeling Heterogeneous Student… · p. 28, §3.1 · p. 31

3.2.1 RL and DRL for Pedagogical Policy Induction

• Pedagogical RL and DRL face sample inefficiency and difficult reward design, with DQN requiring hundreds of millions of interactions to learn and generalize.Hand-designing suitable rewards for human-centric tasks is time-consuming and labor-intensive.Source: §3.2.1 · p. 32

3.2.2 Apprenticeship Learning (AL)

• Offline AL includes behavior cloning, IRL, adversarial imitation, and EDM; EDM is a baseline in this work (Jarrett et al. 2020a).EDM had been shown to outperform IRL-based and adversarial imitation methods on a range of benchmarks.Source: §3.2.2 · pp. 32–33

3.2.3 AL with Multiple Intentions

• Prior multiple-intention AL methods cluster demonstrations by reward, but EM-IRL for student behaviors is limited to discrete states and cannot generalize to large continuous spaces.The cited EM-IRL work used 17 discrete features.Source: §3.2.3 · p. 33

3.2.4 WE, PS, and FWE

• Adaptively alternating among WE, PS, and FWE has improved learning over hand-coded expert rules, but prior work had not combined student choices with RL-induced decisions.WE, PS, and FWE denote Worked Example, Problem Solving, and Faded Worked Example.Source: §3.2.4 · p. 34

3.3 Methodology

• EM-EDM clusters trajectories assumed to arise from multiple policies, then induces a distinct policy for each learned cluster.Its expectation-maximization procedure updates cluster assignments and learns cluster-specific policy parameters via EDM.Source: §3.3 · p. 34

3.3.1 EDM

• EDM is strictly offline and learns from demonstrations without transition-model knowledge, but assumes a single policy driven by one reward function (Jarrett et al. 2020a).Its objective uses occupancy-measure matching; the supplied passage begins describing the challenge of estimating the learned policy’s occupancy without online rollout but is incomplete.Source: §3.3.1 · p. 35
• EDM uses an energy-based model to reformulate the occupancy term and optimize a surrogate objective without online policy rollouts.The occupancy gradient can be solved with existing optimizers, including stochastic gradient Langevin dynamics.Source: §3.3.1 · p. 35

3.3.2 EM-EDM

• EM-EDM clusters trajectories by latent policy and iteratively learns cluster-specific policies with EDM, extending EM-based IRL to continuous ITS state spaces (Yang et al. 2023b).Its E-step estimates trajectory membership, while its M-step updates cluster priors and policy parameters until convergence.Source: §3.3.2 · pp. 36–37

3.3.4 Two Deep RL Baselines : CQL & CQL-T; CQL-T

• CQL learns conservative Q-functions offline, while CQL-T uses the same policy-induction method with a combined reward for student training time and Normalized Learning Gain.CQL-T is motivated by prior evidence that response time reveals proficiency and is negatively correlated with final exam scores.Source: §3.3.4 · p. 39, CQL-T · p. 40

3.4 Experiments; 3.4.1 Data Collection

• The ITS study involved 128 students across two semesters, with textbook study, pretest, training, and posttest stages; training offered solo solving, tutor collaboration, or worked examples.Students practiced twelve problems, and the posttest contained twelve questions.Source: §3.4.1 · pp. 40–41

Continuous State Space; Pedagogical Action Space

• The ITS state representation comprised 142 continuous features spanning autonomy, temporal behavior, problem-solving context, performance, and hint use; actions were elicit, tell, or collaborate.These actions correspond to solving independently, receiving a direct solution, or working collaboratively with the tutor.Source: Continuous State Space · pp. 41–42, Pedagogical Action Space · p. 42

3.4.3 Selecting Demonstrations (A total of 53)

• Quantized Learning Gain classified students by pretest-to-posttest performance-group transitions, yielding 53 high-quality trajectories treated as near-optimal demonstrations.Students were grouped into low, medium, or high performance by percentile; improvement to a higher group or remaining high counted as high QLG.Source: §3.4.3 · pp. 42–43

3.4.4 Trajectories for DRL policy Induction

• The CQL policies were induced offline from tutor decision logs, with one policy trained on 2,716 students’ interactions across 13 semesters and using Normalized Learning Gain as its reward.The logs cover Fall 2015 to Spring 2022.Source: §3.4.4 · p. 43

3.7 Summary

• EM-EDM assumes reward functions may vary across trajectories but remain constant within each trajectory, enabling distinct policies from as few as 24 demonstrations (Yang et al. 2023b).Its clustering groups students following similar strategies for policy induction and statistical analysis.Source: §3.4.5 · p. 45, §3.7 · p. 50

Distribution of the Discovered Clusters

• Four clusters contained 25, 13, 11, and 4 students, but their learning-performance measures showed no significant differences.The clusters were discovered from the 53 good trajectories in Task 1.Source: Distribution of the Discovered Clusters · p. 47

Clustering Prediction Tasks

• On the stricter future-semester task, EM-EDM outperformed all baselines across metrics, indicating that strategies learned from Spring 21 predicted Spring 22 students’ actions.Spring 21’s 24 good trajectories were used for training and Spring 22’s 29 for testing.Source: Table 3.3 · p. 45, §3.5 · p. 46, Clustering Prediction Tasks · p. 48

Distribution among the Discovered Clusters

• Five clusters learned from Spring 21 had no significant distribution difference from Spring 22 (χ2(1, 5) = 3.038, p = .551), consistent with strategies persisting across the semester change.The Spring 21 clusters were applied to Spring 22’s 29 good trajectories.Source: Distribution among the Discovered… · p. 48, Clustering Prediction Tasks · p. 48

3.6 Results for Healthcare

• On MIMIC-III, EM-EDM won every metric, while results were mixed on the smaller CCHS and Mayo datasets, supporting dataset-size- and diversity-dependent gains.On CCHS, EM-EDM won AUC and AP; on Mayo, it won F1 and Jaccard, while other metrics favored EDM or, for CCHS AP, GP+DQN.Source: §3.6 · p. 49, Table 3.5 · p. 49

4.1 Overview

• The framework targets offline learning from selected student demonstrations that showed substantial learning benefit, addressing the need to model changing student goals without risky policy execution.The overview identifies evolving reward functions as important for representing changing learning needs and adaptive strategies across phases.Source: §4.1 · pp. 52–53

4.3.1 THEMES for Evolving Student Pedagogical Strategy

• THEMES partitions expert trajectories into time-aware sub-trajectories and uses offline EM-EDM to cluster them and induce a policy for each cluster.The resulting clusters and policies represent multiple student pedagogical strategies that evolve over time.Source: §4.1 · p. 53, §4.3.3 · pp. 57–58

4.1 Overview

• With 18 prior-semester trajectories, THEMES achieved AUC 0.899 and Jaccard 0.653, outperforming six state-of-the-art baselines and two ablations across performance metrics in later-semester prediction.The task was to predict students’ pedagogical decisions in an ITS using past-semester trajectories for training and a later semester for testing.Source: Modeling Evolving Reward Functions · p. 51, §4.1 · p. 53

4.2.2 Offline Apprenticeship Learning (AL)

• Prior pedagogical RL often required large datasets and predefined rewards, while existing offline AL commonly assumed one reward per dataset or fixed rewards within trajectories.EM-EDM addressed heterogeneous rewards across demonstrations and continuous states, but THEMES targets rewards that evolve over time.Source: §4.2.1 · pp. 53–54, §4.2.2 · p. 54, §4.2.3 · p. 55

4.3.2 Sub-trajectory Partitioning

• A sliding window captures temporal dependencies among student states, while a learned high-level reward regulator refines sub-trajectory partitioning iteratively until convergence.For irregularly timed ITS trajectories, a decay function allows cluster consistency to be reevaluated over time; high-level state-action pairs inform the reward regulator.Source: §4.3.1 · p. 56, §4.3.2 · p. 57

4.3.3 Policy Induction

• In policy induction, EM-EDM clusters the partitioned sub-trajectories in an E-step and applies EDM in an M-step to learn cluster-specific policies.This extends EDM’s single-reward assumption to demonstrations with multiple reward functions.Source: §4.3.3 · pp. 57–58

4.3.7 Multi-modal Imitation Learning (MIL)

• MIL jointly segments unstructured demonstrations and learns skill-specific policies using a latent intention variable (Hausman et al. 2017).Its objective encourages mutual information between intention and generated behavior, allowing one policy to imitate multiple demonstrated skills.Source: §4.3.7 · p. 58

4.3.8 Adapted Hierarchical Inverse Reinforcement Learning (AHIRL)

• HIRL segments long-horizon demonstrations into locally linear sub-tasks and learns rewards over an augmented state tracking sub-goal progress (Krishnan et al. 2016).Sub-tasks are detected with a Bayesian nonparametric Gaussian Mixture Model, and Maximum Entropy IRL learns the reward over the augmented space.Source: §4.3.8 · p. 59

4.5 Results

• THEMES significantly outperformed all other methods across evaluation metrics, including its ablations and baselines.It outperformed THEMES1, supporting the hierarchical reward regulator, and exceeded EM-EDM, AHIRL, and MIL. With 18 S21 trajectories, it reached an AUC of 0.899 and a Jaccard of 0.653 when predicting S22 strategies.Source: Table 4.1 · p. 62, §4.5 · pp. 62–63

4.4 Experiments

• Evaluation used semester-based 3-fold temporal cross-validation and 89 expert trajectories from 221 students in a Probability ITS.The ITS state representation comprised 130 continuous features; AUC and Jaccard were considered the most critical metrics.Source: §4.4.1 · p. 59, §4.4.3 · p. 60, §4.4.2 · p. 60

4.5.1 A Case Study

• A case study shows THEMES capturing evolving intentions across sub-trajectory clusters, unlike a trajectory that remains in one cluster.The changing cluster sequence illustrates why a single-reward method such as EDM may struggle, while THEMES uses sub-trajectory partitioning and clustering to adapt its policy.Source: §4.5.1 · p. 64, Figure 4.5 · p. 64

4.6 Summary

• THEMES induces pedagogical policies by combining time-awareness with hierarchically learned reward patterns, but its evidence is limited to one ITS.The work notes that evolving reward functions in student demonstrations are interpreted relatively narrowly and proposes testing student-ITS logs in diverse contexts.Source: §4.6 · pp. 64–65

Demonstrations with Evolving Rewards

• HALIDE hierarchically infers intent and strategy from suboptimal actions while ranking demonstrations and capturing students’ evolving goals and reward structures.It treats imperfect demonstrations as structured signals rather than noise, distinguishing transient errors from meaningful progress.Source: Demonstrations with Evolving Rewards · p. 66, §5.1 · p. 67

5.1 Overview

• HALIDE consistently outperforms baselines and ablations across evaluation metrics in cross-semester ITS prediction and is validated on three clinical EHR datasets.The ITS models train on past-semester student trajectories and predict pedagogical decisions in a subsequent semester; the clinical datasets concern septic shock prevention.Source: §5.1 · pp. 68–69

Pedagogical Policy Induction in ITSs

• In ITSs, apprenticeship learning can induce policies offline from student interaction logs, but using student choices as demonstrations introduces variability in demonstration quality.This contrasts with RL approaches that require substantial interaction data and carefully designed rewards.Source: Pedagogical Policy Induction in ITSs · p. 69

Apprenticeship Learning from Mixed-Quality Demonstrations

• Unlike approaches using flat policy spaces or coarse trajectory-level weighting, HALIDE incorporates continuous quality signals into hierarchical policy induction while preserving temporal abstraction.This extends THEMES, which assumes demonstrations are uniformly reliable and equally informative.Source: §5.1 · p. 68, Apprenticeship Learning from… · p. 70

Hierarchical Apprenticeship Learning and Evolving Strategies:

• HALIDE builds on THEMES by combining temporally evolving strategies and sub-trajectory partitioning with quality-aware hierarchical policy induction.THEMES models evolving rewards through sub-trajectories and cluster-specific policies, but assumes uniformly reliable demonstrations.Source: Hierarchical Apprenticeship Learning… · p. 70

5.3.1 Demonstration Quality and Weight Construction

• HALIDE maps continuous demonstration quality, which may come from outcomes or confidence estimates, to relative importance weights that modulate policy induction without changing trajectories or segmentation.The quality measure is aligned with demonstrated decisions; trajectory-level quality can be distributed across time as a coarse per-step proxy.Source: §5.3.1 · p. 71

5.3.2 High-level States Extraction (RMT–TICC)

• RMT–TICC uses a learned high-level reward regulator to shape temporal consistency during segmentation, modeling evolving pedagogical strategies without online interaction.The regulator is learned from high-level state-action pairs and distributed to regulate the consistency constraint.Source: §5.3.2 · p. 72

5.3.3 High-level Actions Induction (Weighted EM–EDM)

• Weighted EM–EDM applies decision weights to behavior cloning and occupancy regularization while remaining strictly offline; with all weights equal to 1, HALIDE reduces to standard EM–EDM.EM responsibilities are computed from unweighted likelihood, so quality affects policy induction rather than the assignment step.Source: §5.3.3 · pp. 72–73

5.3.4 High-level Reward Regulator and Effect of Ranking

• The high-level reward regulator shapes induced state–action distributions so higher-quality decisions guide strategy discovery while preserving hierarchical structure and evolving rewards.Source: §5.3.4 · p. 74

Expert Demonstration Selection

• Across five semesters, 244 students followed the same study, pre-test, ITS-training, and post-test protocol; Quantized Learning Gain selection identified 94 expert trajectories, with remaining trajectories treated as imperfect.The ITS offered three instructional choices, and the mixed-quality setting combined expert and imperfect trajectories.Source: Data Collection · p. 74, Expert Demonstration Selection · p. 75

Ranking Signals:

• HALIDE ranks demonstrations using semester-standardized Normalized Learning Gain transformed into smooth, bounded sigmoid weights, retaining useful sub-optimal behavior rather than hard-filtering trajectories.The ranking acts as a soft structural prior that emphasizes stronger learning outcomes while preserving informative decisions from lower-performing trajectories.Source: Ranking Signals · p. 76

5.5 Results for Education

• In education, HALIDE achieves the strongest overall performance across metrics; with expert and sub-optimal data, it reaches AUC 0.827 and Jaccard score 0.658.It substantially outperforms THEMES and weighted flat baselines by selectively reusing mixed-quality data while attenuating low-impact or erroneous decisions.Source: §5.5 · pp. 77, 79, Table 5.1 · p. 78

5.6 Results for Healthcare

• On expert-only data, THEMES with quality weighting significantly improves over unweighted THEMES on CCHS (F1: 0.722 vs. 0.696; AUC: 0.742 vs. 0.725) and Mayo (F1: 0.679 vs. 0.643; Jaccard: 0.535 vs. 0.490).Using all-quality data without quality modeling can help on MIMIC but can degrade performance on smaller CCHS, where EDM with weighting has F1 = 0.527 vs. 0.622 under expert-only training.Source: §5.6 · pp. 79–80

5.7 Discussion, Caveats, and Conclusions

• The work addresses heterogeneous decision quality among students and clinicians: treating demonstrations equally injects noise, while HALIDE modulates influence without discarding imperfect trajectories.This approach aims to preserve useful routine decisions and reduce the impact of low-quality ones across educational and clinical settings.Source: §5.7 · pp. 80–81

5.6 Results for Healthcare

• On MIMIC, HALIDE trained on all demonstrations achieves the overall best F1 = 0.869, AUC = 0.874, AP = 0.839, and Jaccard = 0.790.On Mayo, it attains the best AUC (0.632) and AP (0.559); on CCHS, expert-data THEMES with quality weighting remains top across most metrics.Source: §5.6 · p. 80, Table 5.2 · p. 81

5.7 Discussion, Caveats, and Conclusions

• HALIDE consistently improved performance, while unweighted hierarchical learning did not benefit from heterogeneous data; the largest gains used mixed-quality ITS demonstrations and all-quality MIMIC data.Quality-aware weighting stabilizes low-level policy induction, and hierarchy propagates those improvements to higher-level decision-making.Source: §5.7 · p. 81
• Quality is estimated at the trajectory level and propagated across steps, a scalable approach that only approximates step-level effectiveness in tutoring and clinical settings.This limits how precisely the method can represent the effectiveness of individual decisions.Source: §5.7 · p. 81
• Evaluations are offline and limited to septic shock prevention and one ITS, leaving closed-loop performance and generalization to other conditions and curricula untested.Historical-data biases, including differences in student preparation or engagement and clinician experience or case mix, may influence policy induction.Source: §5.7 · p. 82
• HALIDE depends on relative quality rankings, and noisy or misaligned rankings can attenuate quality-aware weighting’s benefits when they reflect short-term performance instead of true effectiveness.The authors emphasize careful ranking design and validation in practice.Source: §5.7 · p. 82
• HALIDE treats demonstration quality as a graded structural signal, combining quality-aware weighting with hierarchical modeling and evolving rewards rather than selecting demonstrations through a binary criterion.The authors present this approach as a path toward apprenticeship learning in heterogeneous education and healthcare settings.Source: §5.7 · p. 82

6.1 Overview

• Cchal jointly weights decisions by Bayesian outcome attribution and expert-versus-non-expert behavioral criticality, then uses the combined weights for policy induction and Bradley-Terry contrastive reward learning.Its contrastive objective uses observed failure trajectories; the criticality score is computed without a trained policy.Source: §6.1 · p. 83
• Cchal outperforms every measured baseline on every reported metric on CCHS, Mayo, and ITS; on MIMIC-III, it is statistically indistinguishable from the strongest prior baseline on F1 and AUC but exceeds it on APR and Jaccard.The evaluation covers three clinical EHR datasets and one intelligent tutoring system dataset.Source: §6.1 · p. 84

6.2 Introduction

• Cchal builds on THEMES’s hierarchical sub-trajectory clustering to combine per-decision outcome quality and behavioral criticality in a joint attention weight that scales Bradley-Terry contrastive IRL.The framework is evaluated on three clinical EHR datasets for septic shock prevention and one ITS dataset for student modeling.Source: §6.2 · pp. 85–86
• Outcome-attributed weighting redistributes trajectory quality across feature-action pairs, while dual-arm criticality scoring contrasts local action distributions for successful and failed demonstrations matched by initial risk stratum.The criticality measure requires no trained policy, cluster label, or MDP model.Source: §6.2 · p. 86

6.3 Related Work

• Unlike classical AL and methods such as SQIL and IQL, which treat timesteps uniformly, T-REX and DemoDICE address imperfect demonstrations at the trajectory level rather than weighting individual decisions (Reddy et al. 2020; Kostrikov et al. 2022; Brown et al. 2019).THEMES applies trajectory-level quality uniformly across timesteps, whereas Cchal decomposes it within trajectories and contrasts expert with non-expert behavior.Source: §6.3 · pp. 86–87
• Cchal repurposes GP inference to attribute observed trajectory quality to individual decisions without environment rewards, and identifies criticality from success-failure behavioral contrast rather than a trained policy (Ausin et al. 2021; Roth et al. 2019; Eysenbach et al. 2019).The authors suggest that de-emphasizing low-criticality states may reduce reproduction of unnecessary intervention, but direct verification requires action-level overtreatment labels not available in these data.Source: §6.3 · pp. 87–88
• Unlike adversarial IRL’s generated negatives and offline contrastive divergence minimization, Cchal uses observed failure demonstrations as negatives and scales contrastive learning by per-state criticality (Fu et al. 2018; Ni et al. 2021).Cchal extends hierarchical offline apprenticeship learning for EHR by moving from trajectory-level to decision-level attention.Source: §6.3 · p. 88

6.4 Preliminaries

• CCHAL’s setup assigns trajectories outcome labels and quality ranks, divides patients by initial risk and outcome, and uses shared MT-TICC state clusters.The four risk-outcome groups distinguish a favorable target population from shock outcomes used as a contrast reference.Source: §6.4 · p. 89

6.5 The Cchal Framework

• CCHAL proceeds through five phases, from per-decision outcome attribution and temporal clustering to criticality-weighted policy induction and Bradley-Terry contrastive IRL.Figure 6.1 presents the framework overview; Algorithm 3 summarizes the end-to-end procedure.Source: Figure 6.1 · p. 90, §6.5 · p. 90

6.5.1 Phase 0: Outcome-Attributed Decision Weighting

• InferGP uses trajectory quality ranks and a feature-action RBF-kernel GP to distribute outcome attribution across individual decisions, independently of their temporal positions.This estimates decision importance from outcome quality rather than requiring an environment reward signal, unlike RL temporal credit assignment.Source: §6.5.1 · p. 91

6.5.2 Phase 1: MT-TICC Temporal State Clustering

• MT-TICC assigns training timesteps to discrete temporal state clusters in inductive mode, following standard practice without CCHAL-specific modification.MT-TICC is also identified as Hallac et al. 2017 in the preliminaries.Source: §6.4 · p. 89, §6.5.2 · p. 92

6.5.3 Phase 2: Dual-Arm Contrastive Critical State Scoring

• CCHAL scores decision criticality continuously with kNN-JSD in clinical feature space, using independent contrastive arms whose populations share the same initial risk stratum.When the low-risk arm is inactive, CCHAL falls back to a single high-risk arm.Source: §6.5.3 · pp. 92–93

6.5.4 Phase 3: Joint Critical-Quality Weighting and EM-EDM

• Weighted EM-EDM combines InferGP quality attribution with arm-routed criticality, training the policy on favorable demonstrations while excluding failures from its generative model.The joint weight gives greatest influence to high-quality decisions at critical states measured against a risk-matched contrast group.Source: §6.5.4 · pp. 93–94

6.5.5 Phase 4: Bradley-Terry Pairwise Contrastive IRL

• CCHAL learns a linear reward with a Bradley-Terry pairwise preference objective, contrasting positive demonstrations against failures while routing action-aware weights by initial risk stratum.Using all four risk-outcome groups for contrast preserves failure examples for IRL while keeping them out of policy induction.Source: §6.5.5 · pp. 94–95

6.5.6 Full Algorithm

• Algorithm 3 summarizes CCHAL’s end-to-end procedure.The framework overview describes its five phases, from Bayesian attribution through contrastive reward learning.Source: §6.5 · p. 90, §6.5.6 · p. 95

6.6.1 Datasets and Evaluation

• The evaluation covers CCHS, Mayo Clinic, and MIMIC-III septic-shock trajectories plus a Pyrenees ITS dataset, using 10 random 80/20 splits and paired Wilcoxon tests at p<0.05.Metrics are Accuracy, AUC-ROC, AUPRC, F-score, and Jaccard; clinical demonstration ranks derive from an offline shock-outcome predictor, while ITS uses QLG.Source: §6.6.1 · p. 96

Data Collection.

• Clinical data comprise 14 sepsis-related features, with missingness averaging ∼80% for CCHS and Mayo and ∼50% for MIMIC-III; remaining gaps are forward-filled then mean-imputed.The clinical cohorts include hospitalized adults at CCHS and Mayo and ICU visits at MIMIC-III; shock labels follow clinician-applied Sepsis-3 criteria.Source: Data Collection · pp. 97–98

Education (ITS) Dataset

• The ITS cohort includes 221 students across four semesters, with 89 high-QLG trajectories used as expert demonstrations; 130 continuous features represent five interaction categories.Students choose to solve independently, study a worked solution, or collaborate with the tutor; the features cover autonomy, timing, problem-solving context, performance, and hint use.Source: Education (ITS) Dataset · p. 98

Shared Implementation Details

• Across all four datasets, MT-TICC uses K=9 temporal state clusters and dual-arm criticality uses k=15 nearest neighbours with a sparse-support fallback.Results average 10 random 80/20 train/test splits, with significance assessed by paired Wilcoxon signed-rank tests (p<0.05).Source: Shared Implementation Details · pp. 98–99

6.6.2 Baselines

• Seven baselines span behavior cloning, single- and multiple-reward methods, and evolving-reward methods; THEMES is the strongest baseline and Cchal’s direct methodological predecessor.The comparison also includes Cchal-NoGP, Cchal-NoCrit, and Cchal-NoContrast, each removing one component of the joint weighting scheme.Source: §6.6.2 · p. 99

6.6.3 Results

• Cchal outperforms every baseline on every reported metric for CCHS, Mayo, and ITS; on MIMIC-III it is essentially tied with THEMES on F1 and AUC while ahead on APR and Jaccard.The overall comparison is reported in Table 6.2; the MIMIC-III F1/AUC differences are within one reported standard deviation.Source: §6.6.3 · pp. 100–101, Table 6.2 · p. 101

6.6.4 Ablation Analysis

• Cchal-Full matches or exceeds all three ablations across datasets and metrics, with component gains larger on longer clinical trajectories than on the 10-decision ITS trajectories.Removing any component costs an average of 31% relative Jaccard on Mayo, 23% on CCHS, and 13% on MIMIC-III, but under 1% on ITS; the ITS rank mechanism had also been validated by Halide (Islam et al. 2026).Source: §6.6.4 · p. 101

6.6.5 Early Sepsis Intervention & A Case Study

• Cchal’s first treatment in the CCHS case study occurs at 65.3 minutes after arrival, 420.4 minutes (∼7.0h) earlier than the expert’s at 485.7 minutes.The patient showed abnormal vital-sign patterns before the expert’s initial treatment.Source: §6.6.5 · pp. 102, 104, Figure 6.2 · p. 103
• CCHAL’s outcome-attributed GP credit, dual-arm criticality, and pairwise contrastive reward learning focus policy induction on clinically important decision points.The resulting action distribution still closely follows the expert’s subsequent treatment pattern.Source: §6.6.5 · p. 104
• Expert decisions reflect experience, workflow, and available information, so they may not represent the earliest possible intervention.CCHAL is intended to model successful decision-making patterns and support earlier intervention when patient states resemble critical points in the demonstrations, not replace clinical judgment.Source: §6.6.5 · p. 104

Why the margin narrows on MIMIC-III

• Removing dual-arm criticality costs 31.4% relative Jaccard on Mayo, 22.6% on CCHS, and 13.6% on MIMIC-III, where uniform high acuity and a larger training pool leave less headroom.MIMIC-III is exclusively ICU admissions, with a training pool of approximately 6,400 trajectories, roughly two orders of magnitude larger than CCHS or Mayo.Source: §6.7 · p. 104
• CCHAL improves over THEMES on every metric for CCHS, Mayo, and ITS; on MIMIC-III it matches F1 and AUC and exceeds APR and Jaccard.The discussion interprets this as the predicted behavior of a mechanism whose value scales with population heterogeneity, rather than a failure on MIMIC-III.Source: §6.7 · p. 104

Connection to CriticalDRL

• Unlike CriticalDRL’s policy-derived Q-value-spread criticality, CCHAL derives criticality from observed behavioral contrast without a trained policy and uses it to concentrate learning signal.This is advantageous offline when a reliable online DQN cannot be trained; CriticalDRL uses criticality to reduce decision burden.Source: §6.7 · p. 105

InferGP kernel choice

• InferGP uses an RBF kernel in feature-action space so clinically similar state-action pairs share timestep quality credit regardless of trajectory position or length.A Matérn-1/2 kernel could capture sharper transitions, but preliminary experiments found RBF slightly better, consistent with relatively smooth physiological trajectories.Source: §6.7 · p. 105

Limitations

• Dual-arm kNN criticality may become noisy when either pool has too few neighbors, and InferGP does not account for distribution shift between the full training set and train/test splits.The concern is visible on ITS, whose fixed 10-step horizon leaves few within-trajectory contrasts; proposed remedies include pool-size-adaptive k, cluster-level JSD, and per-fold GP inference at higher computational cost. The low-risk arm also becomes inactive when only high-risk positive data are available, degrading to single-arm behavior.Source: §6.7 · p. 105

• Cchal combines GP-based outcome attribution, data-driven contrastive criticality scoring, and Bradley-Terry pairwise IRL to address demonstration-quality variation and decision-importance heterogeneity without a trained policy or reward signal.It prioritizes decisions that contributed to trajectory outcomes and distinguish expert from non-expert clinical behavior.Source: §6.8 · pp. 105–106
• Cchal consistently improves over THEMES and other published baselines on three EHR datasets, outperforms every baseline on an ITS dataset, and benefits independently from all three ablated components.The ablations test outcome attribution, criticality scoring, and pairwise contrastive IRL.Source: §6.8 · p. 106
• Initial ITS evidence suggests Cchal’s principle generalizes beyond clinical EHR: both outcome attribution and behavioral criticality are needed to focus offline imitation learning on important decisions.The proposed scope is settings with access to both successful and failed demonstrations.Source: §6.8 · p. 106

7.1 Overview

• StuEDM uses Energy-based Distribution Matching to imitate expert tutoring behavior offline, intervening only when a student’s chosen presentation mode conflicts with its learned policy.Students chose among Problem Solving, Worked Example, and Collaborative Problem Solving; training used a substantially smaller, semester-limited corpus of student-choice demonstrations than the corpus used for StuCQL.Source: §7.1 · p. 108

7.2 Introduction

• The study compared StuEDM with StuCQL, trained on a large interaction corpus, and an uninformed Expert baseline in a controlled classroom study with 104 students.All three conditions showed significant pre-to-post learning gains (partial η2 = 0.29–0.69).Source: §7.1 · p. 108

7.1 Overview

• StuEDM achieved the highest normalized learning gains across all five measures, significantly outperforming Expert on four and StuCQL on pretest-adjusted posttest.Against Expert, planned comparisons were significant for pretest-adjusted posttest (p=0.017), normalized learning gain (p=0.016), isomorphic normalized learning gain (p=0.038), and score-ratio normalized learning gain (p=0.015). Against StuCQL, only pretest-adjusted posttest was significant (p=0.048); other comparisons trended in the same direction (p=0.067–0.117).Source: §7.1 · pp. 108–109
• StuEDM overrode students on 28.2% of decisions versus 66.7% for StuCQL, steering toward Problem Solving while preserving some mode diversity.StuCQL’s overrides converged almost entirely on Worked Examples and never selected Collaborative Problem Solving; students using StuEDM also engaged with the tutor longer than StuCQL students (p=0.008).Source: §7.1 · p. 109

7.3.1 Student Agency and Mixed-Initiative ITS Policies

• Student-choice-with-override policies share control between learners and systems, but how often and toward which modes learned policies override students had been under-characterized.The study examines whether training from students’ own choices, rather than tutor- or randomly-assigned decisions, affects agency preservation.Source: §7.3.1 · pp. 112–113

7.3.2 Offline Reinforcement Learning for ITS Policy Induction

• Offline RL methods such as CQL address overoptimistic values outside the training distribution but still require large interaction corpora for reliable policy learning.The section contrasts reward optimization and exploratory interaction logs with apprenticeship learning from demonstrations of students’ own decisions.Source: §7.2 · p. 110, §7.3.2 · p. 113

7.3.3 Apprenticeship Learning and Energy-based Models

• Energy-based Distribution Matching enables strictly offline apprenticeship learning by matching learner and expert state occupancies without environment rollouts.This work extends student-choice apprenticeship learning by deploying the induced policy as a live classroom intervention and evaluating agency preservation.Source: §7.3.3 · p. 114

7.3.4 ITS Policy Evaluation in the Classroom

• The classroom evaluation follows established practices including checking pretest equivalence, using normalized learning gain, and administering isomorphic posttests to assess transfer.The section notes that modest classroom sample sizes, variable prior knowledge, and small between-condition effects complicate live ITS policy evaluation.Source: §7.3.4 · pp. 114–115

7.4.2 Experimental Conditions

• StuEDM imitated student demonstrations from 244 students across five semesters, weighting decisions by estimated expertise, whereas StuCQL optimized reward using logs from 2,716 students across 13 semesters.Both policies accepted students’ presentation-mode choices unless they conflicted with the learned policy; Expert assigned modes uniformly at random without student input.Source: §7.4.2 · pp. 116–117

7.6.2 Within-Condition Learning Gains

• All three conditions showed significant pre-to-post gains on both the standard and isomorphic posttests, with the largest effect sizes in StuEDM and StuCQL and the smallest in Expert.For the standard posttest, partial η² was 0.298 for Expert, 0.525 for StuCQL, and 0.561 for StuEDM; isomorphic-posttest values were 0.477, 0.688, and 0.629, respectively.Source: §7.6.2 · pp. 119–120

7.6.3 Between-Condition Comparisons

• On the pretest-adjusted standard posttest, StuEDM beat StuCQL (t(76) = 1.690, p = 0.048) and Expert (t(76) = 2.154, p = 0.017), although the overall condition effect was nonsignificant (p = 0.084).The adjusted model’s condition effect was F(2, 76) = 2.560, p = 0.084, partial η² = 0.063.Source: §7.6.3 · p. 120
• StuEDM’s NLG_post mean was 0.685 versus 0.485 for StuCQL and 0.331 for Expert; it beat Expert (p = 0.016), but not StuCQL (p = 0.111).The overall ANOVA was nonsignificant; the planned contrast against the combined baseline was significant (p = 0.027).Source: §7.6.3 · p. 120
• On post_iso, StuEDM’s mean was 0.948 versus 0.888 for StuCQL and 0.885 for Expert; neither pairwise comparison was significant, but the combined-baseline contrast was (p = 0.038).For NLG_post_SR, StuEDM beat Expert (p = 0.015), not StuCQL (p = 0.114); the combined-baseline contrast was significant (p = 0.026).Source: §7.6.3 · pp. 120–121
• For NLG_post_iso, StuEDM’s mean was 0.829 versus 0.676 for StuCQL and 0.600 for Expert; it beat Expert (p = 0.038), but not StuCQL (p = 0.117).The planned contrast against the combined baseline was significant (p = 0.045), while the overall ANOVA was nonsignificant.Source: §7.6.3 · p. 121
• StuEDM students spent 1.52h on training versus 1.11h for StuCQL (p = 0.008), while StuEDM and Expert did not differ (p = 0.617).Time on task was treated as voluntary engagement; StuCQL students also spent less time than Expert students (1.11h vs. 1.44h; p = 0.029). The StuEDM–StuCQL difference persisted in S26-only data (p = 0.019).Source: §7.6.3 · p. 121

7.6.4 RQ2: Student Agency — Decision Distribution and Policy Overrides

• After override, StuCQL delivered Worked Examples on 76.9% of decisions, while StuEDM delivered Problem Solving on 82.3%; across the remaining nine problems, student choices diverged significantly (chi-square(2)=36.73, p<.001).Source: §7.6.4 · p. 122
• StuCQL overrides went to Worked Examples 85.8% of the time and never to Collaborative Problem Solving; StuEDM overrides went to Problem Solving 81.8% and Collaborative Problem Solving 2.7%.All three pairwise differences in applied decision distributions were individually significant (p< .001).Source: §7.6.4 · pp. 122–123

7.6.5 Summary of Results

• StuEDM had the highest mean across all five outcomes, significantly outperforming Expert on four and StuCQL only on the pretest-adjusted posttest.Three-group ANOVAs were non-significant with N=26–27 per condition; the remaining direct StuEDM–StuCQL comparisons were also non-significant (p=0.067–0.117).Source: §7.6.5 · p. 123

7.6.4 RQ2: Student Agency — Decision Distribution and Policy Overrides

• StuCQL overrode 66.7% (260/390) of decisions, compared with 28.2% (110/390) for StuEDM (χ2(1) = 114.15, p< .001).Twelve of 39 StuCQL students were overridden on all 10 problems, while 13 of 39 StuEDM students were never overridden.Source: §7.6.4 · p. 122, Figure 7.2 · p. 124

7.7 Discussion and Conclusions

• StuEDM’s learning advantages came from training on 244 students’ trajectories from five semesters, versus StuCQL’s 2,716-student, 13-semester corpus.Its advantage over StuCQL was directionally consistent on the other four measures but did not reach significance; the authors characterize that broader advantage as suggestive, not conclusive.Source: §7.7 · p. 125
• The discussion recommends reporting override rate and delivered-mode coverage alongside learning outcomes, and evaluating override rationales rather than assuming transparency helps.The authors note that StuCQL eliminated Collaborative Problem Solving from students’ delivered curriculum, despite soliciting their choices.Source: §7.7 · p. 126

7.8 Ethics and Privacy Statement

• Student interaction logs and test scores were de-identified before analysis and stored securely under an IRB-approved protocol.The policies were specific to the Pyrenees probability tutor and were not intended for deployment outside supervised educational research without further validation.Source: §7.8 · pp. 128–129

8.1 Conclusions

• EM-EDM clustered student trajectories by latent reward and induced cluster-specific policies, outperforming four AL and two DRL baselines with as few as 24 expert demonstrations.The framework addressed heterogeneous reward functions in continuous state spaces.Source: §8.1 · p. 130
• THEMES partitioned trajectories into temporally coherent sub-trajectories using a hierarchical reward regulator and outperformed flat and hierarchical AL baselines across educational and clinical datasets.Using 18 past-semester demonstrations, it achieved an AUC of 0.899 and a Jaccard of 0.653 when predicting future-semester decisions.Source: §8.1 · p. 130
• HALIDE ranked demonstrations using continuous learning-gain-derived quality signals, enabling reuse of mixed-quality data and consistent gains under cross-semester shift, including among expert-labeled trajectories.Incorporating suboptimal trajectories without quality-aware ranking degraded performance relative to unweighted hierarchical baselines.Source: §8.1 · p. 131
• CCHAL assigned quality to individual decisions through Bayesian credit assignment, dual-arm criticality scoring, and contrastive IRL over failures, outperforming AL baselines including HALIDE on clinical and tutoring datasets.Its results support decision-level outcome attribution and behavioral criticality as additions to trajectory-level quality weighting.Source: §8.1 · p. 131
• StuEDM significantly improved posttest outcomes over the Expert baseline and pretest-adjusted posttest scores over StuCQL, while overriding students less than half as often.It was trained on 244 students’ decisions across five semesters, compared with StuCQL’s 2,716-student, 13-semester corpus; StuEDM students also spent significantly longer on task.Source: §8.1 · p. 131

8.2 Future Work

• Future work proposes unifying the frameworks’ mechanisms and reducing dataset-size sensitivity through adaptive neighborhoods, cluster-level criticality, per-fold credit inference, and alternatives to single-signal quality weighting.It also proposes studying quality-transform sharpness and per-decision weighting for HALIDE and StuEDM.Source: §8.2 · p. 132
• Future classroom studies should match time budgets, replicate across semesters, report override rate and delivered-mode diversity, and test whether lightweight override rationales affect student decisions.These follow-ups address longer StuEDM task time, a single-semester study, and the differing override patterns of StuEDM and StuCQL.Source: §8.2 · pp. 132–133
• Further directions include scaling to multimodal human data, improving evaluation with causal and counterfactual methods, adding selective human feedback, and analyzing theoretical reliability.The proposed work includes improved representation learning and energy-based optimization, hybrid simulation–human studies, and analysis of generalization, identifiability, and stability.Source: §8.2 · p. 133

APPENDIX A Acronyms

• Appendix A points readers to Table A.1 for the acronym summary.Source: Appendix A · p. 145
• Table A.1 summarizes the paper’s acronyms in alphabetical order.The appendix identifies a continuation of the table.Source: Table A.1 · pp. 145–146

APPENDIX B Variables

• Table B.1 summarizes common variables and their definitions in alphabetical order, with the table continued in a second passage.The appendix states that the summary covers variables used throughout the dissertation.Source: Table B.1 · pp. 147–148, Appendix B · p. 147