Source-linked AI summary
CuraEval: Evaluating Demonstration Curation on Mixed-Skill Robot Data
Jiuyi Xu, Meida Chen, Yang Ye, Yangming Shi
TL;DR
Selecting useful demonstrations is challenging because mixed-skill data vary and scoring, selection, and evaluation protocols differ. CuraEval combines score analysis with controlled subset comparisons against random selections and full-data training. Its results show that simple heuristics can outperform random selection, while adapted diversity selection can underperform and protocol changes can reverse estimated gains.
Problem
Demonstration scores and evaluation protocols vary, complicating comparisons of whether selection methods outperform simple baselines.
Source: §1 · p. 1Method
CuraEval combines mathematical analysis of demonstration scores with controlled evaluations of mixed-skill robomimic data against random subsets, heuristics, and full-data training.
Source: §1 · pp. 1–2Results
Takeaways & Limitations
Interpret curation comparisons alongside the selected data, selection rule, and evaluation protocol, distinguishing gains over random data from gains over full-data training.
Source: §1 · p. 2, §6 · p. 8Evidence is limited to small state-based datasets and the stated adaptations and policy configurations; common evaluator seeds do not capture uncertainty across independently drawn evaluation sets.
Source: §5 · p. 8
Abstract
from arXiv · showhide
Robot imitation learning uses demonstrations from operators of varying skill, but selecting training data remains challenging. Existing methods score demonstrations using smoothness, diversity, or estimated influence on policy performance, but differences in selection and evaluation protocols make it difficult to determine whether they outperform simple baselines. We introduce \textbf{CuraEval}, a framework combining mathematical analysis of demonstration scores with controlled comparisons against fixed size-matched random subsets and full-data training on mixed-skill robomimic data using Diffusion Policy and BC-RNN. We show that the evaluated PSD score depends only on episode length and squared position magnitudes, and produces nearly the same ranking as episode length on \textit{square} ($\rho=0.999$). With Diffusion Policy, selecting the shortest 50\% of demonstrations improves \textit{square} success from 70.7\% under random selection to 76.5\%. Compared with random selection, our adapted geometric-entropy LOO selector reduces success at 25\% retention from 94.7\% to 74.0\% on \textit{can} and achieves 41.7\% on \textit{square}, below the mean success rates of all three independently sampled random subsets (59.8--62.5\%). On the same 3 \textit{square} Diffusion Policy training seeds, length-based selection at 75\% retention improves success over full-data training from 76.7\% to 86.7\% under the original protocol, but lowers it from 80.5\% to 79.5\% after checkpoint reporting, rollout count, and evaluation seed change jointly. No evaluated curated condition significantly outperforms full-data training after Holm correction, although confidence intervals still permit useful gains. Code is available at https://github.com/jiuyixu25/CuraEval.
1 Introduction
CuraEval studies how mixed-skill demonstrations are scored and selected, comparing curated subsets with random subsets and full-data training. Its results distinguish selection gains over random data from gains over full-data training and show sensitivity to evaluation protocol.
1 Introduction
- Existing demonstration scores use trajectory statistics [3] [4], diversity [5] [6], mutual information [8], or estimated policy influence [7], but differing selection and evaluation protocols complicate comparisons.Prior methods also appear in robot-data pipelines, including JoyAI-RA 0.5 [1] and WARP-RM [2].Source: §1 · p. 1
- CuraEval combines mathematical score analysis with controlled evaluation on state-based, mixed-skill robomimic data [11], using Diffusion Policy [12] and selected BC-RNN comparisons [11].Comparisons include fixed-budget random subsets, simple heuristics, and all data.Source: §1 · pp. 1–2
- 41.7% on square for adapted geometric-entropy LOO falls below all three tested random-subset means (59.8–62.5%), while no curated condition has a Holm-adjusted full-data advantage.Changing the evaluation protocol reverses the estimated benefit of length selection on the same training seeds.Source: §1 · p. 2
- The study analyzes the evaluated PSD score and compares selections with random subsets, simple heuristics, and full data, while testing whether protocol changes reverse estimated curation benefits.Successful demonstrations vary in speed, consistency, and operator skill, and selecting smooth or efficient trajectories may remove useful coverage.Source: §1 · pp. 1–2
2 Related Work
Related work scores demonstrations using trajectory statistics, diversity, information, and estimated influence. CuraEval focuses on how such scores and selection procedures relate to downstream policy performance under controlled comparisons.
2 Related Work
- PSD [3] and RINSE’s spectral arc length [4] assess trajectory statistics, while Geometric Entropy [5] reports a non-monotonic diversity–performance relationship and FAKTUAL [6] notes low-budget selection can trail random selection.CuraEval distinguishes these original methods from its explicitly scoped adaptations.Source: §2 · p. 2
- DemInf [8] estimates state–action mutual information; CUPID [7] estimates demonstration influence from policy rollouts, while QoQ [9] and DataMIL [10] use other influence-based approaches.The paper's CUPID implementation provides a policy-dependent comparison.Source: §2 · p. 2
- Prior work studies demonstration quality and composition [11] [16], injected defects and episode-length confounding [13] [14] [15], and policy-selection uncertainty [11] [17].CuraEval examines natural operator variation and connects score definitions to subset selection and policy performance.Source: §2 · p. 2
3 CuraEval
CuraEval evaluates demonstration scores and selection procedures against size-matched random subsets and full-data training under fixed task–policy training budgets. It analyzes trajectory and diversity measures and implements a policy-dependent influence comparison.
3.1 Evaluation Framework
- Selectors retain m of N demonstrations at retention r = m/N, with policies compared against equally sized random subsets, heuristic selections, and all data under fixed hyperparameters and gradient-step budgets.The overview covers 25/50/75% retention from 300 mixed-skill demonstrations, with Diffusion Policy as the main policy and selected BC-RNN comparisons.Source: §3.1 · p. 2, Figure 1 · p. 3
3.2 Scores and Selection Procedures
- The evaluated scores include PSD, spectral arc length, and heuristics such as episode length, velocity spectral power, and mean squared jerk; lower is preferred for PSD and heuristics.The whole-trajectory Geometric Entropy adaptation resamples trajectories to 50 arc-length points, translates endpoints to the origin, and normalizes by displacement; SigEnt uses a depth-3 signature kernel.Source: §3.2 · p. 3
- LOO ranks full-dataset contributions once and retains the largest, while forward-greedy repeatedly adds the demonstration maximizing the subset objective.Both procedures are applied to the adapted geometric-entropy measure; SigEnt greedy maximizes entropy without FAKTUAL’s determinant-based stage.Source: §3.2 · p. 3
- The CUPID implementation uses TRAK-projected gradients and aggregates influence on successful rollouts minus failed-rollout influence to produce a fixed ranking per retention level.Its ranking averages demonstration ranks from three scoring policies; the ensemble costs 36 GPU-hours before downstream training.Source: §3.2 · p. 3
4 Experiments
Experiments compare fixed-size selections with random subsets and all-data training on mixed-skill robomimic tasks, while examining score behavior and evaluation-protocol sensitivity. Results include a near length-equivalent PSD ranking, heuristic gains over random on square, and no Holm-adjusted curated advantage over full-data training.
4.1 Experimental Settings
- The experiments use 300 successful demonstrations per task, fixed training budgets, size-matched random subsets, all-data controls, and final-checkpoint evaluation.can and square span six operators and three skill levels; transport covers six operator-skill pairings. Final checkpoints use 200 rollouts on square and transport and 100 on can, with a common evaluator seed; the training seed is the analysis unit.Source: §4.1 · pp. 3–4
4.2 What Do Demonstration Scores Measure?
- The evaluated PSD score is unchanged by reordering positions, and its ranking closely tracks episode length when within-task position magnitudes vary little.On square, PSD and length select the same 150 demonstrations at 50% retention, with Spearman correlation 0.999; across eight downstream comparisons, mean success differences are at most 0.030, with intervals including zero, which does not establish equivalence. Velocity-PSD, jerk, and diversity rankings do not uniformly favor the same demonstrations.Source: §4.2 · p. 4
4.3 Downstream Policy Performance
- At 50% retention, square DP length selection raises success from 70.7% to 76.5% versus random, while 25% HG-WT-LOO falls from 62.5% to 41.7%.On can, HG-WT-LOO falls from 94.7% to 74.0%; on square, it and SigEnt-LOO score below all three random-subset means of 59.8%, 60.7%, and 62.5%. On can, selected means of 99.2–99.3% do not significantly outperform random or full data; the highest square BC-RNN 75%-retention mean is 67.5% for SAL versus 67.2% for full data.Source: Table 1 · p. 5, §4.3 · pp. 5–7, Table 2 · p. 6
4.4 What Do the Selectors Retain?
- At 25% retention on square, HG-WT-LOO keeps 51 lower-skill demonstrations, while length selection keeps 60 higher-skill demonstrations.The selected sets can differ in outcome even with matching operator-skill counts: SigEnt-LOO and entropy-greedy each retain 16 higher-, 24 middle-, and 35 lower-skill demonstrations, but achieve 47.0% and 58.0% success. CUPID’s merged ranking correlates −0.14 with length, and selected-set overlap across its three scorers is 0.22–0.24.Source: §4.4 · pp. 7–8
4.5 Sensitivity to Evaluation Protocols
- On the same three square DP training seeds, length selection at 75% retention changes from 86.7% versus 76.7% full-data success under the original protocol to 79.5% versus 80.5% under the updated protocol.Checkpoint reporting, rollout count, and evaluation seed changed jointly, so their individual effects are not isolated. The paper recommends explicit reporting of these choices and independent testing after checkpoint selection, or a prespecified endpoint or averaging rule.Source: §4.5 · p. 8
5 Discussion and Limitations
Curation should be evaluated against distinct goals and budgets, with practical tolerance defining whether fewer demonstrations retain performance. Evidence is limited to small state-based datasets and stated adaptations and policy configurations.
5 Discussion and Limitations
- Curation evaluations should distinguish beating random selection, improving over full-data training, and retaining performance with fewer demonstrations; non-significant loss alone is insufficient.Fixed gradient-step budgets compare performance at the same number of updates, not training time to a target success rate.Source: §5 · p. 8
- Equal transition or runtime budgets would complement fixed gradient-step budgets, since matching demonstration counts does not match transition counts.Source: §5 · p. 8
- Evidence covers small state-based datasets and stated adaptations and policy configurations; can is near ceiling and transport remains under-converged.Source: §5 · p. 8
- Only square-DP at 25% retention tests multiple random subsets, and common evaluator seeds do not measure uncertainty across independently drawn evaluation sets.Full-data DP success on square is 79.7%, versus 87.0% under the PSD paper’s different implementation and data split [3]; the discrepancy remains unresolved.Source: §5 · p. 8
6 Conclusion
CuraEval combines score analysis with controlled selection experiments. Results show that simple heuristics can outperform random selection, while aggressive diversity-based selection can reduce success; no curated condition has a Holm-adjusted full-data advantage.
6 Conclusion
- No curated condition has a Holm-adjusted full-data advantage, although useful gains remain compatible with the intervals.Source: §6 · p. 8
- The evaluated PSD score closely tracks episode length, while simple heuristics can outperform random selection.Source: §6 · p. 8
- Aggressive selection with the diversity adaptations can substantially reduce success, and square losses persist across three random subsets.Source: §6 · p. 8
- Joint protocol changes reverse the estimated benefit of length selection, motivating reports that identify selected data, selection rule, and evaluation protocol together.Source: §6 · p. 8