PaperlayerView exact revision on arXiv

Source-linked AI summary

From 12 to 200 Demonstrations: How ACT, Diffusion Policy, SmolVLA and $\pi_{0.5}$ Scale on a Low-Cost Arm

Simon Cowling

arXiv:2610.37170v1cs.RO

TL;DR

Practitioners choosing a policy for a new task also need to decide how many demonstrations to record. This study examines policy outcomes and failure stages, finding that the leading policy depends on data budget and that π0.5’s fine-tuning recipe strongly affects its result.

  • Problem

    For a new task, practitioners must choose both which policy to train and how many demonstrations to record.

    Source: §1 · p. 1

  • Method

    The study uses joint-state and wrist-camera recordings to place episodes on a cumulative four-stage ladder, and examines distinct π0.5 fine-tuning recipes.

    Source: §3.6 · p. 9, §4.4 · p. 15

  • Results

    With a few dozen demonstrations, fine-tuned π0.5 was far ahead; with 100 or more, a policy trained from scratch did at least as well, while π0.5 recipe choice shifted success from 4% to 65%.

    Source: §6 · p. 21, §9 · p. 24

  • Takeaways & Limitations

    For π0.5, results should be read in view of the fine-tuning recipe, since a reader testing only LoRA would conclude it does not work on this arm.

    Source: §6 · p. 21

    Data budget and training exposure are confounded within every recipe, limiting interpretation of the high-data results.

    Source: §7 · p. 22

Abstract

from arXiv · show

We trained four widely used imitation-learning policies, ACT, Diffusion Policy, SmolVLA and $\pi_{0.5}$, on nested subsets of 12, 25, 50, 100 and 200 teleoperated demonstrations of one pick-and-place task with distractor objects, using one fixed training recipe per policy. Every trained model was then run on a physical SO-101 arm on a fixed set of object placements: 1,340 scored episodes in 126 blocks. ACT, trained from scratch, rose from 1 success in 40 episodes at 12 demonstrations to 37 in 40 at 100. $\pi_{0.5}$ with its action expert fine-tuned was the best policy when data was scarce (16 of 40 at 12 demonstrations, 28 of 40 at 25) and then stayed near 70%; from 100 demonstrations ACT was ahead, by a margin this study does not fully resolve. SmolVLA and Diffusion Policy peaked near 50%, and both were affected by a failure to start: SmolVLA's arm never started moving in 47% of episodes, and Diffusion Policy hardly ever started until an idle lead-in was trimmed from the demonstrations. For $\pi_{0.5}$ the fine-tuning recipe mattered more than the amount of data: low-rank adaptation succeeded in 4 of 110 episodes, against 130 of 200 for action-expert fine-tuning. From the joint and camera recordings we place every episode on a four-stage ladder (started, reached the ball, carried it, delivered it) and show that for ACT more data moves failures from reaching, to grasping, to none, while the failures $\pi_{0.5}$ retains are in grasping. We also measure how weak a ten-episode block is as a measurement (the same checkpoint tested twice differs by 2 successes in 10 on average), separate seed-to-seed variation from that noise, and audit the human scoring against the recordings. Training length was fixed within each recipe, so the amount of data and the number of times it was seen change together; we state which conclusions survive that confound and which do not.

1 Introduction

Low-cost imitation-learning practitioners must choose both a policy and a demonstration budget, but prior comparisons do not settle those choices for this setting. This study compares four policies across nested data budgets and states the limits of its claims.

1 Introduction

  • Four policies were trained on nested subsets of teleoperated demonstrations and tested across fixed object placements on one physical pick-and-place task.The study used several training seeds and scored 1,340 episodes in 126 blocks.Source: §1 · pp. 1–2
  • The scaling comparison spans 12–200 demonstrations and includes a reversal in order between ACT trained from scratch and fine-tuned π0.5.The study presents success curves with confidence intervals and a trend estimate for each policy.Source: §1 · p. 2
  • A four-stage ladder based on joint and wrist-camera recordings tracks where episodes fail and how failure stages shift with data.The stages are used to locate where each policy loses episodes.Source: §1 · p. 2
  • Removing the idle lead-in from Diffusion Policy’s training demonstrations removed its initiation failure in an intervention.Two of the four policies often failed to start moving.Source: §1 · p. 2
  • For π0.5, two fine-tuning recipes yielded 4% and 65% success across the same data budgets, a larger effect than data amount for every policy except ACT.The recipes differed in which weights were trained.Source: §1 · p. 2
  • Repeated checkpoint tests, seed-to-seed variation, and an audit of operator scoring against recordings characterize uncertainty in ten-episode blocks.The study reports these measurements as contributions to evaluation practice.Source: §1 · p. 2

1.1 Scope of claims

  • Claims are limited to one task, arm, and operator and to fixed practitioner-style recipes; fixed training steps confound data budget with training exposure.Testing was dimmer than training, and the study cannot distinguish a high-data ceiling from under-training where curves flatten or fall.Source: §1.1 · p. 2

2 Related work

Prior work studies these policies across varied models, datasets, tasks, and evaluation practices, leaving limited evidence from repeated comparisons on one physical task. This paper situates its study against work on policy design, scaling, idle actions, and evaluation.

2 Related work

3 Experimental setup

The study compares four policies on one physical pick-and-place task using fixed nested demonstration subsets and shared test placements. It records episodes for stage-based failure analysis and evaluates scoring and measurement uncertainty.

3.1 Hardware and workspace

  • Testing used 117–394 lux in dim light, while demonstrations were recorded at about 500 lux; bright testing used 508–725 lux.All main-comparison blocks ran in dim light, a planned robustness test.Source: §3.1 · p. 4

3.2 Task and placements

  • Models picked a green ball and placed it in a basket among two red distractors; success required delivery within 30 seconds on one of ten shared placements.ACT and Diffusion Policy received no language input; SmolVLA and π0.5 received the same instruction at training and test.Source: §3.2 · pp. 4–5, Figure 1 · p. 5

3.3 Demonstrations, subsets and trimming

  • The 200 training demonstrations formed fixed nested subsets of 12, 25, 50, 100, and 200, while 25 others were held out.Diffusion Policy’s trimmed demonstrations removed a median 2.8-second idle lead-in, or 17.4% of frames; the other policies used the original pool.Source: §3.3 · p. 5

3.4 Policies and training recipes

  • All policies used fixed recipes across data budgets; π0.5 was tested with LoRA and expert-only fine-tuning, which train different sets of weights.The π0.5 recipes trained 6,000 steps; LoRA changed about one weight in 1,500, while expert-only training updated about 430 million weights.Source: §3.4 · pp. 6–7

3.5 Evaluation protocol

  • Each block tested one checkpoint once on each of ten placements, with 30 seconds per episode and a 20-second reset.Policies ran through LeRobot at a target 30 Hz with inference inside the control loop; the main comparison included 1,070 episodes in 97 blocks from 86 models.Source: §3.5 · p. 8

3.6 Scoring, audit and automatic measures

  • Joint and camera recordings placed episodes on a cumulative ladder: started, reached the ball, carried it for at least two seconds, and delivered it as the scored success.Grasp failures were divided into location failures, where the ball was never between the jaws at a close, and grip failures, where it was caught and then lost.Source: §3.6 · pp. 8–9
  • The operator’s scoring was checked against recordings for every episode; 27 of 1,340 scores (2.0%) were contradicted, and audited results were used throughout.The audit changed the success count from 473 to 463.Source: §3.6 · pp. 8–9

3.7 Statistics

  • Success rates used Wilson intervals, and trends used logistic regression on log2 demonstration count; a ten-episode cell has an interval of about ±15 percentage points.The study did not correct for multiple comparisons, so p-values are descriptive.Source: §3.7 · pp. 9–10

4 Results

Across the four working recipes, ACT’s success rose sharply with more demonstrations, while fine-tuned π0.5 started strong and plateaued; the leading recipes’ order reversed between 50 and 100 demonstrations. Stage-by-stage recordings show where failures occur, while initiation problems and training recipe changes qualify the raw success curves.

4.1 Success against number of demonstrations

  • 25 of 50 Diffusion Policy episodes succeeded at 50 demonstrations, while SmolVLA succeeded in 21 of 40; both peaked near 50 demonstrations and later fell back.Diffusion Policy’s dip at 100 demonstrations came mostly from one seed scoring 0 of 20; SmolVLA’s dip reflected changes in how often the arm started.Source: §4.1 · p. 10
  • ACT rose from 1 of 40 successes at 12 demonstrations to 37 of 40 at 100, while π0.5 rose from 16 of 40 at 12 to 28 of 40 at 25 and then stayed near 70%.ACT was unchanged at 200 demonstrations (35 of 40); π0.5 recorded 27, 30, and 29 of 40 at 50, 100, and 200. The leading recipes’ order reversed between 50 and 100 demonstrations.Source: §4.1 · pp. 10–11, Figure 2 · p. 11, Table 6 · p. 11
  • 3 of 110 untrimmed Diffusion Policy episodes and 4 of 110 π0.5 LoRA episodes succeeded, with no trend in either recipe across data budgets.Source: §4.1 · p. 11

4.2 Where episodes fail

  • 59% to 89% of started episodes reached the ball across the working recipes, while 69% to 77% of reaches became carries and 89% to 97% of carries were delivered.ACT’s failures shifted from missing the ball at 12 demonstrations, to grip failures at 25 and 50, to nearly every carried ball being delivered at 100 and 200. For expert-only π0.5, 23 of 160 episodes ended in grip failure and 12 with the ball knocked away, versus 6 location failures.Source: §4.2 · pp. 12–13

4.3 Arms that never start

  • 106 of 110 episodes with untrimmed Diffusion Policy never started, compared with 15 of 150 after trimming the idle lead-in; SmolVLA never started in 94 of 200 episodes.A simultaneously tested untrimmed control failed to start in 9 of 10 episodes; the authors’ simplest reading is that the policy learned to stay at the start pose from the demonstrations’ opening frames. SmolVLA’s success among started episodes held at 61% to 71% from 50 to 200 demonstrations.Source: §4.3 · pp. 13–14, Figure 4 · p. 14

4.4 Fine-tuning recipe and EMA

  • 4 of 110 π0.5 LoRA episodes succeeded, versus 130 of 200 with the action expert trained in full and the vision-language model frozen.The LoRA recipe trained 2.7 million weights (0.07% of the model), while the expert-only recipe trained 693 million (16.7%); pilots suggested LoRA may have been under-fitted, but the recipe was not tuned further. EMA made no measurable difference in the reported comparisons.Source: §4.4 · pp. 14–16

4.5 Seed and re-test variability

  • 2.0 successes out of 10 was the average difference between repeat tests of the same checkpoint, and two blocks had to differ by about six successes to distinguish their results from chance.Seed variation was detectable for SmolVLA, mainly through whether the arm started; the study could not rule out seed effects for ACT or π0.5 at its resolution. Diffusion Policy’s dip at 100 demonstrations could not be separated from seed and run variability.Source: §4.5 · pp. 16–18

4.6 Control behaviour

  • 0.17 seconds was π0.5’s action-chunk stall, versus 0.01 seconds for ACT; the study did not test whether these stalls affect success.Successful SmolVLA episodes lasted a median 22.0 seconds, versus 16.5 to 17.1 seconds for ACT, π0.5, and Diffusion Policy, mostly because SmolVLA started late.Source: §4.6 · pp. 18–19

5 Training exposure: the fixed-step confound

Because training steps stayed fixed as data grew, each recipe saw more demonstrations but fewer passes over each frame, confounding data quantity with training exposure. The authors distinguish conclusions that hold under this confound from those that remain unresolved.

5.1 How many times each model saw its data

  • Each doubling of data halves passes within a recipe, so these curves cannot separate the effect of more data from less training per example.Training steps stayed fixed at every budget; Table 14 reports passes over the training data.Source: §5.1 · p. 19, §5 · p. 19, Table 14 · p. 20

5.2 What the robot data say about over- and under-training

  • ACT's success is flat from 100 to 200 demonstrations, at 37 and 35 of 40, leaving a ceiling near 90% and mild under-training both plausible.At 200 demonstrations each frame was seen about eight times; first-grasp accuracy was slightly worse while chunk-to-chunk agreement improved.Source: §5.2 · p. 19
  • For expert-only π0.5, nothing measured improves from 25 to 200 demonstrations, but sparse exposure leaves under-training and a grasp-execution ceiling indistinguishable.At 200 demonstrations, each frame was seen about twice, and the schedule was set using the LoRA recipe's loss curve.Source: §5.2 · p. 19
  • SmolVLA's success among episodes that start rises through 50 demonstrations and holds to 200; its larger-budget manipulation skill shows no decline, while starting remains the limit.Source: §5.2 · p. 19
  • For trimmed Diffusion Policy, under-training is plausible given shorter schedules than the original recipe, but its 100-to-200 dip falls within seed variability.The 100- and 200-demonstration models had a quarter and an eighth of the original recipe's roughly 600 passes.Source: §5.2 · p. 20

5.3 What can and cannot be concluded

  • Below 50 demonstrations, fine-tuned π0.5 far outperforms policies trained from scratch, and ACT needs 50 to 100 demonstrations under its default recipe.The authors identify both findings as surviving the exposure confound.Source: §5.3 · p. 20
  • The initiation failure and LoRA failure occur at every budget and are attributed to the data and recipe, respectively.Source: §5.3 · p. 20
  • The study does not conclude that scaling stops helping π0.5 after 25 demonstrations or ACT after 100, nor that ACT beats π0.5 beyond these recipes.Source: §5.3 · p. 20

6 Discussion

On this task, fine-tuned π0.5 led with few demonstrations, while ACT trained from scratch did at least as well from 100; initiation failures, recipe choice, and noisy small-block measurements qualify the comparison.

Policy and data scaling

  • At 12 demonstrations, π0.5 reached the ball in 27 of 40 episodes, versus 6 of 60 for ACT; from 100 demonstrations, ACT was at least as successful.The study also reports that ACT started more reliably and held a higher control rate at the larger budget.Source: §6 · pp. 20–21

Interpretation and open question

  • Earlier low-cost-arm comparisons at one data size or with fewer trials found pretrained models ahead, whereas this study suggests the comparison can depend on data budget.The authors identify why π0.5 stops improving as an open question: possible under-training, limits from the frozen vision-language model, or physical limits, with ACT’s roughly 90% success suggesting the last is not binding.Source: §6 · p. 21

Initiation failure

  • Trimming Diffusion Policy’s idle lead-in raised its start rate from 4% to 90%, while the account based on prediction horizon does not explain why π0.5 never froze.The demonstrations began with a median 2.8 seconds of no movement; the proposed horizon explanation fits three policies but not π0.5.Source: §6 · p. 21

Recipe sensitivity

  • The two π0.5 fine-tuning recipes differed by 61 percentage points, more than the data-budget difference for every policy except ACT.The failed recipe trained 0.07% of model weights, compared with 16.7% for the working recipe; the authors argue negative results should be read with the recipe in view.Source: §6 · p. 21

Measurement

  • Retests of the same checkpoint moved by 2 successes in 10 on average, and 40-episode cells resolve differences of about 25 points.Staged outcomes and continuous measures added information without extra bench time, while the scoring audit found errors in 2% of episodes.Source: §6 · p. 21

Practical guidance

  • For a new task on this arm class, the authors suggest fine-tuning a pretrained policy below about 50 demonstrations and using ACT from scratch as a baseline at 100 or more.They recommend training the action expert rather than low-rank adapters unless that adapter recipe has been tuned, removing idle frames or recording in motion, and logging whether and when the arm starts; they caution against comparing models on only ten episodes each unless the expected difference is very large.Source: §6 · p. 21

7 Limitations

The limitations bound the findings to a single, controlled task and fixed recipes, with additional uncertainty from training exposure, uneven seeds, lighting, scoring, and evaluation constraints.

Scope

  • The study covers one pick-and-place task, one arm, one operator, and ten placements within the demonstration area; it does not test generalization to new objects, positions, or scenes.The authors therefore frame the practical guidance as starting points from one task.Source: §7 · p. 22

Training design

  • Because training length was fixed within each recipe, data budget and training exposure are confounded, the most important limit on high-data results.Only one recipe per policy was used without hyperparameter tuning; π0.5’s schedule was short relative to published configurations, and Diffusion Policy’s horizon differed from the original paper and library default.Source: §7 · p. 22

Sampling

  • Seed counts were few and unequal, and supplementary probes used only 10 to 80 episodes each.ACT and SmolVLA had four seeds per budget; trimmed Diffusion Policy had two, and π0.5 expert-only models pooled three models from two seeds; the LoRA grid stopped early and half the untrimmed Diffusion Policy models were untested.Source: §7 · p. 22

Lighting

  • Policies ran on different days and under different lighting, leaving the ACT–π0.5 comparison with an unmeasured lighting difference that, if anything, disadvantages π0.5.Testing was also roughly half as bright as training; lighting effects were measured only on one ACT checkpoint and not measured for π0.5.Source: §7 · p. 22

Measurement

  • Scoring was unblinded, and the audit against recordings corrects only detectable errors; the split of grasp failures into location and grip relies on unvalidated color thresholding.The stated audit detects whether the arm moved, whether the ball was held, and whether it left the table.Source: §7 · p. 22

Evaluation

  • Placement comparisons may be affected by two blocks whose ball layouts ran one placement ahead, and no checkpoint selection was done.The 30-second limit turns slow successes into failures; nine episodes ended with the ball in the gripper, and SmolVLA lost three episodes to late starts.Source: §7 · p. 22

8 Future work

Five planned studies target open questions left by this paper, including training-loss validity, the fixed-exposure confound, simulation fidelity, simulated demonstrations, and measurement noise. Smaller checks would test SmolVLA’s initiation failure and Diffusion Policy’s apparent dip.

8.1 Training loss, validation loss and real success

  • Validation loss on 25 held-out demonstrations will be compared with real success across checkpoints, budgets, seeds, and policies to test whether offline loss can guide checkpoint selection without bench time.The plan also calls for running intermediate checkpoints on the robot for a proper test.Source: §8.1 · p. 23

8.2 Training with exposure held constant

  • Retraining with the number of data passes held constant would isolate data amount and test whether the ACT and π0.5 plateaus reflect ceilings or under-training.Training steps would grow with the data budget, making training cost a function of that budget.Source: §8.2 · p. 23

8.3 Simulated against physical success rates

  • A simulated rig will run the 86 main-comparison checkpoints on the same placements, with simulated success and failure stages compared against physical results.Agreement could support using simulation for hundreds of episodes per checkpoint.Source: §8.3 · p. 23

8.4 Simulated reinforcement learning against teleoperated demonstrations

  • The study will compare physical-arm results for policies trained on simulated reinforcement-learning episodes against policies trained on teleoperated demonstrations.The authors plan to extend scaling curves beyond 200 episodes and test whether machine-generated demonstrations avoid the idle lead-in.Source: §8.4 · p. 23

8.5 Automating the test bench

  • An automated bench would detect outcomes and reset objects for unattended, blind-by-construction runs, aiming for about 100 episodes per checkpoint.The plan is to test how much of the measurement noise can be reduced by replacing operator-dependent ten-episode blocks.Source: §8.5 · p. 23
  • Smaller checks would test whether trimmed demonstrations also fix SmolVLA’s initiation failure and whether Diffusion Policy’s dip persists with two more seeds at 50, 100, and 200 demonstrations.The SmolVLA check is framed as testing whether its initiation failure has the same cause as Diffusion Policy’s.Source: §8.5 · p. 24

9 Conclusion

On one pick-and-place task, the leading policy depended on the demonstration budget: fine-tuned π0.5 led with a few dozen demonstrations, while ACT from scratch did at least as well with 100 or more. These conclusions use fixed, near-default recipes, leaving open whether high-data plateaus reflect ceilings or fixed training length.

Conclusion

  • With a few dozen demonstrations, fine-tuned π0.5 was far ahead; with 100 or more, ACT trained from scratch did at least as well.The conclusion also reports that removing an idle lead-in cured a start failure in Diffusion Policy, while the largest model’s success shifted from 4% to 65% depending on which weights were trained.Source: §9 · p. 24
  • The results apply to fixed, near-default recipes, and whether high-data plateaus are ceilings or reflect fixed training length remains unresolved.Recordings supported failure staging, checkpoint retests, and scoring audits; the authors suggest making these routine in real-robot studies of this size.Source: §9 · p. 24

A Results by seed

ACT’s block results vary by seed and budget, while the untrimmed Diffusion Policy and π0.5 LoRA runs are mostly unsuccessful. The table reports successes out of ten for each main-comparison block.

A Results by seed

  • Repeated checkpoint runs are listed in order with arrows, so block outcomes can be compared across runs.Source: A Results by seed · p. 26, Table 15 · p. 27
  • Across ACT seeds, the listed results include a low of 0/10 and a high of 10/10; ACT rows use the first ten placements.Table 15 reports successes out of ten for every block, by seed.Source: A Results by seed · p. 26, Table 15 · p. 27
  • Across the listed blocks, untrimmed Diffusion Policy and π0.5 LoRA runs scored zero or one success in nearly all cases.The displayed block scores for these recipes remain low across the 12-to-200 demonstration budgets.Source: A Results by seed · p. 26

B Changes made by the scoring audit

The scoring audit changed 27 of 28 reviewed episodes, while the recording-derived measures operationalize movement, ball contact, carrying, and control timing. Supplementary blocks add 270 episodes, with lighting varying across conditions.

B Changes made by the scoring audit

C Measures taken from the recordings

  • An episode counted as started when any arm joint moved at least 60° from its first-frame position; reaching required the ball between the jaws around a close, and carrying required two seconds there.The operator’s start label agreed in 280 of 282 cases.Source: C Measures taken from the recordings · p. 28
  • Ball detection used fixed color thresholds, while first-grasp assessment combined ball position across frames with apparent image size rather than miss distance alone.The ball-between-jaws measure also required proximity to a fixed point between the jaws and an area within the range observed during a hold.Source: C Measures taken from the recordings · p. 28
  • Chunk jump measures the joint-angle change across chunk boundaries, and pause per chunk divides the shortfall from 30 Hz by chunks executed.The control rate is estimated from frame count over a full 30-second episode.Source: C Measures taken from the recordings · p. 28

D Supplementary blocks

  • The supplementary set contains 270 episodes, and its bright-light blocks ran at 508–725 lux versus dimmer EMA blocks at 172–190 lux.The compared trimmed and untrimmed cells were measured at 215–366 lux and 243–278 lux, respectively.Source: D Supplementary blocks · p. 29

E Results by placement

Placement success was pooled across budgets, with placement effects detected for the two pretrained policies but not detectably for the policies trained from scratch. For SmolVLA and Diffusion Policy, the placement counts include only episodes in which the arm started.

E Results by placement

  • Pooled over budgets, placement matters for the two pretrained policies but was not detectably associated with outcomes for the two policies trained from scratch.Pooling may hide placement effects for ACT; a within-block permutation test gave p between 0.03 and 0.07 depending on the statistic.Source: E Results by placement · p. 29, Table 18 · p. 30
  • N03 is the weak placement for π0.5, and placement-level counts for SmolVLA and Diffusion Policy exclude episodes in which the arm did not start.Source: E Results by placement · p. 29, Table 18 · p. 30

F Training configurations

The training configurations specify deviations from library defaults and the parameter counts for π0.5’s adaptation recipes. The π0.5 schedule was shortened before robot testing based on a LoRA pilot, and its length was not revisited for expert-only fine-tuning.

F Training configurations

  • Settings that differed from library defaults were stated, and training ran in a Hugging Face Jobs container pinned by digest.Run times did not vary by data budget because training steps were fixed.Source: F Training configurations · p. 30, Table 19 · p. 31
  • π0.5 has 4,143,404,816 total weights; LoRA trains 2,705,408 weights, compared with 693,422,112 for expert-only fine-tuning.Both trainable-weight totals match the training logs.Source: F Training configurations · p. 30, Table 19 · p. 31
  • The π0.5 schedule was reduced from 30,000 to 12,000 and then 6,000 steps before robot testing, based on a 100-demonstration LoRA pilot with flat loss after about 4,000 steps.The 6,000-step choice was motivated by training time, at 6.3 seconds per step, and was not re-examined for expert-only fine-tuning.Source: F Training configurations · p. 30, Table 19 · p. 31

Related papers

Loading 2610.37170v1…