Source-linked AI summary
JitterVLA: A Replay Benchmark for Motion Quality in VLA-Driven Human-Robot Manipulation
Faith Ogunwoye, Didar Seyidov, Konstantin Gubernatorov, Dzmitry Tsetserukou
TL;DR
VLA targets arrive sparsely and irregularly, while remedies are evaluated on different streams, delays, and metrics. JitterVLA replays identical commands and arrival traces through alternative controllers. Interpolation reduced modeled p99 jerk across sources, with a tracking cost, and its effect also appeared in an exploratory UR3 check.
Problem
VLA targets arrive sparsely and irregularly, and existing remedies are evaluated on different streams, delays, and metrics.
Source: §1 · pp. 1–2Method
JitterVLA replays identical command sources and delay realizations through alternative controllers to compare their executed-motion effects.
Source: §1 · p. 2Results
Interpolation cut modeled p99 jerk for every source by 83% on human motion with 200 ms mean delay, at a 0.3–2.4 cm tracking cost, and replicated on 598 new episodes.
Source: §6 · p. 4Takeaways & Limitations
JitterVLA supports screening controller choices before human-centred studies.
Source: §6 · p. 4Main results use one generic point-mass plant and open-loop VLA predictions; closed-loop behaviour, task success, and robot-specific dynamics are outside scope.
Source: §6 · p. 4
Abstract
from arXiv · showhide
Vision-language-action (VLA) policies issue sparse targets whose arrival at the robot controller is delayed and irregular, so a reasonable predicted action can still produce abrupt motion near people. Remedies are evaluated on different streams, delays and metrics. JitterVLA replays identical commands and arrival traces through alternative controllers, isolating the controller's effect. We apply it to synthetic reaches, hand trajectories from 20 people, three open VLA pipelines (838 episodes) and a physical UR3. Interpolation lowered modeled p99 jerk for every source, by 83% (615 to 106 m/s^3) on human motion with 200 ms delay, at a 2.0 cm tracking cost, and the effect replicated on 598 new episodes; jerk-limited generation lowered jerk further at a larger cost. Entropy-discontinuity associations had opposite, replicated signs in two pipelines, and entropy gating helped in one but not the other. On the UR3, interpolation lowered accelerometer-derived jerk by 3.0-14.0% in all six exploratory pairs, while a position-derived metric did not resolve the contrast. JitterVLA supports screening before human-centred studies; code and data will be released.
1 Introduction
Sparse, irregular VLA targets make executed motion—not the predicted action—the relevant object for motion-quality evaluation. JitterVLA compares controller effects through paired replay before candidate behaviours are shown to people.
Motivation
- Because VLA targets arrive sparsely and irregularly, a sensible predicted action can still become discontinuous goals at the low-level controller.People experience the robot’s executed motion, not its predicted action.Source: §1 · p. 1
Comparison problem
- Remedies are hard to compare fairly because prior work evaluates them on different streams, delays, plants, and metrics.JitterVLA evaluates the command-stream-to-executed-motion chain before candidate behaviours are shown to people; it does not replace human studies.Source: §1 · pp. 1–2
Research questions
- The four research questions test timing effects, controller trade-offs, cross-source replication, and whether contrasts appear on a physical UR3.Source: §1 · p. 2
Contributions
- The benchmark pairs identical command sources and delay realizations while comparing interpolation trade-offs, representation-dependent uncertainty associations, and entropy-gating failures.Evaluation spans synthetic reaches, motion from 20 people, three VLA pipelines, new episode draws, and a bounded UR3 check.Source: §1 · p. 2
Terms and scope
- Jerk, tracking error, LDLJ, and SPARC are motion-quality proxies; comfort, preference, and safety were not measured.Source: §1 · p. 2
2 Benchmark design
The benchmark replays matched command sources and delays through alternative controllers on a generic plant, with multiple timing regimes, motion-quality metrics, and frozen or exploratory evidence labels.
Paired replay
- Paired replay gives each controller the same source and delay realization, so differences are attributable to the controller rather than different stochastic inputs.Statistics are paired by source unit, with 95% unit-bootstrap intervals based on 5,000 resamples.Source: §2 · p. 2
Timing pipeline
- VLA targets pass through measured inference times and a FIFO queue before zero, Gamma-distributed, or measured network delays are added.Delivery remains in order, and the newest delivered target wins.Source: §2 · p. 2
Plant and controllers
- Delivered targets drive a generic three-axis point-mass plant, not a UR3 model, so its absolute jerk values are not hardware predictions.The plant uses a 3 kg mass, critically damped impedance, 20 ms force lag, a 3000 N/s force-rate cap, and 1 ms steps.Source: §2 · p. 2
- B0 holds each delivered target, B2 interpolates over one command period, and A2 adds entropy-gated stiffness to interpolation.B3 is an explicit jerk-limited trajectory generator with a 25 m/s^3 jerk limit.Source: §2 · p. 2
Evaluation scope
- VLA actions are open-loop predictions from recorded observations, not commands executed by the robots in those datasets.The design includes 838 VLA episodes and 69,988 paired replays overall.Source: §2 · p. 2, Table 1 · p. 2
Controller comparisons and metrics
- A constant-gain grid tests whether gating beats the best attainable fixed stiffness at no worse tracking, while B3 tests stronger motion generation.The benchmark reports p99 and RMS jerk and tracking RMSE, plus LDLJ and SPARC as additional smoothness checks.Source: §2 · p. 2
Evidence hierarchy
- Protocols were frozen before data were seen, and planned replication, post-outcome robustness checks, and exploratory hardware results remain separately labelled.Source: §2 · p. 2
3 Results
Across synthetic, human, and VLA sources, interpolation consistently trades lower modeled jerk for higher tracking error, while entropy-based results vary by pipeline. A bounded UR3 follow-up also found lower accelerometer-derived jerk, though the planned position-derived contrast was unresolved.
R1: interpolation trades jerk for tracking on every source
- Interpolation lowered modeled p99 jerk and raised tracking error for synthetic reaches, human motion, and all three VLA sources, with the effect holding on new draws.Under measured network traces, the B2−B0 jerk direction also held for every source.Source: §3 · p. 2, Table 2 · p. 3
R2: lower jerk is not free
- On human motion at 200 ms mean delay, B0’s 615 m/s3 p99 jerk falls to B2’s 106 m/s3 as tracking error rises from 9.91 to 11.87 cm.B3 reached 23.5 m/s3 at 14.79 cm; A2 yielded 150 m/s3 at 16.72 cm and was worse than B2 on both axes.Source: §3 · pp. 2–3, Figure 2 · p. 3
R3: uncertainty is representation-dependent
- OpenVLA entropy was weakly negatively associated with successive goal steps, whereas pi0-FAST entropy was positively associated with chunk-boundary jumps; both signs replicated.The panels use different representations and events, so their signs are not a model comparison.Source: Figure 3 · p. 3, §3 · p. 3
- Entropy-gated A2 lowered pi0-FAST p99 jerk versus B2 by 208 m/s3 for +0.18 cm tracking, but did not help OpenVLA (+2.3 [−0.7, 5.2]).All 28 pre-registered pi0-FAST contrasts and 22 of 28 OpenVLA contrasts replicated.Source: §3 · p. 3
R4: a bounded physical check
- On the UR3, exploratory accelerometer-derived jerk fell 3.0–14.0% in all six pairs, while the planned position-derived p99 filtered-jerk contrast was unresolved.The result is not a calibrated tool-centre-point jerk or safety result; sensor gain, bandwidth, and the source of the moving signal were not established.Source: §3 · p. 3
4 Code structure, usage and maintenance
The code is organized into replay, benchmark, command-line, and test components, with a command-line interface for replaying user-provided streams. The section also identifies the scope of replication reporting and a bounded physical check.
Code structure and usage
- The code separates replay mechanics, benchmark protocols and analysis, a command-line tool, and 34 unit tests.The replay command runs time-stamped CSV targets or stored VLA streams under the same arrivals and prints each condition.Source: §4 · p. 3
Maintenance
- Code changes affecting controller semantics or results are released as new protocol versions rather than silently replacing frozen results.The code is planned for release under Apache-2.0 after the review period.Source: §4 · p. 3
Physical check
- The exploratory UR3 figure reports positive B0−B2 differences in p99 tool-accelerometer-derived jerk across six paired repeats.Three repeats per delay are descriptive, not a significance test.Source: Figure 4 · p. 4
Replication reporting
- Table 3 reports replication on new, disjoint episode draws, with 95% unit bootstrap intervals.The accompanying note marks feasible-subset results as post hoc for the original set and pi0.5 as a post-outcome addition.Source: Table 3 · p. 4, §4 · p. 4
5 Availability and ethics
The release plan covers code, tests, protocols, transformed VLA streams, hand trajectories, and latency traces. The paper reports attribution and ethics context for the hand data and states that no human-subject study was performed.
Availability
- The planned release includes code, tests, a CPU demo, frozen protocols, transformed VLA streams, hand windows, and latency traces.Reported tables can be regenerated on a CPU without model weights or a robot.Source: §5 · p. 4
Data and ethics
- The hand trajectories are a secondary analysis of public, de-identified 3D-ARM-Gaze data, redistributed with attribution and a statement of changes.Consent and ethics approval are reported by the dataset authors.Source: §5 · p. 4
- The UR3 teleoperation episodes were recorded by an author, and no human-subject study was performed.The episodes are identified as available on Hugging Face.Source: §5 · p. 4
6 Limitations and conclusion
Across the reported sources, interpolation reduced modeled jerk at a tracking cost, while entropy gating helped on pi0-FAST but not OpenVLA. The authors bound these conclusions by the open-loop point-mass evaluation and the limited, proxy-based hardware and human-interaction evidence.
Limitations
- The main results use a generic point-mass plant with open-loop VLA predictions, leaving closed-loop behavior, task success, and robot-specific dynamics outside scope.The three pipelines also differ jointly in model, dataset, action representation, entropy definition, and discontinuity event, so they are not a controlled model comparison.Source: §6 · p. 4
- The UR3 study is small and exploratory, and comfort, preference, trust, and perceived safety were not measured.The hardware work uses one session each, an uncalibrated accelerometer, and a follow-up designed after the v1.2 null result; the reported metrics are motion-quality proxies.Source: §6 · p. 4
Conclusion
- 83% lower modeled p99 jerk on human motion with 200 ms mean delay came with a 0.3–2.4 cm tracking cost, and the effect replicated on 598 new episodes.Across 69,988 paired replays, interpolation cut modeled p99 jerk for every source; the conclusion also reports 3.0–14.0% lower accelerometer-derived jerk in all six exploratory UR3 pairs.Source: §6 · p. 4