Source-linked AI summary
FrameSkip: Learning from Fewer but More Informative Frames in VLA Training
Bin Yu, Shijie Lian, Xiaopeng Lin, Zhaolong Shen, Yuliang Wei, Changti Wu, Hang Yuan, Haishan Liu, Bailing Wang, Cong Huang, Kai Chen
TL;DR
Dense VLA demonstrations treat every frame as equally useful, despite sparse manipulation-critical transitions. FrameSkip reallocates training toward informative frames and raises macro-average success from 66.50% to 76.15% across three benchmarks while retaining 20% of unique frames.
Problem
Dense VLA demonstrations sample frames as equally useful, despite higher failure rates at sparse manipulation-critical transitions such as alignment, grasping, and release.
Method
FrameSkip is an architecture-agnostic data-layer framework that scores frames using trajectory cues and gripper-transition preservation, then selects informative frames under a retention budget.
Results
76.15% macro-average success versus 66.50% for full-frame training across RoboCasa-GR1, SimplerEnv, and LIBERO, retaining 20% of unique frames.
Takeaways & Limitations
Frame-level supervision allocation is a practical lever for VLA training under a compressed trajectory view.
Takeaways & Limitations
Without stage annotations, FrameSkip uses a Gaussian prior assuming manipulation-critical stages are more likely near the trajectory middle.
Abstract
from arXiv · showhide
Vision-Language-Action (VLA) policies are commonly trained from dense robot demonstration trajectories, often collected through teleoperation, by sampling every recorded frame as if it provided equally useful supervision. We argue that this convention creates a temporal supervision imbalance: long low-change segments dominate the training stream, while manipulation-critical transitions such as alignment, contact, grasping, and release appear only sparsely. We introduce FrameSkip, a data-layer frame selection framework that scores trajectory frames using action variation, visual-action coherence, task-progress priors, and gripper-transition preservation, then remaps training samples toward high-importance frames under a target retention ratio. Because FrameSkip operates only in the dataloader, it leaves the VLA architecture, action head, training objective, and inference procedure unchanged. Across RoboCasa-GR1, SimplerEnv, and LIBERO, FrameSkip improves the success-retention trade-off over full-frame training and simpler frame selection variants, achieving a macro-average success rate of 76.15% across the three benchmarks compared with 66.50% for full-frame training while using a compressed trajectory view that retains 20% of unique frames in the main setting.
1 Introduction
Dense robot demonstrations create a temporal supervision imbalance: abundant low-change frames can dilute sparse, manipulation-critical transitions where failures concentrate. FrameSkip addresses this by reallocating frame-level supervision toward informative frames through an architecture-agnostic data-layer selection framework.
- Motivation: Dense teleoperated trajectories contain long low-change segments, while alignment, contact, grasp closure, and release occupy only a small fraction of recordings.This temporal structure makes uniform frame sampling poorly matched to the moments that define task outcomes.
- Problem: Uniform sampling creates temporal supervision imbalance because rare decision-critical transitions are diluted by abundant, weakly informative observations under a fixed optimization budget.The issue is framed as how supervision is distributed across frames, not merely as data-volume reduction.
- Motivation: Failures concentrate in sparse interaction stages such as alignment, grasping, and release, whereas routine approach and return stages are often handled reliably.This pattern suggests policies adapt globally to dominant smooth motions while remaining brittle at sparse manipulation-critical transitions.
- Method: FrameSkip scores frames using action variation, visual-action coherence, task-progress priors, and gripper-transition preservation, then remaps training toward retained high-importance frames.It constructs compressed trajectory views under target retention ratios without modifying the VLA architecture, action head, loss function, or inference procedure.
- Evaluation: The evaluation compares full-frame, random, action-variation-only, and progressively stronger importance-based selection across RoboCasa-GR1, SimplerEnv, and LIBERO.The study treats frame selection as a success-retention trade-off and includes matched-ratio baselines plus ablations over retention ratios, importance metrics, and warmup schedules.
2 Related Work
Prior VLA research advances unified vision-language-action policies through stronger models, action representations, action heads, and broader data, while generally retaining original temporal density. FrameSkip is positioned as complementary to these methods and distinct from coarse dataset reweighting, trajectory filtering, and learned transition curation.
- Vision-language-action models: VLA models unify visual grounding, language conditioning, and action prediction within a single policy interface.The cited work frames VLAs as a unified policy interface combining these three capabilities.
- Vision-language-action models: Recent VLA advances use stronger VLM initialization, action tokenization, diffusion or flow-matching action heads, and large-scale cross-embodiment data.These improvements span model initialization, action representation, action prediction, and training-data scale.
- Vision-language-action models: Existing advances generally consume training data at its original temporal density, whereas FrameSkip asks whether the same VLA families can use fewer, more informative frames.FrameSkip is described as complementary to the existing VLA families rather than as an architectural replacement.
- Data Curation for Robot Learning: Coarse-grained curation reweights datasets or filters trajectories while treating intra-trajectory frames uniformly, whereas Scizor curates transitions with a learned task-progress predictor.The passage contrasts these approaches with FrameSkip’s stated objective and mechanism, though the supplied text ends before fully specifying that mechanism.
3 Method
FrameSkip is a training-time data-layer framework that scores trajectory frames, retains a target-ratio subset, and remaps dataloader indices without changing the VLA model, objective, or inference procedure. Its scoring combines action dynamics, visual-action coherence, task-progress priors, and gripper-transition preservation to prioritize informative manipulation events.
- Data-layer intervention: FrameSkip precomputes important frame indices and remaps dataset queries during training, leaving the VLA architecture, action head, loss function, and inference procedure unchanged.The original trajectory index space remains unchanged, enabling architecture-agnostic use and compatibility with existing dataset mixtures and sampling weights.
- Retention and pruning: The pruner targets approximately rT frames using an importance quantile, while explicitly retaining trajectory boundaries, gripper transitions, and top-decile action changes when enabled.If quantile selection misses the target count, frames are added or removed by descending importance until the target is met.
- Frame scoring: Frame importance combines action variation, visual-action coherence, and task-progress signals, with AVI dominant and default weights α = 0.6, β = 0.2, and γ = 0.2.All component scores are min-max normalized within each trajectory before combination; VAC uses DINOv2 visual features and is enabled in the main experiments.
- Task-progress prior: Task-progress scoring uses a dataset-adaptive GMM prior over annotated manipulation-stage locations, with a Gaussian fallback requiring no annotations and using σ2 = 0.2.The annotations estimate an offline progress prior only and are not provided to the policy during training or evaluation.
- Event preservation: Gripper-aware scoring multiplies the combined score using absolute gripper or end-effector state changes, helping preserve contact-related stages without adding a model component.If those action dimensions are unavailable, the factor falls back to the action-variation signal captured by AVI.
4 Experiments
Experiments evaluate FrameSkip across three simulation benchmarks under a controlled, unchanged VLA training recipe. FrameSkip raises macro-average success from 66.50% to 76.15% while retaining 20% of unique frames, with ablations examining retention, scoring cues, and warmup.
- Experimental setup: The controlled comparisons keep the VLA architecture, optimizer, and remaining training configuration fixed, using r = 0.2 and a 5:1 pruned-to-anchor mini-batch schedule by default.All experiments use StarVLA with a two-expert architecture and a global batch size of 128.
- Benchmarks: Experiments cover RoboCasa-GR1, SimplerEnv, and LIBERO, spanning bimanual dexterous, WidowX held-out, and Franka language-conditioned manipulation settings.Each benchmark uses training data from its corresponding robot embodiment or the benchmark’s prescribed demonstrations.
- Main results: 76.15% macro-average success versus 66.50% for full-frame training is achieved across the three benchmarks while retaining 20% of unique frames.The result is reported under the same VLA architecture and training recipe, with a compressed trajectory view.
- Effect of retention ratio: Retention-ratio ablations on RoboCasa-GR1 test r ∈ {10%, 20%, 30%, 40%, 50%, 60%, 100%} under the same model and training budget.The 100% setting serves as the full-frame reference, probing the trade-off between trajectory context and concentrated supervision.
- Effect of importance metric: Importance-metric ablations compare random, AVI-only, AVI+TPI, AVI+VAC, AVI+VAC+TPI, and FRAMESKIP Full under the same retention ratio and schedule.FRAMESKIP Full additionally includes gripper-transition preservation, while random serves as a pruning-only control.
- Effect of warmup steps: 5000 warmup steps yield the best RoboCasa-GR1 average success, while changing warmup from 2500 to 15000 steps has only a modest effect.After warmup, variants use the same retention ratio and pruned/full-frame mini-batch schedule.
5 Conclusion
FRAMESKIP is a training-time frame-pruning framework that selects informative frames under a retention budget while leaving the VLA architecture unchanged. Across three benchmarks, it raises macro-average success from 66.50% to 76.15% while retaining 20% of unique frames.
- Contribution: FRAMESKIP is a training-time frame-pruning framework for VLA models that addresses structured temporal redundancy in robot trajectories.It treats frame importance as a practical lever for VLA training.
- Method: It combines action variation, visual-action coherence, task-progress priors, and gripper-transition preservation to select informative frames under a target retention budget.The method leaves the VLA architecture unchanged.
- Results: 76.15% macro-average success rate improves over 66.50% across RoboCasa-GR1, SimplerEnv, and LIBERO while retaining 20% of unique frames.This result uses the main compressed trajectory setting.
A Additional Implementation Details · A.1 Pruning Cache
FrameSkip caches trajectory-level pruning outputs to support reuse across compatible experiments and efficient distributed training. The cache records importance scores and retained indices for each configured pruning ratio.
- A.1 Pruning Cache: FrameSkip stores pruning results at the trajectory level in a cache.The cache contains the original importance scores and retained indices.
- A.1 Pruning Cache: The cache preserves the original importance scores used for pruning.
- A.1 Pruning Cache: The cache stores retained frame indices for each configured pruning ratio.
- A.1 Pruning Cache: Cached pruning outputs can be reused across experiments when importance and pruning configurations remain compatible.
- A.1 Pruning Cache: During distributed training, cache construction can be restricted to rank zero.Other workers load the cache after synchronization.
- A Additional Implementation Details: Other distributed workers load the constructed cache after synchronization.
A.2 Frame-Score Preprocessing
Frame-score preprocessing extracts sparse visual features for VAC and fits benchmark-specific progress priors for GMM-TPI. The pipeline also records preprocessing failures and supports a dataset-agnostic Gaussian prior when annotations are unavailable.
- VAC: VAC extracts DINOv2 visual features from at most 16 sparsely sampled video frames per trajectory before interpolating scores to the original length.The implementation records frame-extraction failures and trajectories without usable visual features to identify unreliable preprocessing runs before training.
- GMM-TPI: GMM-TPI fits a three-component one-dimensional GMM separately for each benchmark using progress annotations from 5% of training trajectories.Annotations contain normalized progress locations for manipulation-critical stage centers, including alignment, grasping, and release, without action, success, or per-frame policy supervision.
- GMM-TPI: When progress annotations are unavailable, GMM-TPI uses the dataset-agnostic Gaussian prior described in Section 3.
A.3 Main Training Schedule
Main experiments train primarily on an 80%-pruned trajectory view, periodically anchoring with full-frame minibatches, while evaluation uses standard unpruned inference. Training budgets vary by benchmark: 100K steps for RoboCasa-GR1, 60K for SimplerEnv, and 30K for LIBERO.
- Main Training Schedule: r = 0.2 retains 20% of frames, and training alternates five pruned-view mini-batches with one full-frame mini-batch at r = 1.0.The full-frame mini-batch serves only as a periodic context anchor.
- Main Training Schedule: Standard policy inference evaluates the model without frame pruning.Frame pruning is used during training, not evaluation.
- Main Training Schedule: 100K optimization steps are used for RoboCasa-GR1, 60K for SimplerEnv, and 30K for LIBERO.The schedules follow each benchmark’s established training budget rather than enforcing one shared step count.
B Full RoboCasa-GR1 Results
Table 7 reports VLA model results evaluated with the GR1 robot in the RoboCasa-GR1 Tabletop simulation environment, including Isaac-GR00T N1.5 and N1.6 results sourced from NVIDIA’s official repository.
- Table 7 evaluates VLA models with the GR1 robot in the RoboCasa-GR1 Tabletop simulation environment.
- Isaac-GR00T N1.5 and N1.6 results come from the official Isaac-GR00T GitHub repository.
- The table marks the best results in bold and the second-best results with underlining.