Hierarchical Skill Retrieval for Data-Efficient Adaptation of Vision-Language-Action Models

Haoran Hao; Shahram Najam Syed; Jeff Schneider; Jeffrey Ichnowski

Source-linked AI summary

Hierarchical Skill Retrieval for Data-Efficient Adaptation of Vision-Language-Action Models

Haoran Hao, Shahram Najam Syed, Jeff Schneider, Jeffrey Ichnowski

arXiv:2608.24042v1Aug 25, 2026cs.ROcs.AIcs.LG

TL;DR

VLA adaptation can be difficult with limited target demonstrations, while existing retrieval methods may miss the hierarchical structure of long-horizon tasks. HSR decomposes tasks into reusable skills, performs hybrid skill-level retrieval, and uses two-stage adaptation; it reports 10.3% and 21.3% success-rate gains over the strongest baseline on LIBERO and real-world tasks, respectively.

Problem
Existing retrieval methods may overlook task hierarchy and language semantics, even though complete long-horizon task matches are rare and reusable skills are abundant.
Method
HSR decomposes target tasks into skill sequences, evaluates candidate plans, combines subtask-language retrieval with behavior-feature reranking, and adapts policies in two stages.
Results
10.3% and 21.3% success-rate improvements over the strongest baseline are reported on LIBERO and real-world tasks, respectively.
Takeaways & Limitations
HSR enables pretrained VLA policies to reuse transferable skills from prior datasets rather than relying on exact task-level matches.
Takeaways & Limitations
The framework is evaluated only on manipulation tasks and a single VLA backbone.

Abstract

from arXiv · show

While Vision-Language-Action (VLA) models pretrained on large-scale robot datasets provide a strong foundation for robot manipulation, their performance can degrade when adapted to new tasks with limited task-specific demonstrations. Retrieval offers a practical way to reuse existing demonstrations for data-efficient adaptation, but existing methods often rely on visual similarity, state-action representations, or task-level language matching. These approaches may overlook the hierarchical structure of long-horizon manipulation tasks, where complete task matches are rare but reusable skills are often abundant. To address this challenge, we propose Hierarchical Skill Retrieval (HSR), a retrieval framework for data-efficient VLA adaptation. Specifically, HSR first decomposes a target task into candidate skill sequences. It evaluates each plan based on both semantic plausibility and skill reliability estimated from the prior dataset. The selected decomposition is then used for hybrid retrieval. This combines subtask-level language retrieval with behavior-feature reranking to identify demonstrations that are both semantically relevant and compatible with the target task. Finally, we adapt the policy through a two-stage pretraining and finetuning pipeline, which separates general skill acquisition from task-specific adaptation. Experiments on the LIBERO benchmark and several real-world robot manipulation tasks show that HSR improves the average success rate by 10.3% and 21.3% over the strongest baseline, respectively. These results demonstrate the effectiveness of structured skill-level retrieval for data-efficient VLA adaptation. Videos and code are available at https://hoar012.github.io/HSR-Project.

I. INTRODUCTION

Existing retrieval methods can miss the hierarchical structure and language semantics of long-horizon tasks, where reusable skills are more common than complete task matches. HSR addresses this with task decomposition, skill-level retrieval, and two-stage adaptation, improving success rates on LIBERO and real-world tasks.

Motivation: Existing retrieval approaches use prior demonstrations but often rely on visual, motion, or state-action similarity.These methods can expand the effective training set for downstream adaptation.
Motivation: Language provides a semantic signal for task generalization, but full-instruction retrieval is difficult for complex long-horizon tasks.Complete task matches are rare, whereas component skills such as grasping a teapot or picking up a tea bag are common.
HSR: HSR decomposes target tasks into skill sequences grounded in reusable prior-dataset skills and retrieves demonstrations using the resulting subtasks.An LLM planner proposes decompositions, which are scored for semantic relevance and support from prior data.
Results: 10.3% and 21.3% success-rate improvements over the strongest baseline are reported on LIBERO and real-world tasks, respectively.The framework achieves its largest gains on long-horizon composite manipulation tasks.
HSR: HSR uses a two-stage adaptation strategy that first acquires transferable skills and then finetunes with target demonstrations and feature-reranked retrieved samples.This separates general skill acquisition from task-specific adaptation.

II. RELATED WORK

VLA policies build on multimodal foundation models but remain difficult to adapt when downstream manipulation data are scarce. Related retrieval methods reuse prior datasets through visual, motion, state-action, or multimodal similarity.

Vision-Language Conditioned Manipulation: Vision-language-action policies extend pretrained vision-language representations to robot control and downstream manipulation.Their broad multimodal knowledge does not eliminate the need for task-specific adaptation.
Vision-Language Conditioned Manipulation: Adapting VLA models remains an open problem when downstream tasks provide insufficient data for training or finetuning.The challenge concerns few-shot or data-scarce downstream manipulation.
Retrieval-Based Adaptation: Prior retrieval methods select data from datasets such as DROID and Open X-Embodiment using learned trajectory representations and similarity.Representations may encode state-action pairs, optical flow, sub-trajectories, or multiple modalities.

C. Robot Task Planning

Robot task-planning work decomposes complex tasks into reusable skills, supporting generalization and reducing the data needed for composite tasks. HSR identifies semantically relevant skill sequences and retrieves demonstrations supporting those skills for policy adaptation.

C. Robot Task Planning: Task planning combines symbolic reasoning, motion planning, and learning-based methods to decompose and schedule complex tasks.Effective planning can reuse existing skills for zero-shot generalization to novel tasks.
C. Robot Task Planning: HSR identifies a sequence of semantically relevant skills and retrieves training demonstrations that support each skill.This connects task-level planning with data-efficient policy adaptation.
C. Robot Task Planning: The adaptation objective combines a small target demonstration set with a useful subset of a diverse prior dataset.The selected prior demonstrations are intended to improve expert-behavior matching and target-task performance.

A. Hierarchical Task Decomposition

HSR decomposes long-horizon tasks into candidate skill sequences and selects a plan using semantic plausibility and skill reliability. Reliability is estimated from held-out behavior cloning loss and trajectory length without costly rollouts.

Skill Reliability Estimation: HSR uses held-out BC loss as a lightweight, rollout-free reliability proxy because real-world rollouts are computationally expensive.
Skill Reliability Estimation: Skill reliability is estimated by finetuning a pretrained VLA model on each skill cluster’s training split and measuring validation BC loss.
Skill Reliability Estimation: The Skill Confidence Score combines normalized validation loss and average trajectory length, with fixed weights α = 0.6 and β = 0.8.
Plan Generation and Selection: HSR prompts an LLM to generate candidate subtask plans grounded in reusable skill clusters, then selects the highest-scoring plan for retrieval.
Plan Generation and Selection: Candidate plans combine an LLM-derived semantic score with confidence scores for their constituent skills.

B. Hybrid Skill Retrieval

HSR retrieves demonstrations hierarchically by using subtask language similarity first and behavior-feature compatibility second. This addresses the mismatch between semantically similar demonstrations and differing execution contexts.

Hybrid Retrieval: The framework filters low-consistency data by combining semantic relevance at the episode level with behavior compatibility at the frame level.
Hybrid Retrieval: HSR first retrieves episodes using subtask language similarity, then reranks candidate frames using behavior features from target-task demonstrations.
Hybrid Retrieval: Behavior-feature reranking retains frames whose latent representations are closest to target frames.

C. Policy Adaptation

HSR separates reusable skill acquisition from task-specific adaptation through two-stage policy training. It first pretrains on language-retrieved episodes and then finetunes on target data plus reranked frames.

Two-Stage Adaptation: HSR pretrains the policy on language-retrieved episodes to acquire reusable behaviors before task-specific finetuning.
Two-Stage Adaptation: The second stage finetunes on Dt ∪ Drerank, rather than jointly co-training on all retrieved and target data.
Two-Stage Adaptation: The two-stage design avoids treating all retrieved data as equally informative.

V. EXPERIMENTS

The experiments evaluate HSR against target-only training, data-mixing, language-based, optical-flow, subtrajectory, and state-action retrieval baselines in simulation and real-world settings. The setup uses limited target demonstrations and prior datasets, with retrieval and adaptation procedures tailored to each environment.

Models and Training: HSR trains a pretrained SmolVLA-450M policy while updating only its approximately 100M-parameter action expert.Qwen3-VL-4B generates candidate task-decomposition plans, and the highest-scoring plan is selected.
Evaluation Settings: HSR is evaluated in both the LIBERO simulation benchmark and real-world manipulation settings.The real-world experiments use a 7-DoF xArm, while simulation uses LIBERO.
Simulation Setup: LIBERO experiments use 5 target demonstrations, LIBERO-90 as the prior dataset, and two-stage retrieval retaining approximately 3% of prior frames.The first stage retrieves the top 10% of episodes, followed by reranking the top 30% of those candidates.
Real-World Setup: Real-world experiments use 20 target demonstrations and 10k sampled DROID trajectories, with retrieved data used for skill pretraining before target-only finetuning.Target-only finetuning is used in the second stage because of the cross-embodiment gap between source data and the xArm platform.
Baselines: The comparison includes behavior cloning, random and all-data mixing, language retrieval, optical-flow retrieval, subtrajectory retrieval, and state-action retrieval methods.The state-action baselines include Behavior Retrieval and Importance Weighted Retrieval.

A. Simulation Experiments

On LIBERO, target-only finetuning and indiscriminate prior-data mixing provide limited benefits, while retrieval methods improve performance to varying degrees. HSR performs best on average, with its advantage most evident on complex composite tasks requiring reusable skills.

Baseline Analysis: Adding more prior data alone yields only modest gains when the dataset contains redundant or task-irrelevant demonstrations.This pattern is observed for both Random and All Data relative to target-only behavior cloning.
Baseline Analysis: Language retrieval improves over random sampling but remains limited on complex instructions, while state-action methods gain less on composite tasks.STRAP’s vision-based criterion is less effective at capturing the semantic structure of long-horizon tasks requiring reusable skills.
LIBERO Results: HSR achieves the best average performance on LIBERO and is especially advantageous on complex composite tasks.On simpler single pick-and-place tasks, HSR performs comparably to BR.
LIBERO Results: 27.3%: HSR outperforms STRAP on the Mug-M task.STRAP is identified as the strongest competing method on this task.
LIBERO Results: HSR is the only method that improves over pure behavior cloning on the challenging Bowl-C task.

B. Real-World Experiments

Real-world experiments show that HSR improves success rates under scarce target demonstrations, while ablations identify task decomposition, feature reranking, and two-stage training as important components.

Real-World Performance: HSR achieves higher success rates across real-robot tasks of varying horizons than baseline methods under substantial cross-embodiment gaps.Behavioral retrieval helps address the difficulty of directly transferring prior data to target tasks.
Task Decomposition: Removing task decomposition noticeably reduces performance, especially for complex instructions such as Mug-P and Moka-M.Decomposition enables more precise retrieval by breaking instructions into semantically meaningful subtasks.
Task Decomposition: 43.5±3.3 average success rate is achieved by HSR across LIBERO-10 tasks, compared with 37.1±2.5 for LLM-only plans and 37.7±1.2 for LLM-plus-skill-set plans.Manually selected plans achieve 44.9±1.0, comparable to HSR.
Feature Reranking: Removing feature reranking causes a clear performance drop, particularly on Soup-S, where low-level execution compatibility matters.Reranking filters behaviorally incompatible demonstrations after language-based retrieval.
Retention Ratio: Retaining a moderate fraction of reranked data performs best on most tasks, whereas retaining too much adds noise and retaining too little discards useful demonstrations.The retention ratio therefore trades off retrieved-data coverage against noise.
Training Method: Two-stage pretraining and finetuning outperforms single-stage co-training on most tasks by first learning general skills and then adapting with relevant demonstrations.Co-training can bias optimization toward larger, noisier retrieved datasets when their gap from target data is substantial.
Training Method: Combining two-stage adaptation with BR improves over BR alone but still underperforms HSR, supporting the benefit of HSR retrieval.The comparison separates the contribution of the retrieval strategy from the adaptation schedule.

D. Retrieved Data Study

HSR retrieves demonstrations for individual subtasks of a composite microwave task, combining object-placing and microwave-closing skills to improve task relevance.

Retrieved Data: HSR retrieves demonstrations across subtasks covering both object-placing and microwave-closing skills.The example task is “put the yellow and white mug in the microwave and close it.”
Retrieved Data: Subtask-specific retrieval produces demonstrations more closely aligned with each step’s requirements, yielding more task-relevant retrieved data.This contrasts with retrieval based only on the original full-task description.

VI. CONCLUSION

HSR uses hierarchical task structure and behavior-aware retrieval to reuse transferable skills for VLA adaptation with limited target data. The paper reports consistent improvements, while noting scope boundaries around architectures, embodiments, modalities, and closed-loop control.

Conclusion: HSR combines semantic task decomposition with low-level behavior-aware retrieval instead of relying on exact task-level matches.This lets the policy reuse transferable skills from prior datasets.
Conclusion: Experiments on LIBERO and real-world long-horizon manipulation tasks consistently improve adaptation performance under limited target data.The conclusion frames the result across both simulated and real-world settings.
Limitations: HSR is evaluated only on manipulation tasks and a single VLA backbone, leaving robustness across architectures, embodiments, and data modalities untested.The authors identify broader evaluation as an important direction.
Limitations: The framework does not yet tightly integrate language models into closed-loop control for online replanning or failure recovery.The authors also note that VLA deployment failures remain possible even with additional training data.