Source-linked AI summary
HiVe: Beyond Static Prompts for Multitask Learning via Hierarchy-based Vertical Mixture-of-Experts
HyeonJik Bae, Minyeol Kim, Susik Yoon
TL;DR
Existing prompt-tuning methods use flat prompts or fixed hierarchical compositions, limiting input-dependent specialization. HiVe learns a multi-level prompt hierarchy from task relationships and uses vertical mixture-of-experts routing to select and compose prompts to the needed depth. It consistently outperforms strong prompt-tuning baselines across question answering and summarization while remaining parameter-efficient.
Problem
Existing methods use flat prompt structures or fixed hierarchical prompt composition, limiting adaptive specialization for inputs requiring different levels of shared or task-specific knowledge.
Method
HiVe induces a base-group-task prompt hierarchy through stability-driven grouping and residual decomposition, then uses key-based V-MoE routing to select specialization depth per input.
Results
HiVe consistently outperforms strong prompt-tuning baselines across in-domain and out-of-domain question answering and summarization benchmarks while maintaining parameter efficiency.
Takeaways & Limitations
Dynamically controlling prompt specialization provides a parameter-efficient way to balance shared and task-specific knowledge across diverse inputs and tasks.
Takeaways & Limitations
HiVe is evaluated on within-task distribution shifts, while generalization across task types remains outside the study’s scope.
Abstract
from arXiv · showhide
As large language models (LLMs) continue to scale, parameter-efficient fine-tuning (PEFT) has become a practical alternative to full-parameter adaptation. Prompt tuning is effective, but existing approaches either use flat prompt structures or hierarchical structures with fixed prompt composition, limiting adaptive prompt specialization. To address this limitation, we propose HiVe, a prompt tuning framework that models prompts at multiple levels and enables input-dependent specialization. HiVe constructs a prompt hierarchy by leveraging inter-task relationships during training, and employs a vertical mixture-of-experts (V-MoE) mechanism at inference time to compose prompts up to the level of specialization required for each input. Experiments show that HiVe consistently outperforms strong prompt tuning baselines across diverse tasks.
1 Introduction
HiVe addresses the limitations of flat or fixed hierarchical prompt structures by constructing a learned prompt hierarchy and adaptively selecting specialization depth for each input. It combines stability-driven task grouping, residual prompt decomposition, and vertical MoE routing, achieving strong results across question answering and summarization benchmarks.
- Motivation: Flat prompt spaces treat all experts at one level, although inputs may require either shared knowledge or task-specific specialization.Existing flat MoE approaches jointly encode general and specialized knowledge within a few prompts, limiting adaptation to varying input needs.
- HiVe: HiVe learns a prompt hierarchy and uses it as a routing space for input-dependent specialization.The framework organizes prompts into progressively specialized representations and selects the required level through key-based routing.
- HiVe: HiVe constructs base, group, and task prompts through stability-driven task grouping and residual decomposition.Shared information is extracted into higher-level prompts while task-specific information remains at lower levels, reducing redundancy and interference.
- HiVe: Vertical MoE routing determines how deeply to traverse the hierarchy and progressively composes prompts to the selected level.This differs from routing among parallel experts and balances shared with task-specific knowledge according to each input.
- Evaluation: HiVe outperforms prompt-tuning baselines in F1 on 12 MRQA datasets and ROUGE-L on 11 summarization datasets across in-domain and out-of-domain settings.It remains competitive with full fine-tuning and low-rank adaptation despite using a limited prompt parameter budget.
2 Related Work
Related work connects HiVe to mixture-of-experts parameter-efficient tuning and task-relatedness methods for organizing multi-task learning. HiVe’s framework combines these themes through hierarchical prompts and routing.
- MoE-based PEFT: Recent PEFT methods use mixture-of-experts mechanisms to conditionally activate lightweight adapter, LoRA, or prompt experts.Examples include AdaMix, HydraLoRA, ATTEMPT, SMoP, and PT-MoE.
- Task Relatedness: Task relatedness matters because jointly learning loosely related tasks can cause interference and negative transfer.Prior work estimates task relationships through shared latent components or gradient-based task affinity.
- HiVe: HiVe organizes task prompts into a base-group-task hierarchy and routes inputs to an appropriate specialization level.Its framework uses task clustering during training and routing-key matching during inference.
3 Methods
HiVe builds a base-group-task prompt hierarchy from learned task relationships, then uses key-based vertical routing to select and compose prompts to each input’s required specialization level.
- Hierarchy construction: HiVe constructs a three-level base-group-task hierarchy, with grouping finalized through stability-based clustering during training.Tasks are clustered from prompt similarities using HAC, silhouette-based cluster selection, and ARI stability checks before Phase 2 begins.
- Prompt decomposition: Residual prompt decomposition transfers shared information into higher-level prompts while preserving task-specific representations at lower levels.Base, group, and task prompts are encouraged to encode representations at their respective granularities.
- Hierarchy construction: Grouping stability is measured with mean ARI, and the hierarchy is finalized only after the stability threshold is exceeded for consecutive checks.This avoids unreliable early grouping caused by immature prompts while limiting delays to hierarchical learning.
- Vertical mixture-of-experts: V-MoE uses key-based routing to select the hierarchy level for each input and progressively composes prompts from the base level to that selected level.The selected key is obtained by cosine similarity between a projected input query and routing keys, without requiring task IDs.
- Vertical mixture-of-experts: Decoupled training constructs prompts with the ground-truth hierarchy while separately optimizing routing keys, preventing routing errors from contaminating hierarchy-level representations.Task keys are optimized directly, while higher-level keys are derived as centroids and prompt construction uses the full ground-truth hierarchy.
- Vertical mixture-of-experts: The selected hierarchical prompt is projected through W and prepended to the input sequence for inference.The composed prompt ranges from the base prompt alone to the full base-group-task hierarchy.
4 Experiments
HiVe is evaluated on question answering and summarization against adaptation baselines using standard metrics across in-domain and out-of-domain settings. Results and analyses show strong performance, adaptive specialization, stable grouping, and sensitivity to several design choices.
- Overall Performance: HiVe achieves 75.4 average F1 in-domain on MRQA, versus 75.2 for full fine-tuning while updating approximately 0.007% of model parameters.It improves over SMoP and PT-MoE by 2.1 and 2.6 F1 points, respectively.
- Overall Performance: On MRQA out-of-domain data, HiVe outperforms full fine-tuning by 2.7 average F1 points and achieves the best result on four of six datasets.It also improves over SMoP and PT-MoE by 1.4 and 2.2 F1 points, respectively.
- Overall Performance: HiVe reaches 26.1 average ROUGE-L in-domain for summarization, versus 26.0 for full fine-tuning, and leads on five of six datasets.Its average ROUGE-L exceeds SMoP and PT-MoE by 0.8 and 1.0 points, respectively.
- Ablation Study: Removing residual decomposition causes larger performance drops out-of-domain, indicating that disentangling shared and task-specific representations supports hierarchy-level roles.Without decomposition, shared and task-specific representations become entangled.
- Ablation Study: Single-level hierarchies underperform the full Base–Group–Task model, with task-only prompts degrading more under domain shifts.Base-level and group-level-only variants show lower overall performance.
- Routing and Specialization: Static single-key routing generally degrades performance, while task-specific routing preferences and progressively specialized predictions support input-dependent multi-level routing.RE favors group-level routing, DRC relies more on base-level routing, and different hierarchy levels produce different semantic specificity.
5 Conclusion
HiVe adaptively controls prompt specialization through vertical mixture-of-experts routing and disentangles shared and task-specific knowledge across hierarchical levels. It consistently outperforms strong prompt-tuning baselines across question-answering and summarization benchmarks while remaining parameter-efficient.
- HiVe adaptively selects each input’s required prompt-specialization level through vertical mixture-of-experts routing.The learned hierarchy serves as an input-dependent routing space for controlling specialization depth.
- Stability-driven task grouping and residual prompt decomposition disentangle shared and task-specific knowledge across hierarchical levels.
- HiVe consistently outperforms strong prompt-tuning baselines on question-answering and summarization benchmarks in both in-domain and out-of-domain settings.
- HiVe maintains parameter efficiency while achieving competitive performance with resource-heavy alternatives.
Limitations
HiVe’s stability-driven task grouping can vary with configuration choices and may reduce downstream performance. Its evaluation covers within-task distribution shifts, while broader task-type generalization and substantial shifts remain outside the learned hierarchy’s coverage.
- Task grouping can depend on random seeds, prompt length, low-rank dimensionality, and grouping hyperparameters.
- Prematurely finalized grouping can produce different group structures and reduced downstream performance.
- Evaluation focuses on within-task distribution shifts, leaving generalization across task types outside the study’s scope.
- Adaptive routing cannot capture patterns beyond the learned hierarchy under substantial distribution shifts.
A Dataset Details
The experiments use capped question-answering and summarization datasets with unified conversational formatting. Prompts are initialized from generic or task-specific natural-language templates and projected into a shared low-rank space.
- Summarization training, validation, and test samples are capped at 40K, 10K, and 3K per task, respectively.
- Question-answering and summarization inputs use unified conversational templates based on the Llama-3.2-1B-Instruct chat format.Question answering places context and question fields in the user message, while summarization places the document field there.
- Training appends the target sequence after the assistant prefix, whereas inference excludes the target sequence.
- The base prompt and projection matrix use generic task-independent instruction templates instead of random-vector initialization.
- Task prompts begin from task-specific template embeddings projected into a shared low-rank space, with task keys initialized from mean-pooled embeddings.
D.1 Experiment Settings
The experiments report benchmark comparisons for MRQA and summarization under in-domain and out-of-domain settings, using controlled hyperparameter and parameter-budget choices. HiVe is evaluated only during Phase 2.
- Training and generation settings are summarized for MRQA and summarization experiments, with HiVe evaluated only during Phase 2.
- MRQA results include EM and F1 comparisons across in-domain and out-of-domain datasets, while summarization results use ROUGE-L comparisons.
- Baseline and HiVe comparisons use method-family learning rates, fixed prompt lengths, and adjusted low-rank dimensions for comparable parameter budgets.
E.1 MRQA EM Results
This section reports detailed MRQA exact match results across in-domain and out-of-domain datasets and places them alongside inference-efficiency comparisons with prompt-tuning baselines.
- MRQA evaluation uses exact match (EM) across both in-domain and out-of-domain datasets.
- Table 10 provides the detailed MRQA EM results.
- Table 13 compares inference efficiency with prompt-tuning baselines.
E.2 Additional Backbone Experiments
Additional experiments show that HiVe retains advantages with Llama-3.1-8B-Instruct, uses fewer parameters, and maintains comparable inference efficiency while ablations clarify its design choices.
- Additional Backbone Experiments: HiVe achieves higher average performance on both benchmarks with Llama-3.1-8B-Instruct while using fewer parameters.
- Additional Backbone Experiments: Relative FLOPs and inference latency are evaluated as efficiency measures.
- Additional Backbone Experiments: HiVe achieves inference efficiency comparable to existing prompt-tuning baselines.
- Additional Backbone Experiments: Tables 14 and 15 report full ablation results on MRQA and summarization benchmarks.
- Additional Backbone Experiments: Residual decomposition preserves high-level task grouping while producing clearer separation between task groups on MRQA.This suggests that shared prompt components are removed while task-specific structure is preserved.
- Additional Backbone Experiments: Stability-driven grouping performs better than both the two-level Base+Task variant and the Step-0 static variant.The grouping uses stabilized task-prompt representations rather than omitting the group level or fixing groups before prompt training.
F.4 Ablation of Key Design Choices
The ablation shows that learned task-prompt representations support robust clustering, whereas independently learned keys and negative Euclidean matching weaken prompt selection and performance.
- HAC and K-means yield comparable performance when clustering learned task-prompt representations.This indicates robustness to the choice of clustering algorithm.
- Independently learning Base and Group keys causes most inputs to match the Base key, substantially reducing in-domain performance.
- Replacing cosine similarity with negative Euclidean distance degrades performance.