PaperlayerView exact revision on arXiv

Source-linked AI summary

HydraLLM: Adaptive Width Slicing for Elastic Text and Vision-Language Models

Oscar Kemme, Janek Haberer, Ali Hojjat, Olaf Landsiedel

arXiv:2610.23455v1cs.CLcs.LG

TL;DR

Fixed-width LLMs and LVLMs are difficult to reuse across heterogeneous hardware, while many elastic approaches vary only one model component. HydraLLM converts a pretrained checkpoint into nested width-configurable subnetworks and extends width control to multimodal projection. It reports stronger Trivia-QA accuracy than a larger-parameter nested baseline, multimodal CLIPScore above a much larger model, and higher token-generation throughput at reduced width.

  • Problem

    Fixed-size LLMs and LVLMs limit reuse across deployment settings with different compute, latency, and model-size targets, while many elastic methods focus on a single component.

    Source: §1 · p. 1

  • Method

    HydraLLM converts a pretrained checkpoint into nested MQA decoder subnetworks controlled by active head count, using weight reordering and stochastic subnetwork training, and makes multimodal projection width follow active text width.

    Source: §1 · p. 2, §3.2.2 · p. 4, §3.3 · p. 4

  • Results

    At 75% active heads, HydraLLM improves Trivia-QA accuracy by 2.2 percentage points over a baseline configuration with 17.25% more parameters.

    Source: §1 · p. 2

  • Takeaways & Limitations

    HydraLLM covers a wider parameter range at competitive accuracy than MatFormer-style FFN slicing and achieves larger token-generation speed-ups, though prompt-processing improvements are modest.

    Source: §6 · p. 8

    The multimodal model remains near random accuracy on MMMU, with qualitative analysis showing hallucinations and weak object grounding.

    Source: §6 · p. 8

Abstract

from arXiv · show

LLMs and LVLMs are increasingly deployed across heterogeneous hardware, ranging from data-center GPUs to resource-constrained edge devices. Their fixed-size architectures limit the ability to reuse a single model across deployment settings with different compute, latency, and model-size targets. We introduce HydraLLM, an elastic MQA-based architecture. Its number of active attention heads determines the effective widths of the embeddings, attention projections, feed-forward layers, and the output head, forming a nested family of subnetworks within a single parameter set. We introduce a static-to-elastic conversion pipeline that, starting from a pre-trained model checkpoint, includes importance-driven weight reordering and stochastic subnetwork training with knowledge distillation. We extend this principle to multimodal inference through an elastic projection module whose output width follows the active text width. HydraLLM covers a wider range of active parameter configurations than FFN-only elastic architectures. With 75% of heads active, HydraLLM achieves 2.2 percentage points higher TriviaQA accuracy than a state-of-the-art nested baseline configuration which has 17.25% more parameters. With 50% of heads active, our multimodal model achieves a CLIPScore comparable to that of the 7$\times$ larger Gemma 3 4B model. With only 25% of heads active, HydraLLM increases token-generation throughput by 75.3% over the full-width configuration.

1 Introduction

Fixed-width LLMs and LVLMs are difficult to reuse across hardware with different compute, latency, and model-size requirements, while many elastic methods slice only one component. HydraLLM makes decoder width configurable from one pretrained checkpoint and extends that control to multimodal projection, with reported text, vision-language, and throughput results.

1 Introduction

  • HydraLLM addresses static-model deployment across heterogeneous hardware and the limits of elastic methods that focus on one component or require multi-step checkpoint extraction.The introduction identifies differing latency, compute, and model-size requirements and says training separate models for each configuration is computationally prohibitive.Source: §1 · pp. 1–2
  • At 75% active heads, HydraLLM improves Trivia-QA accuracy by 2.2 percentage points over a baseline configuration with 17.25% more parameters.At 50% active heads, it reaches a CLIPScore of 66.9, surpassing the 65.3 score of the 7× larger Gemma 3 4B model; at 25%, throughput increases by 75.29% versus the full model.Source: §1 · p. 2
  • A single head-count control selects consistent prefixes of embedding, attention, FFN, and output dimensions to form nested decoder subnetworks from one checkpoint.The active heads determine embedding, attention projection, and feed-forward widths.Source: Figure 1 · p. 1, §1 · p. 2
  • The paper contributes a unified training recipe for converting a static pretrained LLM into an adaptively sliceable model and an elastic projection for LVLMs.The projection maps visual features into text embedding spaces of varying dimensionality.Source: §1 · p. 2

2 Related Work

Related work spans nested representations and elastic architectures, model compression and extraction, and projection-based multimodal models. HydraLLM is distinguished by runtime joint width slicing of a pretrained LLM and an elastic multimodal projection.

2 Related Work

  • Unlike MatFormer’s FFN-only nesting, HydraLLM jointly scales embedding, attention, FFN, and output widths at runtime in one elastic checkpoint.OFA progressively shrinks CNNs for architecture search and extraction, whereas HydraLLM does not perform architecture search or progressive shrinking.Source: §2 · p. 2
  • HydraViT applies nested slicing to embeddings, attention heads, and FFN layers; HydraLLM adapts this training and architectural approach to LLMs and extends it to vision.Stochastic subnetwork sampling draws one head count per step and updates only that subnetwork, biasing optimization toward earlier heads.Source: §2 · pp. 2–3
  • FLEXTRON and Nemotron Elastic use routers for nested attention and FFN components but are text-only, while MINITRON extracts static submodels unsuitable for elastic inference.The cited work describes FLEXTRON as slicing attention heads and FFN, and Nemotron Elastic as targeting an architecture distinct from HydraLLM’s MQA transformer.Source: §2 · p. 3
  • Projection-based multimodal models map vision features into an LLM’s text embedding space, while HydraLLM combines those principles with an elastic LLM and multimodal projection.The passage names CLIP-connected LLMs, Molmo, SmolVLM, and Gemma 3 as related approaches.Source: §2 · p. 3

3 HydraLLM: Architecture and Pipeline

HydraLLM converts a static pretrained checkpoint into nested decoder subnetworks controlled by one runtime head-count parameter, then applies the same width control to multimodal projection. The pipeline reorders weights before stochastic subnetwork training.

3 HydraLLM: Architecture and Pipeline

  • The pipeline converts a static pretrained text checkpoint into a runtime-width-controlled elastic decoder and extends the same elasticity principle to multimodal projection.The section covers the text architecture, conversion procedure, optimization objective, and multimodal assembly.Source: §3 · p. 3

3.1 Runtime Width-Control

  • A runtime choice of k retains the first k query heads and corresponding prefixes of embedding, attention-output, and FFN dimensions, producing nested subnetworks that share parameters.Active embedding width scales as k/Hq of the full embedding width; the design targets MQA, where query heads share one key-value head.Source: §3.1 · p. 3

3.2 Static-to-Elastic Conversion

  • Conversion first reorders important units toward slice prefixes, then samples one active subnetwork per training step and updates only that subnetwork.Earlier units receive more gradient updates because they are active for more sampled values of k.Source: §3.2 · p. 3

3.2.1 Importance Scoring and Reordering

  • Importance scores from calibration-corpus activation statistics determine permutations of embedding channels, query heads, and FFN neurons while preserving the full model’s function.The compared measures are activation magnitude, VxC, and CETT; embedding reordering is propagated consistently through connected tensors.Source: §3.2.1 · p. 4

3.2.2 Optimization Objective

  • Training combines next-token cross-entropy, weighted z-loss, and logit-based knowledge distillation, with the teacher kept frozen.The passage defines temperature-scaled teacher and student output distributions for the distillation objective.Source: §3.2.2 · p. 4

3.3 Elastic Multimodal Assembly

  • The multimodal model combines an elastic decoder with a static pretrained SigLIP-based vision encoder, setting projection output width equal to active text embedding width.The same runtime k therefore selects a correspondingly smaller visual projection output; Figure 3 depicts this width relationship.Source: §3.3 · p. 4, Figure 3 · p. 4

4 Experiment Setup

The experiments train elastic text and multimodal models from Gemma 3 checkpoints, then evaluate language, multimodal, and throughput performance across width configurations.

4.1 Training Recipe

  • Text training uses frozen full-width Gemma 3 270M as teacher, FineWeb-Edu data, and runtime width sampling that emphasizes the full-width model threefold.The optional text fine-tuning stage uses unpacked samples; the reported perplexity tables measure Shakespeare-play samples.Source: §4.1 · p. 4, Table 1 · p. 5, Table 2 · p. 5
  • Multimodal training combines elastic Gemma 3 270M with a vision encoder and projection initialized from Gemma 3 4B, using a frozen 4B teacher and uniform head sampling.Training includes captioning, question-answering, and text-only datasets; projection alignment uses 128,000 captioning samples.Source: §4.1 · pp. 4–5

4.2 Metrics and Evaluation Datasets

  • Text evaluation uses ARC-Easy accuracy, five-shot exact-match TriviaQA, and Shakespeare perplexity; multimodal evaluation reports CLIPScore on COCO validation images and qualitative captions.ARC-Easy also reports answer-length-normalized accuracy, and CLIPScore is multiplied by 100 for readability.Source: §4.2 · p. 5
  • Throughput is measured with 2,048-token prompts and 512-token autoregressive generations on an NVIDIA RTX PRO 4000.The setup measures both prompt-processing and token-generation speed.Source: §4.2 · p. 5

5 Evaluation

HydraLLM’s results vary by task and width: it improves selected text and throughput comparisons, while multimodal caption quality remains unreliable despite strong CLIPScore results.

5.1 Effect of Importance Reordering

  • Magnitude embeddings with CETT heads and FFN, or magnitude across components, improves submodel perplexity; VxC helps only H1.After 20,000 training steps, perplexity improves across head configurations, but the benchmark accuracy of the no-reordering model is lower than the static baseline.Source: §5.1 · pp. 5–6

5.2 HydraLLM vs. MatFormer

  • ARC-Easy results show HydraLLM spanning a wider parameter range with competitive accuracy; W0.25 has 17.25% more parameters than H3, with raw-accuracy and normalized-accuracy comparisons near or statistically indistinguishable.The raw accuracy advantage for W0.25 over H3 is not statistically significant, and their log-probability-normalized accuracies are virtually the same.Source: Figure 4 · p. 6, §5.2 · p. 6

5.2 HydraLLM vs. MatFormer; 5.3 Multimodal Pipeline Results

  • 2.21 percentage points higher TriviaQA accuracy at H3 than similarly sized W0.25 accompanies 75.29% token-generation scaling and CLIPScore matching or exceeding Gemma 3 4B at H2–H4.CLIPScore is sensitive to response style and does not by itself guarantee stable caption quality.Source: Figure 5 · p. 6, Table 3 · p. 6, §5.2 · pp. 6–7

5.2 HydraLLM vs. MatFormer

  • Prompt-processing gains are modest, while token-generation speed increases 75.29% from H4 to H1 for HydraLLM versus 11.17% for MatFormer, without scaling one-to-one with parameter count.The reported gains mostly reflect smaller embeddings and query, output, and FFN projections; MQA precludes width-based key-value cache savings.Source: Table 3 · p. 6, §5.2 · pp. 6–7
  • TriviaQA favors H3 over W0.25, but W1.0 exceeds H4 by 0.68 percentage points, while HydraLLM H1 and H2 remain near 0% accuracy.HydraLLM starts at 49.34M parameters versus MatFormer’s 215.01M.Source: §5.2 · p. 6

5.3 Multimodal Pipeline Results

  • Projection-only alignment yields modest CLIPScore gains and plateaus after 300–350 steps; joint projection-and-text training substantially improves scores, with pre-alignment performing best across head configurations.The projection has 25% of Gemma 3 4B’s parameters; all-parameter joint optimization produces similar results across starting points.Source: §5.3 · p. 7
  • MMMU accuracy remains around 25% versus about 39% for Gemma 3 4B, and captions contain hallucinated or incorrect scene details across all head configurations.The authors identify the vision-to-language interface as a suspected bottleneck; Figure 6 illustrates the qualitative captioning gap.Source: §5.3 · pp. 7–8, Figure 6 · p. 8

6 Conclusion

HydraLLM uses one head-count control to create nested text and vision-language subnetworks, with reordering and distillation supporting their performance. Its results span text accuracy, multimodal caption scoring, and token-generation speed, while multimodal reasoning and grounding remain weak.

6 Conclusion

  • Magnitude-based embedding reordering with CETT-based head and FFN reordering reduces subnetwork perplexity by up to approximately 4,500× versus no reordering, but training reduces the effect to at most 14%.The cited reordering combination produces the strongest subnetworks.Source: §6 · p. 8
  • HydraLLM covers a wider active-parameter range than MatFormer-style FFN slicing, with up to 75.3% larger token-generation speed-ups but only modest prompt-processing improvements.Knowledge distillation improves TriviaQA exact-match accuracy by 97.3% at 75% active heads.Source: §6 · p. 8
  • Knowledge distillation (Hinton et al., 2015) is crucial during training and improves TriviaQA exact-match accuracy by 97.3% at 75% of active heads.Source: §6 · p. 8
  • With two, three, and four active heads, scaled multimodal training exceeds Gemma 3 4B’s CLIPScore of 65.3 by 2.5%, 5.1%, and 6.6%, respectively.The model remains near random accuracy on MMMU, with hallucinations and weak object grounding; the authors identify a small single-layer projector and limited multimodal training as directions for improvement.Source: §6 · p. 8

A Reproducibility Details

The reproducibility section collects implementation and hardware details, runtime configurations, multimodal evaluation and dataset information, and training settings.

A Reproducibility Details

A.1 Implementation and Hardware

The appendix documents the software stack, hardware requirements, throughput setup, and runtime width configurations used in the experiments.

A.1 Implementation and Hardware

  • The implementation uses PyTorch and Hugging Face Transformers, with Accelerate and DeepSpeed for distributed training and LightEval for text benchmarks.Optional FlashAttention 2 acceleration has a fallback to standard PyTorch and Transformers backends.Source: §A.1 · p. 12
  • Text experiments use two NVIDIA RTX PRO 4000 GPUs with at least 24 GB VRAM each, while multimodal training requires at least 48 GB VRAM and roughly 60 GB for joint training.Multimodal experiments run on two NVIDIA Tesla L40 or two NVIDIA A100 80 GB GPUs; throughput uses llama.cpp build 4df6e859e (7349).Source: §A.1 · p. 12

A.2 Runtime Configurations

  • Table 6 lists active component dimensions for head configurations k ∈ {1, 2, 3, 4}.Source: §A.2 · p. 12

A.3 Multimodal Dataset Composition

The appendix specifies multimodal dataset mixtures, CLIPScore evaluation, and training settings, then identifies additional text experiments on ablations and reordering.

A.3 Multimodal Dataset Composition

  • The scaled 16,000-step multimodal runs preserve the broad dataset mixture while increasing sample counts.The mixture includes captioning, question answering, code generation, and text-only data; projection alignment uses 128,000 samples split evenly between COCO Recaptioned and PixMo-Cap.Source: §A.3 · p. 12

A.4 CLIPScore Evaluation

  • CLIPScore is calculated on the entire COCO validation set using clip-vit-large-patch14.The passage describes CLIPScore as measuring caption quality, not defining the dataset or the underlying CLIP model.Source: §A.4 · p. 13

A.5 Training Hyperparameters

  • Scaled multimodal runs train for 16,000 steps on the larger dataset mixture, with optional vision-encoder training and adjusted text and vision warmup settings.They use a lower text-backbone learning rate, a separate projection learning rate, a larger effective batch size, higher distillation weight when using Gemma 3 4B as teacher, and uniform head sampling.Source: §A.5 · p. 13

B Additional Text Experiments

B.1 Training Recipe Ablation

The section compares training recipes and importance-based reordering for elastic subnetworks, then reports how reordering affects downstream accuracy. It also introduces three unit-scoring approaches and points to additional multimodal results.

B.1 Training Recipe Ablation

  • The training-recipe ablation varies teacher use, head sampling, training configurations, distillation weight, data, EMA decay, batch size, learning rate, and training duration.The long-run variant uses one million steps and the larger FineWeb-Edu 350BT subset.Source: §B.1 · p. 13

B.2.1 Magnitude-Based Importance

  • Magnitude scoring ranks units by activation size, but does not account for how strongly downstream weights consume their outputs.The criterion assumes consistently larger activations contribute more strongly.Source: §B.2.1 · pp. 13–14

B.2.2 Variance × Consumer Norms

  • VxC ranks units using activation variance weighted by downstream consumer strength, and aggregates embedding scores across layers into a global ranking.Units that vary substantially but have weak consumption rank below units whose variation is amplified by large outgoing weights.Source: §B.2.2 · p. 14

B.2.3 CETT-Based Importance

  • CETT scores each unit by its direct contribution to component output; its normalized form estimates contribution relative to total output and targets prefix-slicing error.The normalized score is identified as most relevant for prefix slicing because slicing discards the tail of a dimension.Source: §B.2.3 · pp. 15–16

B.2.4 Effect of Importance Reordering

  • Mag-CETT-CETT improves TriviaQA exact-match accuracy across all head configurations, while ARC-Easy and HellaSwag remain close to the no-reordering baseline.The comparison is after 20,000 training steps.Source: §B.2.4 · p. 16

C Additional Multimodal Results

C.1 Text Recovery after Multimodal Training

The section compares text-recovery outcomes across continued-training and fine-tuning variants, including different teachers. It also reports that one-head projection alignment converges to higher loss than wider configurations.

C.1 Text Recovery after Multimodal Training

  • The Gemma 3 4B teacher consistently outperforms the 270M teacher at the four-head configuration, suggesting greater transfer of factual knowledge to the student.The section compares text-only continued pre-training with multimodal or text-only fine-tuning variants.Source: §C.1 · p. 16

C.2 Projection Alignment Loss Curves

  • One-head projection alignment converges to higher loss, while configurations with two or more heads reduce loss rapidly and stabilize within the first 100 steps.The one-head subnetwork is described as reaching its capacity limit earlier.Source: Figure 7 · p. 16, §C.2 · p. 16
Loading 2610.23455v1…