PaperlayerView exact revision on arXiv

Source-linked AI summary

Retrofitting Recurrent Vision-Language Models from Pretrained Backbones

Madhav Iyengar, Andreas Geiger, Hilde Kühne

arXiv:2610.22075v1cs.CV

TL;DR

Retrofitting depth recurrence into pretrained VLMs offers a way to reduce language-model parameters while retaining visual-language performance. The paper prunes and loops layers in a pretrained VLM, then applies staged recovery training with distillation. On TinyLLaVA-1.4B, the retrofitted model outperforms the unpruned recovery-trained baseline with 23.3% fewer language-model parameters, with gains concentrated on reasoning benchmarks.

  • Problem

    Existing test-time scaling approaches for VLMs can increase inference costs because each generated token requires a full forward pass and expands the context window.

    Source: §1 · p. 1

  • Method

    The method prunes intermediate language-backbone layers in a pretrained VLM, loops a subset of the remaining layers with an adapter, and uses staged recovery training with distillation.

    Source: §1 · p. 2, §3.1 · p. 4, §3.2 · p. 5

  • Results

    The retrofitted model outperforms the unpruned baseline trained with the same recovery data while storing 23.3% fewer language-model parameters, with gains concentrated on reasoning-oriented benchmarks.

    Source: §1 · p. 2, §6 · p. 9

  • Takeaways & Limitations

    On the evaluated setup, retrofitting recurrence into a pretrained VLM outperforms adding a vision encoder to a pretrained recurrent language model.

    Source: §6 · p. 9

Abstract

from arXiv · show

Recent work shows that depth recurrence can be retrofitted into pretrained language models. In this work, we adapt this paradigm for visual understanding, using depth recurrence to reduce the parameter footprint of a pretrained vision-language model (VLM). To this end, we prune layers from the language backbone of the VLM and loop over a subset of the remaining layers. At each step of the loop, a linear adapter merges the block's output with the token representations from the preceding layers and projects the fused representation back into the block. As the pruning impacts the original model performance, we propose a three-stage recovery training: first, we leverage a language-only pretraining to recover the performance of the language backbone, followed by an alignment of the vision-language projector, and a full fine-tuning of the model via visual instruction tuning. In the first and the last stage, we distill the outputs of the pretrained non-recurrent model into the recurrent model, which further improves the recovery. We evaluate the proposed approach on the TinyLLaVA-1.4B backbone without introducing any new vision-language data during retrofitting. Our experiments demonstrate that the retrofitted recurrent model outperforms the unpruned baseline trained with the same recovery data while storing 23.3% fewer language parameters. We further show that the proposed retrofitting recurrence outperforms the setup of converting a recurrent LM into a VLM on the same data. Additionally, our ablations show how design choices, such as the adapter between recurrences and a larger recurrent block, lead to the improved performance of our approach. Finally, we find that our proposed model gains on reasoning-oriented benchmarks, whereas its performance on basic visual understanding benchmarks remains on par with the baseline.

1 INTRODUCTION

This work retrofits recurrence into a pretrained VLM by pruning and repeatedly executing some language-backbone layers, then uses a three-stage curriculum to recover performance and projector alignment. On TinyLLaVA-1.4B, the approach outperforms an unpruned baseline trained with the same recovery data while using fewer language-model parameters, with gains concentrated on reasoning-oriented benchmarks.

1 INTRODUCTION

  • Depth recurrence allocates test-time compute to repeated layer execution rather than generated tokens, keeping context size and memory overhead smaller independent of reasoning length.The passage contrasts this with test-time methods that require a full forward pass for each generated token and expand the context window.Source: §1 · p. 1
  • Pruning intermediate language-backbone blocks and looping a subset of the remaining layers defines the paper’s prune-then-loop retrofit of a pretrained VLM.A linear adapter fuses the recurrent block’s output with preceding-layer token representations; recovery proceeds through text-only training, projector alignment, and visual instruction tuning, with distillation in the first and final stages.Source: §1 · p. 2
  • The retrofitted model outperforms the unpruned baseline trained with the same recovery data while using 23.3% fewer language-model parameters.Its gains concentrate on reasoning-oriented benchmarks; performance on basic visual-understanding benchmarks remains on par with the baseline.Source: §1 · pp. 2–3
  • The retrofit is evaluated on TinyLLaVA-1.4B using the base model’s training data, without introducing new vision-language data.The setup uses 60B text tokens, 493k image–caption pairs, and 660k instruction-tuning samples.Source: §1 · p. 2
  • The retrofitted model achieves higher average accuracy than a recurrent language model extended into a VLM, and the reported gain over the unpruned baseline is attributed to recurrence rather than recovery training.The evaluation also compares against an unpruned model with identical recovery training, an unpruned model without the text-only stage, and a pruned model without recurrence.Source: §1 · p. 2

2 RELATED WORK

Prior recurrent-VLM approaches either add vision to a recurrent language model or loop a pretrained VLM without pruning. This work instead follows a prune-then-loop retrofit, situated among methods that reduce pretrained-model depth before installing recurrence.

Depth recurrence and architectural retrofitting

  • The paper adopts structural pruning before recurrence, drawing on retrofitting approaches that loop surviving layer blocks or merge neighboring layers into looped blocks.Earlier work also loops middle blocks without pruning for latent reasoning; the cited TinyLlama example drops layers 4 to 9 and uses a 4/8/4 allocation with a linear adapter.Source: §2 · p. 3

Recurrence in vision-language models

  • Prior recurrent VLMs either add a vision backbone to a recurrent language model or loop layers of a pretrained VLM without pruning.The latter approaches retain a near-constant number of parameters; the cited examples are RecursiveVLM and ReLoop-UME.Source: §2 · p. 3
  • HIVE extends the pretrained recurrent Huginn-3.5B with an InternViT-300M encoder and trains on 6.5M samples, so its model and data scale prevent direct checkpoint comparison.The authors replicate HIVE’s setting as a baseline rather than compare against its checkpoint.Source: §2 · p. 3
  • The authors present prune-then-loop retrofitting of a pretrained VLM evaluated on VQA as the first approach of its kind to their knowledge.They distinguish it from prior visual-action recurrence work, which operates in a different domain; the cited pruned method also lacks a prune-only baseline.Source: §2 · p. 3

3 METHOD: PRUNE-THEN-LOOP RETROFITTING

The method retrofits a pretrained VLM by pruning language-model layers and repeatedly applying a subset of the remaining layers. A three-stage curriculum repairs the model and trains the adapter, with distillation used to limit representation drift and improve recovery.

3.1 RETROFITTING EXISTING VLMS

  • The prune-then-loop design deletes intermediate language-model layers and partitions the rest into prelude, recurrent, and coda blocks.The prelude and coda remain outside the recurrent block.Source: §3.1 · p. 4
  • The model applies the prelude once, repeats the adapter-plus-recurrent block r times, and applies the coda once to produce output logits.The adapter fuses recurrent-block output with prelude context by concatenating tokens and projecting them back to dimension H.Source: §3.1 · p. 4
  • The recurrence count can be fixed or sampled during training, while inference uses a fixed count that may differ from the training configuration.Source: §3.1 · p. 4
  • The method considers cache sharing across recurrent passes, including reusing the first pass’s keys or keys and values to reduce cache size.One option changes only inference; the other changes the forward pass and is active during both training and evaluation.Source: §3.1 · p. 4

3.2 TRAINING STRATEGY

  • The three-stage curriculum repairs the pruned language backbone on text, realigns the projector, then fine-tunes the full model on visual instruction data.Source: §3.2 · p. 5
  • Distillation uses the original unpruned model as teacher to limit representation drift, and the authors report that it further improves stage-S2 performance.The stage-S0 objective compares a single teacher pass with the recurrent student’s iterative execution on text-only data.Source: §3.2 · p. 5

4 IMPLEMENTATION DETAILS

The implementation retrofits TinyLLaVA-1.4B by deleting six contiguous language layers and using a fixed six-step recurrence. Training follows the three-stage curriculum on the base model’s text, image-caption, and instruction data.

4 IMPLEMENTATION DETAILS

  • The base TinyLLaVA-1.4B checkpoint combines a 22-layer TinyLlama-1.1B language model, CLIP ViT-L/14-336 vision tower, and LLaVA-style projector.Source: §4 · p. 5
  • Deleting layers 4–9 leaves a 16-layer language model allocated as 4/8/4 prelude, recurrent block, and coda, with six recurrences.The resulting language model, including its adapter, has 844.4M parameters versus 1,100.3M in the base language backbone.Source: §4 · p. 5
  • Training uses 60B text tokens, 493k image-caption pairs, and 660k instruction samples across the three stages.The stages use Muon on matrix parameters and AdamW on the remainder; distillation is used in S0 and S2.Source: §4 · p. 5
  • The retrofit outperforms the baselines on average over nine benchmarks while storing 23.3% fewer language-model parameters.Source: Table 1 · p. 6

5 EVALUATION

Across nine VQA benchmarks, the retrofitted recurrent VLM outperforms the baselines on average, with gains concentrated on reasoning-oriented datasets. Ablations assess recovery stages, recurrence design, distillation, depth flexibility, cache sharing, and qualitative examples.

5 EVALUATION; 5.3 QUALITATIVE EXAMPLES

  • Evaluation covers nine VQA benchmarks; Figure 2 shows one item per reasoning benchmark where RR-VLM is correct and all trained non-recurrent baselines are wrong.The evaluation defines Avg as the arithmetic mean across the nine benchmarks.Source: §5 · p. 6, §5.3 · p. 9, Figure 2 · p. 9

5.1 COMPARISON TO BASELINES

  • RR-VLM outperforms every baseline by at least 2.3 pp on average, with gains concentrated on reasoning-oriented datasets and basic visual-understanding performance on par with baselines.The reasoning-oriented gains include MMBench, RealWorldQA, SEED-Bench, AI2D, and CV2D.Source: §5.1 · pp. 6–7

5.2 ABLATIONS

  • Removing text-only recovery S0 lowers average performance by 3.68 pp, while removing alignment stage S1 costs 1.28 pp, with S0 having the larger impact.The authors attribute the losses to later stages having to repair the pruned language backbone or perform both alignment and instruction tuning.Source: §5.2 · p. 7
  • At fixed recurrent depth, the 4/8/4 allocation beats 6/4/6 by 3.19 pp and the narrower 4/6/4 configuration by 1.60 pp.The comparison uses a fixed 56 recurrent layers and a 7.5B-token S0 budget.Source: §5.2 · p. 7
  • The linear adapter performs best; the adapter-free variant is only 0.73 pp behind it and beats the gated and Parcae adapters by 1.31 pp and 0.53 pp.These variants are compared on a 4/6/4 architecture with r=6 and a 7.5B-token S0 budget.Source: §5.2 · p. 7
  • Block looping beats per-layer looping by 2.83 pp at r=2 and 5.40 pp at r=6, while per-layer looping loses performance as recurrence count increases.The comparison uses a 4/6/4 architecture and a 7.5B-token S0 budget.Source: §5.2 · pp. 7–8
  • S2 distillation improves performance most when applied to both image-text and text-only data, gaining 1.48 pp over the undistilled recurrent model.The corresponding unpruned baseline gains 0.25 pp from S2 distillation.Source: §5.2 · p. 8, Table 4 · p. 8
  • Sampled recurrence stays within 2 pp of its best value from r=2 to r=16 but does not scale beyond its trained depth; cache sharing trades memory for accuracy.At n=3 the cache is 1.45× the base model's with a 0.75 pp cost; at n=1 it is 0.727× with a 5.43 pp cost.Source: §5.2 · pp. 8–9, Table 6 · p. 9

6 CONCLUSION

The prune-then-loop retrofit outperforms the unpruned baseline with 23.3% fewer language-model parameters, with gains concentrated on reasoning-oriented benchmarks. Its recovery depends on text-only training, and it outperforms the route that adds a vision encoder to a pretrained recurrent language model.

6 CONCLUSION

  • 23.3% fewer language-model parameters accompany performance gains over the unpruned baseline, concentrated on reasoning-oriented benchmarks.The conclusion attributes the gain over the pruned single-pass baseline to recurrence itself.Source: §6 · p. 9
  • The text-only training stage is required for recovery and yields the best performance among the evaluated training strategies.The three-stage recovery also restores projector alignment using the base model’s vision-language training data.Source: §6 · p. 9
  • Retrofitting recurrence into a pretrained VLM outperforms adding a vision encoder to a pretrained recurrent language model among the evaluated baselines.Source: §6 · p. 9

A.1 TRAINING HYPERPARAMETERS

Training uses Muon for matrix parameters and AdamW for other parameters across all stages, with different schedules and workloads for language-only pretraining, projector alignment, and full-model fine-tuning.

A.1 TRAINING HYPERPARAMETERS

  • All stages use Muon for matrix parameters and AdamW for other parameters, with weight decay 10^-4, gradient clipping at 1.0, and 5% warmup.Source: §A.1 · p. 14
  • S0 runs for 228,874 steps on 16 GPUs at 60B tokens, while the ablations’ 7.5B runs use 28,611 steps.S0 uses a batch of 128 sequences of 2,048 tokens and decays the learning rate to zero over the last 35% of steps.Source: §A.1 · p. 14
  • S1 trains the projector for 6,426 steps and S2 trains the whole model for 6,408 steps, both on 4 GPUs with one pass over the data.S1 uses a batch of 96 image-text pairs; S2 uses a batch of 128, including interleaved text batches in the step count.Source: §A.1 · p. 14

A.2 DISTILLATION PLACEMENT AT THE 7.5B BUDGET

The ablation varies the stages in which the distillation term is applied and its weight, using a 4/6/4 layer allocation and r=6 at the 7.5B S0 budget.

A.2 DISTILLATION PLACEMENT AT THE 7.5B BUDGET

  • The placement ablation tests distillation in S0, S1, and S2; in S0 only; or in S0 and S2.Source: §A.2 · p. 14
  • For the S0-and-S2 condition, the tested distillation weights are λ=0.50 and λ=0.25.Source: §A.2 · p. 14
  • The table covers the 4/6/4 layer allocation with r=6 at the 7.5B S0 budget, and its runs are separate from the 4/6/4 row of Table 3(b).The table caption specifies Seed 42.Source: Table 7 · p. 14

A.3 KV-CACHE SHARING DURING TRAINING

The training ablation studies KV-cache sharing across passes for the 4/8/4 model at the 7.5B S0 budget, comparing key sharing with sharing both keys and values.

A.3 KV-CACHE SHARING DURING TRAINING

  • The models are trained with sharing active, using the 4/8/4 layer allocation at the 7.5B S0 budget.Source: §A.3 · p. 14
  • The cache ablation compares sharing keys across passes with sharing both keys and values.Source: Table 8 · p. 14
  • Table 8 reports KV cache per token as a multiple of the base model’s cache and uses Seed 42.Source: Table 8 · p. 14

A.4 TEXT INTERLEAVING IN S2

Stage S2 interleaves text-only batches with multimodal batches. The reported comparison finds K=4 performs better than removing text batches or using K=2 or K=6.

A.4 TEXT INTERLEAVING IN S2

  • Removing text batches costs 0.69 pp, while K=2 and K=6 cost 0.20 pp and 0.07 pp, respectively, relative to K=4.Table 9 varies the number of multimodal batches per text batch on the 60B S0 budget and the 4/8/4 layer allocation, without text distillation.Source: §A.4 · p. 14, Table 9 · p. 15

A.5 EARLY EXIT

The early-exit section describes an adaptive halting criterion and compares read-out-only with compute-skipped exits. Qualitative examples show the proposed model answering selected reasoning-benchmark items correctly when three trained non-recurrent baselines answer incorrectly.

A.5 EARLY EXIT

  • The threshold ϵ sets the accuracy–compute tradeoff; read-out-only reads from pass k⋆ after computing every pass, whereas compute-skipped omits later passes.Table 10 reports both forms on GQA against the same model without exit.Source: §A.5 · p. 15
  • Fixed-depth models do not settle, so the early-exit rule almost never fires on them, with mean pass reads of 6.00 and 5.92.Table 10 defines settling as successive readouts having KL divergence below 5·10^-3 at pass 6.Source: Table 10 · p. 15

A.6 QUALITATIVE EXAMPLES

  • Figures 3 and 4 include two further items from each of four reasoning-oriented benchmarks where the proposed model is correct and three trained non-recurrent baselines are wrong.The figures show answers from every Table 1 model; check marks denote correct options.Source: §A.6 · p. 15, Figure 3 · p. 16, Figure 4 · p. 16

Related papers

Loading 2610.22075v1…