PaperlayerView exact revision on arXiv

Source-linked AI summary

Recursive CLIP: Vision-Language Distillation with Looped Transformer Blocks

Szymon Mazurek, Alberto Presta, Michał Byra

arXiv:2610.34563v1cs.CVcs.AI

TL;DR

The paper asks whether CLIP can preserve transferable capability while reusing transformer blocks across depth. It studies recursive encoders through direct training, distillation, and downstream evaluation; results show substantial recurrence is tolerated, with effects varying by encoder and task. The authors frame recurrent block reuse as a practical design space with parameter efficiency arising from shared architecture.

  • Problem

    It is unclear whether CLIP needs distinct transformer parameters at every depth to preserve transferable capabilities.

    Source: §1 · p. 1

  • Method

    The authors apply a smaller set of blocks recurrently in vision-only rCLIP or joint vision–text r²CLIP, keeping computational depth fixed, and study direct training and teacher distillation.

    Source: §3.1 · p. 3

  • Results

    Substantial recurrence preserves much of CLIP’s transferable capability, but its effects vary by encoder and downstream task.

    Source: §5 · p. 7

  • Takeaways & Limitations

    Recurrent block reuse offers a practical, flexible CLIP design with parameter efficiency arising from fewer distinct blocks and adapters.

    Source: §1 · p. 2, §5 · p. 8

    Most experiments use a single training run, small differences warrant caution, and adaptation protocols inherited from non-recursive models may be hyperparameter-sensitive.

    Source: §5 · p. 8

Abstract

from arXiv · show

Contrastive vision-language models such as CLIP typically rely on independently parameterized transformer blocks across depth. We investigate the extent to which these blocks can instead be reused recurrently while preserving the transferable capabilities of CLIP. We study recursive transformer backbones in which a small set of $N$ transformer blocks is applied $M$ times per forward pass. By holding $N \cdot M$ fixed to the depth of a standard ViT-B, our models preserve computational depth while progressively increasing parameter sharing. We consider two recursive CLIP model families: rCLIP, where only the vision encoder is recursive, and r$^2$CLIP, where recurrence is also introduced in the text encoder. We study these models under both direct contrastive training and through knowledge distillation from a strong ViT-B/16 teacher pretrained on DFN-2B, using the substantially smaller CC3M and CC12M datasets. We evaluate the resulting models on zero-shot classification and cross-modal retrieval, and further study their adaptability through few-shot learning, visual grounding, and concept unlearning. Our results show that CLIP can tolerate substantial recurrent block reuse, although its effect depends on the encoder and downstream task. Recurrence also provides parameter efficiency by construction, since fewer distinct transformer blocks need to be stored and adapted. Our results establish recurrent block reuse as a practical design space for vision-language models and highlight its implications for knowledge transfer, downstream adaptation, and parameter efficiency.

1. Introduction

The paper asks whether CLIP needs distinct transformer parameters at every depth to retain transferable capabilities. It studies recurrent block reuse in vision-only and joint vision–text encoders, including its effects on distillation, downstream adaptation, and parameter efficiency.

1. Introduction

  • The central question is whether independently parameterized blocks at every depth are necessary to preserve CLIP’s transferable capabilities.The paper frames recurrence as a controlled way to study depth-wise parameter sharing and representation quality.Source: §1 · p. 1
  • Recurrent block reuse reduces the number of unique transformer blocks, and adapters attached to shared blocks are reused across recurrent executions.The paper presents this as parameter and adaptation efficiency, not as a claim that every downstream task benefits equally.Source: §1 · pp. 1–2
  • The study compares rCLIP, with a recursive vision encoder, and r²CLIP, with recursive vision and text encoders.Both use N distinct blocks applied M times, with N · M matching the reference encoder’s depth.Source: §1 · pp. 1–2
  • The authors examine direct contrastive training and distillation from a DFN-2B ViT-B/16 teacher, alongside classification, retrieval, few-shot learning, grounding, and unlearning.They also ask whether distillation objectives for conventional students suit models that reuse blocks.Source: §1 · p. 2

2. Related work

Prior work explores parameter sharing and efficiency in transformers, including vision and vision–language models. This paper instead studies recurrence directly in CLIP’s encoders at fixed computational depth, across distillation and downstream tasks.

2. Related work

  • Earlier CLIP compression work includes cross-modal affinity mimicking, weight inheritance, progressive distillation, teacher–student transfer, hybrid backbones, and token pruning.The paper contrasts these approaches with retaining effective ViT computational depth while replacing depth-specific blocks with shared recurrent blocks.Source: §2 · p. 2
  • Recursive transformer research includes recurrent layers in the Universal Transformer, cross-layer sharing in ALBERT [17], and looped computation for reasoning and scaling effective depth [8].These examples establish recurrence and weight sharing as alternatives to independently parameterized depth.Source: §2 · p. 2
  • Vision-transformer approaches vary in how they share weights: SReT and MiniViT add transformations, while bViT reuses standard blocks and Raptor distills recurrent approximations.The cited works explore reducing layer-specific parameterization through recurrent computation.Source: §2 · p. 2
  • Prior vision–language sharing work includes cross-layer and cross-modal sharing with pruning and recurrent adapters on frozen CLIP encoders, rather than recurrent CLIP backbones.This work varies distinct blocks and recurrent executions at fixed computational depth in vision-only and joint vision–text CLIP encoders.Source: §2 · p. 2

3. Methods

The methods replace distinct transformer blocks with a smaller set reused across depth, then distill recursive students and evaluate them across classification, retrieval, and adaptation tasks. The configurations preserve reference computational depth while reducing distinct parameters.

3. Methods

  • The experimental pipeline builds recursive CLIP variants, distills recursive and non-recursive students, then evaluates classification, retrieval, grounding, and unlearning.The overview also includes zero-shot and few-shot classification.Source: §3 · p. 3, §3.2 · p. 4, §3.3 · p. 4

3.1. Recursive transformers

  • Recursive encoders use N distinct blocks repeatedly applied M times, with shared parameters and no iteration-specific embeddings; the final representation uses the standard CLIP output mapping.For ViT-B/16, N · M = 12, so the configurations execute 12 blocks per forward pass and are FLOP-matched to the non-recursive reference while using fewer distinct parameters.Source: §3.1 · p. 3

3.2. Model training and knowledge distillation

  • Distillation transfers a DFN-2B ViT-B/16 teacher into rCLIP, r²CLIP, and a randomly initialized non-recursive reference using CC3M and CC12M image-caption pairs.The evaluated objectives include feature distillation, contrastive relational distillation, interactive contrastive learning, and activation matching; the activation-matching alignment assumes recurrent executions correspond to the teacher’s layer-wise trajectory.Source: §3.2 · p. 4

3.3. Downstream tasks

  • Evaluation covers zero-shot classification and retrieval, few-shot CLIP-LoRA, HiVG visual grounding, and DIET concept unlearning.Few-shot learning uses 16 shots per class; grounding adapts on five datasets, and unlearning reports forget-class and full-test accuracy averaged over five seeds.Source: §3.3 · p. 4

4. Experiments

Experiments compare recursive configurations under direct CLIP training and DFN-2B distillation, then evaluate selected distilled models on downstream tasks. Results vary by task: more unique blocks generally improve classification, while grounding is relatively insensitive to recurrence.

4.1. Baseline CLIP training

  • Without distillation, rCLIP ImageNet accuracy rises from 32.8% with one block to 36.5% with six; rCLIP and r²CLIP are nearly identical from two blocks onward.Performance also increases with unique blocks on shifted datasets and retrieval, with gains saturating after several distinct blocks.Source: §4.1 · p. 5, Figure 3 · p. 5

4.2. Choosing the distillation objective

  • FD is the strongest individual distillation signal on both architectures, while AM retains 50% of Reference CLIP accuracy on rCLIP 1b12i and is excluded from later experiments.The selected subsequent objective uses λCRD = 1, λICL = 0, and λFD = 104.Source: §4.2 · p. 5, Table 2 · p. 6

4.3. Distillation results

  • At one block, distilled ImageNet zero-shot accuracy is 56.7% for rCLIP and 52.0% for r²CLIP versus 72.5% for ViT-B; at six blocks, the gap is 1.1 points.Flickr I2T reaches 85.4 for rCLIP and 83.7 for r²CLIP, indicating a larger retrieval gap. The comparison is shown with the teacher and a non-recursive distilled reference.Source: §4.3 · pp. 5–6, Figure 4 · p. 6

4.4. Downstream task adaptation

  • At 16 shots, rCLIP 6b2i scores 72.13 on ImageNet versus 75.18 for the reference, and comes within one to two points on several datasets.Zero-shot gaps are larger on fine-grained datasets: Food101 is 75.39 versus 81.05, and Aircraft is 12.87 versus 16.59. The table reports zero-shot and 16-shot accuracy across ten datasets.Source: §4.4 · p. 6, Table 3 · p. 7
  • Grounding shows little difference between recursive and non-recursive models or between rCLIP and r²CLIP, with little effect from block count.On RefCOCO testB, rCLIP 6b2i reaches 78.73 compared with 78.85 for the distilled ViT-B; on ReferItGame it reaches 69.41, slightly exceeding the ViT-B references at 68.26–68.98.Source: §4.4 · p. 6
  • DIET lowers recursive models’ average forget-set accuracy to 5.1–15.2%, while post-unlearning test accuracy generally benefits from retaining more unique blocks.Average test accuracy falls by 22.2 points for the distilled ViT-B and 14.1 for the OpenAI checkpoint; matched rCLIP and r²CLIP post-DIET test accuracy is within 0.6 points at each shown block count.Source: §4.4 · p. 7

5. Conclusions

The conclusions find that substantial block reuse preserves much of CLIP’s transferable capability, with effects that vary by encoder and downstream task. Recurrence reduces distinct parameters, while the authors flag single-run comparisons and adaptation protocols inherited from non-recursive models as limitations.

5. Conclusions

  • Increasing block reuse reduces distinct parameters, and the cost of text-tower recurrence becomes small as more distinct blocks are retained.The authors describe this as a controllable trade-off between parameterization and representation quality.Source: §5 · p. 7
  • Recursive models remain highly competitive for grounding and benefit strongly from few-shot adaptation, while heavily shared configurations can be more sensitive during unlearning.The effect of recurrence depends on the downstream task.Source: §5 · p. 7
  • Block reuse also reduces the number of distinct LoRA adapters required for downstream adaptation.The grounding table reports accuracy at IoU ≥0.5 following HiVG adaptation; the unlearning table reports zero-shot and post-DIET results.Source: §5 · p. 8, Table 4 · p. 8, Table 5 · p. 8
  • Most experiments use a single training run, so small performance differences should be interpreted cautiously.The authors also note that downstream adaptation protocols were designed for non-recursive models, so their performance may be sensitive to hyperparameter choices and recursive-specific tuning could improve results.Source: §5 · p. 8

Supplementary Material

The supplementary material defines the training and distillation losses and explains how activation matching aligns recursive student states with a teacher trajectory. The objectives supervise final embeddings, cross-modal batch relations, or intermediate vision states.

A.1. CLIP Contrastive Loss

  • CLIP’s InfoNCE objective pulls paired image–text embeddings together and pushes unpaired embeddings apart, averaging image-to-text and text-to-image losses.The dot product measures similarity, and τ is a learnable temperature.Source: §A.1 · p. 11

A.2. Feature Distillation (FD)

  • Feature Distillation (FD) minimizes mean squared error between the teacher’s and student’s L2-normalized image and text features.All students preserve the ViT-B/16 embedding width, so the optional projection head is not used.Source: §A.2 · p. 11

A.3. Contrastive Relational Distillation (CRD)

  • Contrastive Relational Distillation (CRD) transfers teacher cross-modal similarity distributions by matching teacher and student distributions with KL divergence in both directions.The distributions are defined over the mini-batch.Source: §A.3 · p. 11

A.4. Interactive Contrastive Learning (ICL)

  • Interactive Contrastive Learning (ICL) contrasts student image features with teacher text features and student text features with teacher image features.Minimizing its loss maximizes a lower bound on mutual information between teacher and student representations.Source: §A.4 · p. 11

A.5. Activation Matching (AM)

  • Activation Matching (AM) aligns intermediate student and teacher vision states at matching execution indices, with each reused student block supervised against multiple teacher states.The student and teacher trajectories have equal length because N · M = L; all considered students use the identity map for width alignment, and the loss compares unnormalized full-token hidden states with CLS and patch tokens weighted equally.Source: §A.5 · pp. 11–12

A. Distillation loss details

  • The full objective combines the CLIP contrastive loss with selected distillation terms, while activation matching supervises intermediate vision-tower representations.The stated main-experiment settings use α = 0, λCRD = 1, λICL = 0, and λFD = 104.Source: §A.5 · pp. 11–12

B. Extended distillation objective results

This section reports supplementary distillation-objective experiments on a single recursive configuration, using a fixed training setup and ImageNet accuracy as the primary comparison.

B. Extended distillation objective results

  • The extended experiments cover component ablations, Activation Matching weight sensitivity, and a weighting sweep over retained distillation components.They support the analysis of the distillation objective in Section 4.2.Source: B. Extended distillation objective… · p. 12
  • All runs use CC3M, a shared training budget and optimizer settings matching the main distillation runs, and a single seed.The recursive backbone is rCLIP 1b12i, the most constrained configuration in the grid.Source: B. Extended distillation objective… · p. 12
  • Configurations are compared by ImageNet accuracy, with three distribution-shifted evaluation variants reported for completeness.The section seeks to improve performance for the selected recursive backbone.Source: B. Extended distillation objective… · p. 12

B.1. Component ablation

The extended component ablation preserves the reported ordering across evaluation sets: feature distillation is strongest individually, while activation matching behaves very differently for reference and recursive backbones.

B.1. Component ablation

  • FD is the strongest individual component on both backbones; CRD and ICL fall well short of the full composition, while AM ranks second on the reference and last on the recursive student.This ordering is preserved on every evaluation set in the extended table.Source: §B.1 · p. 12
  • FD alone scores 37.3 versus 37.0 for the full composition on reference ImageNet, but costs 2.1 points on the recursive student.On ImageNet-Sketch, FD alone scores 27.9 versus 27.0 for the full composition on the reference backbone.Source: §B.1 · p. 12
  • AM reaches 47.5 on reference ImageNet-R versus 46.9 for the full composition, but scores 17.0 on the recursive backbone versus 33.2 for the full composition.AM is the weakest component on the recursive backbone across all four reported sets.Source: §B.1 · p. 12

B.2. Sensitivity of Activation Matching to its weight

Activation Matching (AM) is highly weight-sensitive on the non-recursive reference but has little effect on the recursive student across the tested weight range. This suggests the recursive result is not explained simply by an insufficient AM weight.

B.2. Sensitivity of Activation Matching to its weight

  • A 2000× increase in α raises ImageNet accuracy from 20.4 to 31.0 on the reference, but shifts recursive-student accuracy from 15.7 to 15.5.On the recursive student, changes were at most 1.1 points across the evaluated sets.Source: §B.2 · p. 12, Table 7 · p. 13
  • The sensitivity test evaluates AM as the only distillation term alongside the primary CLIP contrastive loss, using a single seed on CC3M.The caption specifies α = 2000 for the AM setup; the sensitivity comparison also evaluates α = 1.Source: §B.2 · p. 12, Table 7 · p. 13
  • AM therefore provides essentially no usable signal for the tested recursive student across the 2000× weight sweep, unlike on the reference.The authors interpret this as consistent with Eq. (16)'s constraint being unsatisfiable for a looped block, rather than merely weak.Source: §B.2 · pp. 12–13

B.3. Loss weighting

The recursive-student loss search found that increasing FD weight improved ImageNet accuracy throughout the explored range, while ICL and CRD had smaller effects. The selected recipe also improved the non-recursive reference under the same budget.

B.3. Loss weighting

  • ImageNet accuracy rises from 26.7 to 32.0 as FD weight increases from 500 to 5000, and reaches 33.8 at 10^4.No turning point was observed before the search stopped for budget reasons; the optimum was therefore bounded from below but not bracketed.Source: §B.3 · p. 13, Table 8 · p. 13
  • At FD weight 10^4, removing ICL yields 33.8 ImageNet top-1 versus 33.6 with ICL at half weight, while CRD changes accuracy by at most 0.6 points across weights 0.5–2.At FD weight 5000, ICL weights 2, 1, 0.5, and 0 yield 31.3, 32.0, 32.7, and 32.5, respectively.Source: §B.3 · p. 13, Table 8 · p. 13
  • The selected recursive recipe is CRD = 1, ICL = 0, and FD = 10^4; it is also strongest on ImageNet-V2 and ImageNet-Sketch.ImageNet-R instead favors CRD = 2, ICL = 0.5, and FD = 5000, at 36.8, but ImageNet top-1 was the selection criterion.Source: §B.3 · p. 13, Table 8 · p. 13
  • Under the same budget, the selected configuration raises Reference CLIP ImageNet top-1 from 37.3 to 47.7.The configuration was verified on the non-recursive backbone before adoption, and the authors state the recipe was not tuned in favor of recursive models.Source: §B.3 · p. 13

B.4. Limitations

The extended evaluations report results across zero-shot and retrieval, few-shot classification, and visual grounding, alongside limitations on interpreting the loss search and its transfer. Across these tasks, outcomes vary with the number of blocks and evaluation setting.

B.4. Limitations

  • Every configuration in the loss search was trained once, so differences of a few tenths of a point should not be over-interpreted.The selected configuration was transferred from the CC3M search to combined CC3M and CC12M without re-verification, and two changes were made together without training the intermediate point.Source: §B.4 · pp. 13–14

C.1. Zero-shot classification and retrieval

  • Under direct CLIP training, rCLIP recovers 2.7 of its total 3.7-point ImageNet gain between one and two blocks, while distilled rCLIP leads across most datasets at matched configurations.The supplementary section provides the full grids underlying the main-text figures.Source: §C.1 · p. 14

C.2. Few-shot classification

  • Across ten datasets, few-shot adaptation narrows the zero-shot gap to Reference CLIP from 12.5 to 3.2 points for rCLIP and from 16.4 to 4.5 for r²CLIP as blocks increase.Adaptation narrows the gaps further, and the average rCLIP–r²CLIP gap falls from 3.9 points zero-shot to 0.3 after adaptation at matched block counts.Source: §C.2 · p. 14
  • EuroSAT has non-monotone performance and 47–56-point adaptation gains for recursive backbones, while Aircraft remains far from the reference after adaptation.The authors connect EuroSAT's gains to a possible domain gap and Aircraft's remaining gap to limited fine-grained signal surviving distillation on CC3M and CC12M.Source: §C.2 · p. 14

C.3. Visual grounding

  • Across ten grounding splits, recursive-model accuracy spans 71.9–75.3 for rCLIP and 72.1–75.6 for r²CLIP, against 75.7–76.6 for non-recursive references.The best recursive configurations come within 0.2 points of the strongest reference on RefCOCO testB and RefCOCO+ val, and exceed it on ReferItGame; neither recursive family dominates overall.Source: §C.3 · p. 14

C.4. Concept unlearning

The concept-unlearning evaluation measures both forgetting and test-set retention after DIET. Forgetting is effective across backbones, while recursive configurations show a broad range of retention and some particularly low-retention cases.

C.4. Concept unlearning

  • After DIET, average forget-set accuracy falls from 73.0–88.7% to 4.0–15.2%, with no systematic difference between recursive and non-recursive backbones.Retention is 57.6–74.4% across the recursive grid, compared with 82.8% for OpenAI, 80.5% for Teacher, and 72.2% for Reference CLIP.Source: §C.4 · pp. 14–15
  • The rCLIP 2b6i configuration combines the lowest average post-DIET forget-set accuracy in the grid, 4.0%, with the second-lowest retention.Its per-dataset test accuracies have standard deviations up to 17.8 points over five seeds; the authors attribute this to DIET's learning-rate sensitivity, not architecture.Source: §C.4 · p. 15
  • The authors caution that individual configurations should not characterize their block count, and report the weaker pattern that heavily shared backbones retain less on average than non-recursive references.The selection criterion is a harmonic mean of forgetting efficacy and retention, and efficacy saturates near complete forgetting.Source: §C.4 · p. 16

D.1. CLIP and distillation training configuration

The supplementary sections specify training and downstream adaptation protocols, with recurrent block sharing affecting how adapter parameters are allocated and unlocked.

D.1. CLIP and distillation training configuration

  • CLIP and distillation runs share a training recipe and use the combined CC3M and CC12M datasets, with 13,874,493 image–text pairs, for 32 epochs at effective batch size 1024.The runs differ in trained architecture; optimization uses AdamW with a maximum learning rate of 1 × 10−3.Source: §D.1 · p. 16

D.2. Visual grounding configuration

  • For grounding, Hi-LoRA stages vision adapters across distinct blocks while text adapters unlock together, and shared blocks are adapted at every recurrent execution.The grounding recipe freezes pretrained encoders and trains only adapters, bridges, fusion transformer, and prediction heads; recursive-model bridges attach to executions rather than layers.Source: §D.2 · pp. 17–18, Table 14 · p. 19

D.3. Few-shot adaptation configuration

  • r²CLIP at N=1 uses 15,360 adapter parameters versus 184,320 in the non-recursive control, and all configurations stay at or below 0.13% of backbone parameters.rCLIP uses 82,944–129,024 adapter parameters from N=1 to N=6; r²CLIP uses 15,360–92,160, and its most constrained configuration uses an order of magnitude fewer adapters than the control.Source: §D.3 · p. 19

D.4. Unlearning configuration

  • DIET-based unlearning removes one randomly selected class using a 16-shot forget set, with no retain data and a 30-step budget, across three datasets and five seeds.Its retain term is a surrogate based on similarity structure within the forget batch, rather than retained examples.Source: §D.4 · p. 20

Related papers

Loading 2610.34563v1…