PaperlayerView exact revision on arXiv

Source-linked AI summary

Self-Routed Tensor Adapters for Parameter-Efficient Universal Visual Adaptation

Suraj Yadav

arXiv:2608.16384v1cs.CVcs.LG

TL;DR

Adapting frozen vision backbones across heterogeneous domains is difficult because fixed low-rank subspaces and externally routed experts limit input-dependent specialization while adding routing parameters. SRTA self-routes through a shared Tucker core and achieves competitive or slightly stronger average accuracy than MoE-style PEFT baselines with substantially fewer parameters across five benchmarks.

  • Problem

    Fixed low-rank subspaces restrict heterogeneous-domain adaptation, while external routers add parameters and separate routing from the adapted representation.

  • Method

    SRTA derives routing from low-rank input representations to blend domain-indexed slices of a shared Tucker core, with progressive depth-weighted supervision across adapter layers.

  • Results

    Across five multi-domain visual classification benchmarks, SRTA achieves competitive or slightly stronger average accuracy than MoE-style PEFT baselines using substantially fewer trainable parameters.

  • Takeaways & Limitations

    SRTA offers an accuracy-parameter trade-off for multi-domain visual adaptation without large independent expert banks or an external gating network.

  • Takeaways & Limitations

    Evaluation is limited to multi-domain image classification, and routing supervision currently uses known domain labels rather than unknown or continuously changing domains.

Abstract

from arXiv · show

Universal visual representations require adaptation mechanisms that adapt across heterogeneous domains without fragmenting knowledge into domain-specific modules. Parameter-efficient fine-tuning adapts frozen visual foundation models efficiently, but standard low-rank adapters use a fixed subspace for all inputs, which can be restrictive when domains differ in style, background, and semantic context. MoE-based adapters improve specialization through multiple expert pathways, but often rely on external routers and large expert banks, adding parameters and separating routing from adaptation. We propose \textbf{Self-Routed Tensor Adapters}, a compact framework for multi-domain visual adaptation. SRTA projects each input into a low-rank space, computes routing weights from this representation using a learnable domain matrix, and uses these weights to blend slices of a shared Tucker core. This produces a sample-specific adaptation matrix without an external gating network, allowing shared visual factors to be reused while supporting domain-aware specialization. To strengthen pathway learning, we introduce a progressive depth-weighted routing objective that supervises routing decisions across adapter layers. Across five heterogeneous multi-domain visual classification benchmarks, SRTA achieves competitive or slightly stronger average accuracy than MoE-style PEFT baselines while using substantially fewer trainable parameters. At rank 64, SRTA uses 2.77M parameters in the 4-domain setting compared with 9.52M for MoLoRA, and 3.00M in the 6-domain setting compared with 14.31M. Overall, SRTA offers an effective accuracy-parameter trade-off for adapting visual foundation models toward universal multi-domain representations. \href{https://github.com/surajyadav-research/SRTA}{GitHub}

1 Introduction

SRTA addresses heterogeneous multi-domain visual adaptation with a shared, structured tensor space and intrinsic routing derived from the adapter’s own low-rank representation. It combines input-conditioned Tucker adaptation with progressive routing supervision to support specialization while preserving compactness.

  • Motivation: Fixed low-rank adaptation subspaces can be too restrictive across heterogeneous domains, causing negative transfer and uneven optimization.Different domains vary in visual style, background context, acquisition conditions, and object appearance.
  • Motivation: MoE-based adapters improve specialization but typically require separate gating networks, adding routing-specific parameters and separating expert selection from adaptation.Examples include MoLoRA, MOELoRA, and MALoRA.
  • SRTA: A shared Tucker core tensor is dynamically blended per input to produce a sample-specific low-rank adaptation matrix rather than isolated domain experts.The shared core acts as a common adaptation basis while supporting domain-aware specialization.
  • SRTA: SRTA computes routing probabilities directly from the adapter’s low-rank representation and a learnable domain-coordinate matrix, eliminating external gating.This intrinsic routing uses the same representation for routing and adaptation.
  • Training objective: Progressive depth-weighted routing supervision provides direct signals to intermediate adapter-layer decisions, strengthening reusable and discriminative pathways.The objective mitigates weak supervision in early layers and helps reduce domain interference.
  • Results: Across five multi-domain visual classification benchmarks, SRTA achieves competitive or slightly better average accuracy than MoE-style PEFT baselines with substantially fewer parameters than MoLoRA.This establishes the framework’s intended accuracy-parameter trade-off.

2 Related Work

Related work frames universal visual adaptation through domain generalisation, parameter-efficient fine-tuning, expert-based specialization, and tensor factorization. These directions motivate compact methods that preserve transferability while adapting efficiently across heterogeneous domains.

  • Domain Generalisation: Domain generalisation evaluates whether visual representations transfer across heterogeneous or unseen conditions, using benchmarks including PACS, VLCS, Office-Home, Digits-DG, and NICO++.These benchmarks vary in style, objects, and appearance.
  • Parameter-Efficient Fine-Tuning: Parameter-Efficient Fine-Tuning updates a small parameter subset while freezing the backbone; Prefix-Tuning, Prompt-Tuning, and LoRA represent major approaches.LoRA learns trainable low-rank updates to frozen weight matrices.
  • Mixture-of-Experts Adaptation: Mixture-of-Experts adapters combine low-rank efficiency with specialized pathways, including MoLoRA, MOELoRA, and MALoRA for multi-domain or multi-task adaptation.These methods use routing or gating mechanisms, with MALoRA sharing parts of the low-rank structure across experts.
  • Tensor Decomposition: Tensor decompositions compactly represent high-dimensional parameter spaces through multilinear factors, with CP, Tucker decomposition, and FacT explored for efficient adaptation.FacT applies tensor-based factorization to lightweight adaptation in vision transformers.

3 Methodology

SRTA is a compact input-conditioned adapter that derives routing from its own low-rank representation and blends slices of a shared Tucker core. Progressive depth-weighted routing supervision strengthens layer-wise pathway learning while domain labels remain training-only.

  • Training and inference: Domain labels are used only during training, while inference computes routing directly from input representations without domain labels or additional inference-time parameters.The auxiliary routing supervision therefore improves pathway learning without changing inference inputs.
  • Self-routed tensor adaptation: SRTA derives domain-aware routing intrinsically from the adapter’s low-rank representation, avoiding a separate external gating network.Routing and feature adaptation are coupled within the same representation space.
  • Self-routed tensor adaptation: The adapter uses input and output projections, a shared Tucker core, and a domain-coordinate matrix whose columns define learnable pathway coordinates.The number of core slices r3 is set to the number of benchmark domains in the experiments.
  • Routing and adaptation: Pooled patch-token representations are compared with domain coordinates, softmax routing weights are computed, and weighted core slices form a sample-specific adaptation matrix.This softly combines shared and domain-specific factors rather than selecting independent expert matrices.
  • Routing supervision: Progressive depth-weighted routing supervision assigns smaller weights to shallow layers and larger weights to deeper layers, providing direct signals to intermediate routing decisions.The final objective combines classification loss with the auxiliary weighted routing loss.

4 Experiments

Across five heterogeneous multi-domain visual benchmarks, SRTA achieves competitive accuracy with self-routing and a shared Tucker core while using substantially fewer adaptation parameters than MoE-based baselines. Ablations show benefits from routing supervision, stable rank scaling, and domain-dependent pathway sharing.

  • Experimental Setup: Experiments use five heterogeneous benchmarks, an 80/20 split with 2,000 training and 400 validation images, and a frozen ViT-B/16 backbone with only adapters and the classifier trained.The benchmarks are PACS, VLCS, Office-Home, Digits-DG, and NICO++; each domain is treated as a distinct task or pathway.
  • Main Results: 88.1% average accuracy at rank 64 makes SRTA the strongest overall method, slightly ahead of MoLoRA while avoiding an external gating network.SRTA performs best on PACS and VLCS, remains strong on Digits-DG and NICO++, and is slightly below MoLoRA on Office-Home.
  • Parameter Efficiency: 2.77M and 3.00M trainable parameters are required by SRTA in the 4-domain and 6-domain settings, versus 9.52M and 14.31M for MoLoRA at rank 64.Parameter counts exclude the classifier and reflect adaptation-module cost only.
  • Routing Supervision: 87.1% to 88.1% average accuracy results from adding routing supervision, with the clearest gains on VLCS, Office-Home, and NICO++ while PACS remains unchanged.The comparison uses λroute = 0.0 versus λroute = 1.0 at rank 64.
  • Rank Scaling: Rank-32 SRTA is already competitive, and rank 64 gives its strongest overall result, reflecting more stable scaling than methods whose gains can saturate or become non-monotonic.MoLoRA is highly competitive at rank 32 but does not consistently improve at rank 64.
  • Routing Behavior: Routing is sharp and nearly diagonal for separable domains but softer for Office-Home and NICO++, indicating distinct pathways alongside shared Tucker-core slices.VLCS shows largely domain-specific routing with some sharing between related domains.

5 Conclusion

SRTA provides compact, input-conditioned adaptation by deriving routing from low-rank representations and blending slices of a shared Tucker core without an external router. Across five benchmarks, it matches or slightly exceeds MoE-style PEFT accuracy with substantially fewer parameters while learning domain-aware pathways.

  • Framework: SRTA derives routing probabilities from each adapter’s low-rank representation and blends shared Tucker-core slices to produce input-conditioned adaptation.The framework avoids an external MoE router and operates within a compact shared parameter space.
  • Results: Across five multi-domain visual classification benchmarks, SRTA achieves competitive or slightly stronger average accuracy than MoE-style PEFT baselines with substantially fewer trainable parameters.Its reported strength is the accuracy-parameter trade-off.
  • Results: 3.4× fewer parameters than MoLoRA in the 4-domain setting and 4.8× fewer parameters in the 6-domain setting are achieved at rank 64.These reductions accompany competitive or slightly stronger average accuracy than MoE-style PEFT baselines.
  • Routing analysis: SRTA learns sharper routing for visually distinct domains and softer sharing for overlapping domains, yielding interpretable domain-aware pathways.The analysis supports shared tensor-factorized adaptation rather than separate expert modules.

6 Limitations

SRTA is currently evaluated on multi-domain image classification, uses known domain labels for routing supervision, and leaves broader visual tasks and adaptive routing for future work.

  • Task and routing limitations: SRTA has not yet been extended to detection, segmentation, video understanding, or vision-language learning.These broader visual tasks remain an important direction for future work.
  • Task and routing limitations: Current experiments use known domain labels for routing supervision, limiting direct applicability to unknown or continuously changing domains.Future extensions may explore unsupervised or adaptive routing mechanisms.

7 LLM Usage

The authors used an LLM only for grammatical and stylistic editing, with all changes reviewed by the authors.

  • The LLM assisted only with grammatical and stylistic editing, while the authors reviewed all changes and retained responsibility for the final manuscript.
Loading 2608.16384v1…