PaperlayerView exact revision on arXiv

Source-linked AI summary

GeoStack: A Framework for Quasi-Abelian Knowledge Composition in VLMs

Pranav Mantini, Shishir K. Shah

arXiv:2605.06477v1cs.CV

TL;DR

VLM knowledge composition must accumulate expertise across tasks without making joint training infeasible or forgetting prior knowledge. GeoStack independently trains geometrically constrained experts, composes them through stacking and weight folding, and reports preserved foundational performance, constant-time inference, and reduced forgetting across multi-domain and incremental-learning settings.

  • Problem

    Joint multitask training becomes infeasible as task numbers increase, while sequential fine-tuning risks catastrophic forgetting of previously learned knowledge.

  • Method

    GeoStack trains domain-specific GeoLayers with geometric constraints, composes them by sequential matrix multiplication, and folds the result into one projection matrix.

  • Results

    GeoStack preserves foundational and domain performance while maintaining O(1) inference complexity and significantly mitigating catastrophic forgetting across multi-domain and class-incremental experiments.

  • Takeaways & Limitations

    Independent experts can be composed into a unified VLM while retaining foundational knowledge and avoiding inference-cost growth with the number of experts.

  • Takeaways & Limitations

    As stacking deepens, accumulated perturbation can violate the stability condition, causing gradual domain-performance degradation or manifold collapse.

Abstract

from arXiv · show

We address the challenge of knowledge composition in Vision-Language Models (VLMs), where accumulating expertise across multiple domains or tasks typically leads to catastrophic forgetting. We introduce GeoStack (Geometric Stacking), a modular framework that allows independently trained domain experts to be composed into a unified model. By imposing geometric and structural constraints on the adapter manifold, GeoStack ensures the foundational knowledge of the base model is preserved. Furthermore, we mathematically demonstrate a weight-folding property that achieves constant-time inference complexity ($O(1)$), regardless of the number of integrated experts. Experimental results across multi-domain adaptation and class-incremental learning show that GeoStack provides an efficient mechanism for long-term knowledge composition while significantly mitigating catastrophic forgetting. Code is available at https://github.com/QuantitativeImagingLaboratory/GeoStack.

1 Introduction and Motivating Work

Knowledge composition combines expertise across tasks or domains, but existing VLM approaches face scalability, sharing, and catastrophic-forgetting challenges. GeoStack addresses these constraints by independently training and stacking geometrically constrained domain experts while preserving foundational knowledge and constant inference complexity.

  • Joint multitask training becomes infeasible as task count grows, while sequential fine-tuning is vulnerable to catastrophic forgetting.Catastrophic forgetting means failing to retain original knowledge after fine-tuning on new data.
  • Adapter methods reduce trainable parameters but generally isolate task knowledge or require fusion mechanisms for multitask composition.Prior approaches include shared frozen backbones, knowledge extraction and composition stages, and checkpoint fusion.
  • An ideal VLM composition framework should support independent training, modular integration, order-invariance, foundational preservation, and computational efficiency.These requirements avoid cross-domain or historical data, ensemble retraining, combinatorial ordering, capability degradation, and excessive architectural growth.
  • BiCLIP uses a learnable geometric transformation for domain adaptation, but its transformations are domain-specific and transfer poorly across domains.A DTD expert reaches 71.01% on DTD but 42.70% on EuroSAT, whereas a EuroSAT expert reaches 84.75% on EuroSAT but 41.44% on DTD.
  • GeoStack trains domain experts independently as geometrically constrained GeoLayers, then stacks them into a multi-expert model with zero additional inference complexity.The framework targets applicability across multiple tasks and folds composed experts into a single weight matrix with O(1) complexity.
  • The paper presents geometric constraints, stackability theory, constant-time weight folding, and empirical validation across multi-domain adaptation and class-incremental learning.The stated empirical goal is superior performance with resistance to catastrophic forgetting.

2 GeoStack Theory

GeoStack composes domain-specific geometric adapters while preserving previously established margins through structural constraints and small perturbations. Weight folding reduces the composed stack to constant-time inference, although sufficiently deep stacks can suffer margin erosion.

  • Composition and Stability: GeoStack composes experts sequentially as matrix products while requiring the composite margin to remain positive.For domains Da and Db, Wg = WaWb, with stability requiring Ma > 0 => Mg > 0.
  • Geometric Constraints: Upper-triangular closure keeps every composed operator within the same valid transformation class.Because upper-triangular matrices are closed under multiplication, Wtotal remains upper-triangular.
  • Geometric Constraints: Identity initialization represents each adapter as W = I + ∆, making domain expertise a geometric perturbation of the base model.The perturbation prior initializes each learnable adapter at the identity matrix.
  • Composition and Stability: Small perturbations keep inter-domain interference below the previously established margin, preserving positive composed margins.The guarantee depends on small spectral norms, with ∥ϵ∥ ≪ Ma when ∥∆b∥2 < δ.
  • Composition Properties: Quasi-additive composition preserves foundational and previously learned knowledge, while making expert order largely irrelevant.The framework approximates Wg ≈ I + Σ∆i and therefore WaWb ≈ WbWa.
  • Inference: Weight folding pre-computes the adapter stack into one projection matrix, yielding O(1) inference independent of the number of tasks.The effective projection remains structurally identical to the original CLIP projection and adds no deployment latency or memory footprint.
  • Limitations: Deep stacks can accumulate interference linearly until the stability condition fails, causing gradual domain-specific degradation or manifold collapse.This failure mode is termed Margin Erosion and occurs when ∥ϵtotal∥ < Ma is eventually violated.

3 GeoLayer

GeoLayers extend BiCLIP adapters with dual training objectives: domain alignment and geometric constraints for stable knowledge composition. Their combined loss balances task adaptation against stackability and preserves feature norms through near-isometric transformations.

  • GeoLayer Design: A GeoLayer is a geometrically constrained BiCLIP adapter trained to align a domain while preserving previously acquired knowledge.GeoLayers are designed for stable composition into a multi-domain GeoStack.
  • Alignment Objective: InfoNCE alignment loss learns a domain-specific transformation that matches transformed image features with their corresponding text embeddings.The transformed features are I′j = IjWi, and cosine similarity with temperature τ defines the alignment objective.
  • Stackability Objective: Orthogonality loss minimizes deviation from orthogonality, bounding perturbation size and keeping transformations near-isometric.This preserves feature norms during training and inference, with ∥IWi∥ ≈ ∥I∥.
  • Joint Optimization: The COA loss is a convex combination of alignment and stackability objectives, controlled by λ.As λ approaches 0, alignment is prioritized; as λ approaches 1, stability is prioritized.

4 Experimental Methodology

GeoStack is evaluated through modular multi-domain adaptation and class-incremental learning experiments using independently trained GeoLayers, stacked under geometric constraints. Across these settings, it preserves accuracy and foundational knowledge while reducing order sensitivity, forgetting, and orthogonality error.

  • Evaluation Setup: Experiments cover Multi-Domain Adaptation and Class-Incremental Learning using independently trained GeoLayers with a frozen OpenCLIP ViT-B/16 backbone.The evaluation includes diverse datasets and four-task CIFAR-100 incremental learning.
  • Knowledge Integration: GeoLayers are trained independently with a 16-shot protocol, then composed by multiplying their transformations into a unified GeoStack.For example, the sequence i → c → fo → e yields Wg = Wi · Wc · Wfo · We.
  • Stack Configurations: Depth-4 stacks span easy, moderate, and hard domain transitions to expose long-term stability under increasing geometric complexity.The configurations range from coarse-to-fine semantic progression to shifts from general images toward domain-specific visual content.
  • Multi-Domain Adaptation: 62.8% ImageNet accuracy with OE 0.013 is maintained by GeoStack in the Hard Stack, compared with BiCLIP’s decline to 52.6% as OE reaches 0.070.Task Arithmetic with constrained GeoLayers reaches 62.1% ImageNet accuracy, versus 49.0% for unconstrained BiCLIP in the Hard Stack.
  • Class-Incremental Learning: 69.47% accuracy is maintained after composing four incremental tasks, outperforming the baseline by 9.39%.GeoStack’s Task-0 retention loss is 2.12%, compared with 14.16% for BiCLIP and 3.92% for Task Arithmetic.
  • Permutation Stability: 84.49 ± 0.42% EuroSAT accuracy remains stable across stacking positions, supporting GeoStack’s Quasi-Abelian behavior under geometric constraints.The result is based on multiple stacking permutations of four domain experts.
  • Deep Stacking and Sensitivity: 56.00% initial-task accuracy is retained after deep stacking, while the BiCLIP baseline falls to 21.50%.GeoStack’s final ImageNet accuracy is 57.2%, a 19.4% margin over Task Arithmetic with BiCLIP; increasing λ reduces OE from 0.0332 to 0.0078 with a 2.6% accuracy impact.

5 Conclusion

GeoStack composes independently trained VLM domain experts while preserving the foundational CLIP latent space and mitigating catastrophic forgetting. Its weight-folding property reduces inference to a single O(1) projection matrix, and experiments show performance comparable to domain-specific models.

  • Conclusion: GeoStack uses Convex Orthogonality Alignment to create stackable GeoLayers that preserve the foundational CLIP latent space.The framework addresses knowledge accumulation and catastrophic forgetting in VLM adapters.
  • Conclusion: O(1) inference complexity is achieved by folding an arbitrary number of domain experts into a single projection matrix.This property makes inference complexity independent of the number of integrated experts.
  • Conclusion: Experiments on Multi-Domain Adaptation and Class-Incremental Learning show performance matching domain-specific models while significantly mitigating catastrophic forgetting.The conclusion summarizes the framework’s reported empirical scope.

A.1 Relation between Orthogonality Error and Spectral Norm

GeoStack represents each GeoLayer as an identity matrix plus an upper-triangular perturbation and links Orthogonality Error to perturbation stability. Minimizing Orthogonality Error suppresses spectral-norm growth, supporting stable composition of multiple experts.

  • A.1 Relation between Orthogonality Error and Spectral Norm: GeoLayers are initialized as identity matrices and constrained to be upper-triangular, with W = I + ∆.Here, ∆ represents the learned upper-triangular perturbation.
  • A.1 Relation between Orthogonality Error and Spectral Norm: Stability holds when the spectral norm of the perturbation ∥∆∥2 is small.
  • A.1 Relation between Orthogonality Error and Spectral Norm: Orthogonality Error is defined as the Frobenius norm of deviation from the identity matrix.
  • A.1 Relation between Orthogonality Error and Spectral Norm: When perturbations are small, the second-order term ∆⊤∆ is negligible in the approximation.
  • A.1 Relation between Orthogonality Error and Spectral Norm: Minimizing Lortho suppresses spectral-norm growth and enables multiple experts to be integrated without destructive interference.

B.1 Dual Stack Analysis

Dual-stack experiments compare GeoStack with BiCLIP using Orthogonality Error and ImageNet retention as stability measures. Across five sequences, GeoStack maintains higher foundational accuracy, especially when baseline error becomes large.

  • B.1 Dual Stack Analysis: Dual-stack stability is evaluated using Orthogonality Error and retention of foundational ImageNet knowledge.
  • B.1 Dual Stack Analysis: Across all five sequences, GeoStack consistently maintains higher ImageNet accuracy than the BiCLIP baseline.
  • B.1 Dual Stack Analysis: 0.050 OE in Sequence D causes a 7.1% ImageNet drop for BiCLIP relative to zero-shot, whereas GeoStack limits OE to 0.009 and preserves knowledge within 0.4%.

B.2 Triple Stack Analysis

Triple- and hexa-stack experiments test composition as more independently trained experts are folded into the backbone. GeoStack limits Orthogonality Error and preserves ImageNet or task accuracy better than BiCLIP in the reported challenging stacks.

  • B.2 Triple Stack Analysis: Triple-stack evaluation covers Easy, Moderate, and Hard compositions across three domain sequences.
  • B.2 Triple Stack Analysis: 55.2% ImageNet accuracy is reached by BiCLIP in the hard triple stack, while GeoStack records 63.6%, 8.4% higher than the baseline.
  • B.2 Triple Stack Analysis: GeoStack maintains an Orthogonality Error of 0.013 in the hard triple stack.
  • B.2 Triple Stack Analysis: In the hexa-stack, BiCLIP accumulates OE 0.1359, while GeoStack maintains OE 0.0142, ten times lower than the baseline.
  • B.2 Triple Stack Analysis: 86.3% Oxford-Pets accuracy and a +9.4% average-accuracy lead are reported for GeoStack in the hexa-stack.

C.1 Orthogonality Error as a Metric for Stackability

The paper calibrates normalized Orthogonality Error as a stackability metric by folding synthetic experts with controlled error into a frozen backbone. Accuracy remains stable below 0.015, declines gradually from 0.015 to 0.06, and degrades rapidly at or above 0.06.

  • C.1 Orthogonality Error as a Metric for Stackability: The normalized Orthogonality Error metric scales by 1/d^2 so error remains comparable across embedding dimensions.Here, d denotes the model dimensions.
  • C.1 Orthogonality Error as a Metric for Stackability: Synthetic experts with target OE γ ∈ [10^-5, 1.7] are folded into a frozen backbone and evaluated on EuroSAT.
  • C.1 Orthogonality Error as a Metric for Stackability: S < 0.015 defines a stable plateau where accuracy remains consistent and perturbations do not displace previous knowledge.
  • C.1 Orthogonality Error as a Metric for Stackability: 0.015 ≤ S < 0.06 defines a graceful-degradation zone with a 1% → 5% accuracy drop.
  • C.1 Orthogonality Error as a Metric for Stackability: S ≥ 0.06 defines a catastrophic-forgetting horizon where foundational knowledge is corrupted and performance rapidly degrades.
  • C.1 Orthogonality Error as a Metric for Stackability: Weight folding enables multiple domain experts to share a single O(1) inference operation, reducing computational energy and memory overhead.

D.2 Negative Societal Impact & Limitations

GeoStack inherits the biases and fairness issues of its CLIP backbone, and it does not inherently audit or filter the semantic content of stacked experts. Its transparent linear transformations are more amenable to weight-space auditing, while misuse could still involve stacking harmful or biased experts.

  • GeoStack inherits inherent biases and fairness issues from the CLIP backbone.
  • The framework does not inherently audit or filter the semantic content of stacked experts.
  • Intentional misuse could sequentially stack harmful or biased experts.
  • Transparent d × d linear transformations make GeoStack more amenable to weight-space auditing than deep black-box adapters.
  • Table 7 documents asset versions, licenses, and access URLs.

Appendix F: Compute Resources.

The experiments used a single NVIDIA GeForce RTX 2080 Ti with 11GB VRAM. Training a GeoLayer required approximately 8.2GB peak memory, while folded inference retained the vanilla projection-head footprint.

  • All experiments ran on a workstation with an NVIDIA GeForce RTX 2080 Ti and 11GB VRAM.
  • 8.2 GB peak memory was required to train a single GeoLayer with a ViT-B/16 backbone.
  • After weight folding, inference memory matched the vanilla CLIP-ViT-B/16 projection head.
  • 10 to 30 epochs were used per domain expert, averaging 25 minutes per expert on one RTX 2080 Ti.
Loading 2605.06477v1…