PaperlayerView exact revision on arXiv

Source-linked AI summary

Hyperbolic Multimodal Continual Learning: A Closest-Admissible Solution

Jiahong Liu, Ming Shen, Xiaohao Liu, Rex Ying, Menglin Yang, Tat-Seng Chua, Irwin King

arXiv:2609.29329v1cs.CVcs.AI

TL;DR

Existing hyperbolic multimodal continual-learning methods do not explicitly preserve the shared Lorentz relations that encode similarity and hierarchy. HMCL constrains updates through a shared hyperbolic isometry, with CA and MR variants, and reports improved performance and retention across backbones.

  • Problem

    Existing hyperbolic multimodal models mainly address static training, while preserving their shared similarity and hierarchy relations during sequential learning remains open.

    Source: §1 · p. 2

  • Method

    HMCL restricts updates to a first-order admissible family derived from a shared hyperbolic isometry, with CA permitting a shared rotation and MR fixing it to zero.

    Source: §1 · p. 2

  • Results

    Across backbones, both HMCL variants improve final performance and retention over external baselines, and CA achieves the highest Overall scores.

    Source: §1 · p. 2, §8 · p. 17

  • Takeaways & Limitations

    The reported gains coincide with more stable radial and relational structure, with CA strongest on Overall scores and MR on hierarchy scores.

    Source: §8 · p. 17

    The MR ablation uses stronger task anchoring than its main-table configuration, so the two settings represent different stability–plasticity trade-offs.

    Source: §7.6 · p. 16

Abstract

from arXiv · show

Existing continual-learning methods protect parameters, replayed examples, or Euclidean feature subspaces. When applied to hyperbolic multimodal models, they do not explicitly preserve the Lorentz geometry that jointly encodes within-modality similarity, cross-modal correspondence, and semantic hierarchy; sequential updates can therefore retain task scores while still distorting previously learned relations. We address this gap with Hyperbolic Multimodal Continual Learning (HMCL). We show that preserving the old multimodal geometry amounts to restricting all modalities to one shared hyperbolic isometry, which induces a family of admissible first-order parameter changes. We formulate a joint closest-admissible (CA) correction that retains the shared rotation best matching the candidate modal updates; its minimal-rotation (MR) special case fixes this rotation to zero. Both variants correct the displacement realized by AdamW, and task anchoring bounds within-task accumulation while preserving learning freedom. Across a unified 16-task classification-retrieval stream with three hyperbolic backbones, HMCL improves final performance and backward transfer over sequential fine-tuning and four continual-learning baselines; HMCL-CA gives the highest Overall score on every backbone. A modality-extended stream confirms the retrieval gains. Representation analyses find 81.2 to 95.5 percent less radial, angular, cross-modal, and paired-distance drift; ImageNet-WordNet results show better semantic ancestry and radial hierarchy.

1 Introduction

HMCL treats continual learning as preserving the shared hyperbolic geometry that encodes multimodal similarity and hierarchy while retaining freedom to learn new tasks. It formalizes admissible updates and evaluates them across hyperbolic backbones.

Motivation

  • Existing hyperbolic multimodal models mainly address static training, leaving preservation of shared Lorentz relations during sequential acquisition open.The relations include within-modality similarity, cross-modal correspondence, and radial ordering of semantic abstraction.Source: §1 · p. 2

Our Approach

  • HMCL relates retention of old multimodal geometry to a shared hyperbolic isometry and derives the corresponding first-order parameter updates.Its joint closest-admissible solution permits a shared rotation, whereas HMCL-MR fixes that rotation to zero; task anchoring bounds within-task accumulation.Source: §1 · p. 2

Contributions

  • Across three backbones, both HMCL variants outperform external baselines in final performance and retention, while CA achieves the highest Overall scores.The component study identifies post-AdamW correction with task anchoring as the strongest stability–plasticity configuration.Source: §1 · pp. 2–3

Extension beyond the conference version

  • The rotation-permitting CA update generalizes the zero-rotation MR update, and both correct the parameter displacement realized after AdamW moment estimation and preconditioning.The shared-rotation solution is obtained from one Lyapunov equation.Source: §1 · pp. 2–3

Hierarchy-aware large-scale evaluation

  • The evaluation adds ImageNet-1K to form a unified 16-task stream and uses WordNet structure to assess semantic ancestry and abstraction order.The hierarchy evaluation tests whether predictive-retention gains coincide with better preservation of those structures.Source: §1 · p. 3

Controlled component and trajectory analysis

  • Component and trajectory analyses find that correcting AdamW’s realized displacement is stronger than correcting its raw gradient, with task anchoring providing further stability.Representation-drift and hierarchy analyses also report less radial and relational distortion.Source: §1 · p. 3

2 Related Work

Related work covers hyperbolic multimodal representations, continual learning, and projected updates with adaptive optimizers. The paper distinguishes its focus on preserving shared Lorentz relations across sequential multimodal tasks.

2.1 Hyperbolic Multimodal Learning

  • Hyperbolic multimodal work uses geodesic proximity and radial position to represent similarity and specificity, but prior approaches focus on pretraining or task-specific applications.The surveyed work includes compositional entailment, angular cross-modal alignment, and hyperbolic multimodal adaptation.Source: §2.1 · p. 3

2.2 Continual Learning

  • Continual-learning methods use replay, regularization, parameter isolation, distillation, or adapters, while prior hyperbolic approaches do not enforce one shared Lorentz isometry across modalities and tasks.Continual vision–language learning also seeks to retain cross-modal alignment alongside modality-specific features.Source: §2.2 · pp. 3–4

2.3 Projected Updates with Adaptive Optimizers

  • With AdamW, projecting a gradient or descent direction need not equal projecting the realized parameter displacement because of momentum, element-wise preconditioning, and decoupled decay.The paper’s constraint concerns earlier outputs on a shared Lorentz-isometry orbit, rather than parameters lying on a manifold.Source: §2.3 · p. 4

3 Preliminaries

The preliminaries introduce the Lorentz realization of hyperbolic space and the multimodal similarity and entailment machinery used to represent hierarchy.

3.1 Hyperbolic Geometry

  • Hyperbolic space has constant negative curvature; the Lorentz model represents it as the upper sheet of a hyperboloid in Minkowski space.The Lorentz realization is used because it supports stable numerical optimization.Source: §3.1 · p. 4

3.2 Hyperbolic Multimodal Learning

  • The multimodal setup uses negative hyperbolic distance for pairwise similarity and entailment cones to encode a cross-modal partial order.Smaller spatial norms correspond to broader concepts with wider cones, while larger norms correspond to more specific concepts with narrower cones.Source: §3.2 · p. 5

4 Problem Formulation

The formulation models sequential tasks in one shared hyperbolic space and specifies geometric stability conditions alongside freedom to learn the current task.

4.1 Continual Hyperbolic Multimodal Learning

  • The task sequence uses a shared hyperbolic space, with each stage learning the current task while retaining earlier within-modality, cross-modal, and hierarchical relations.The setup follows a multimodal contrastive continual-learning protocol.Source: §4.1 · p. 5
  • Each modality uses a frozen pretrained encoder followed by a trainable Lorentz transformation at the task stage.The passage also describes processing previous-task representations through the stage transformation.Source: §4.1 · pp. 5–6
  • Under the preservation conditions, all modalities undergo one common hyperbolic isometry, and first-order radial preservation restricts permitted motion to a tangent direction.The figure associates geodesic proximity with semantic similarity and radius with specificity.Source: Figure 2 · p. 6

4.2 Geometric Stability and Plasticity

  • Stability is described through three invariants that protect within-modality relations, cross-modal correspondences, and radial levels in the entailment hierarchy.Together these preserve semantic similarity, modality alignment, and specificity.Source: §4.2 · p. 6
  • The stability conditions are paired with plasticity: the current task’s transformation layers learn new task-specific patterns and associations.The text frames this as retaining enough representation capacity for the current task.Source: §4.2 · p. 6
  • Applying the preservation conditions directly during gradient-based training is difficult, motivating an equivalent geometric view that yields practical parameter-update constraints.Source: §4.2 · p. 7

5 Geometric Conditions for Admissible Updates

The section characterizes geometry-preserving updates: under stated assumptions, one shared transformation across modalities preserves the old representations. Its first-order hierarchy condition requires spatial motion tangent to the current radial direction.

5.1 Characterizing Preserved Representations

  • Under the full-rank and identity-connected-path assumptions, preserving (P1), (P2), and (P3) is equivalent to one transformation acting on every modality.The proof identifies this as a shared hyperbolic isometry whose spatial part is a rotation.Source: §5.1 · p. 7
  • Full rank uniquely identifies the shared isometry over the ambient space, while practical HMCL protects a retained low-rank subspace without requiring global uniqueness.Source: §5.1 · p. 7

5.2 First-Order Preservation of Hierarchy

  • A first-order hierarchy-preserving change makes an old embedding’s spatial displacement orthogonal to its radial direction, avoiding radial drift.Source: §5.2 · p. 7
  • Hyperbolic distance tracks semantic similarity and radius tracks specificity; general updates may distort this geometry even when task loss is small.Source: §5.2 · p. 7

6 Hyperbolic Multimodal Continual Learning

The section transfers representation-space preservation constraints into parameter updates for Lorentz layers. HMCL projects AdamW’s realized steps onto a shared-isometry-admissible family and uses task anchoring to limit accumulation.

6.1 Geometrically Admissible HMCL Updates

  • Infinitesimal parameter changes are admissible only when one skew-symmetric generator produces the first-order spatial motion of every modality.This permits common coordinate rotation while excluding independent modal rotations that alter cross-modal relations.Source: §6.1 · p. 8
  • HMCL-CA jointly chooses admissible modal displacements and the shared rotation that best matches candidate updates, with the rotation obtained from one Lyapunov equation.Source: §6.1 · p. 8
  • HMCL-MR fixes the shared rotation to zero, selecting the zero-rotation member of the same closest-admissible family.The closest displacement induces no first-order motion on the protected old-feature subspace.Source: §6.1 · p. 8

6.2 Stepwise Correction and Task Anchoring

  • HMCL projects AdamW’s realized parameter displacement after optimization, removing only the component that violates the shared-isometry constraint, then anchors parameters toward the task start.Task anchoring limits cumulative drift while keeping updates admissible; the proposed AdamW displacement already includes its optimizer adjustments and weight decay.Source: §6.2 · p. 9

7 Experiments

Across three hyperbolic backbones, HMCL improves continual-learning performance and retention while reducing representation drift and preserving semantic hierarchy. The experiments compare task-level outcomes, geometric change, hierarchy measures, qualitative traversals, and update-placement choices.

7.1 Experimental Setup

  • The 16-task stream combines 14 multimodal classification tasks with COCO and Flickr30K retrieval, and compares HMCL with fine-tuning, regularization, replay, distillation, and null-space baselines.ImageNet-1K is inserted after Caltech-101; retrieval uses symmetric R@5, classification uses top-1 accuracy, and BWT is final minus immediate performance.Source: §7.1 · p. 10

7.2 Main Results (RQ1)

  • Both HMCL variants outperform every external baseline in final classification, retrieval, and Overall scores across three backbones; CA gives the highest Overall score in every backbone block.Their BWT values also move substantially toward zero. Table 1 summarizes the unified 16-task results over five paired seeds.Source: §7.2 · pp. 10–11, Table 1 · p. 11
  • On the modality-extended stream, both HMCL variants improve Overall and BWT over Vanilla on every backbone, while CA improves retrieval R@5 across backbones.In 36 fine-grained retrieval comparisons, HMCL-CA has the highest score; under alternative task orders, both variants retain higher scores and less negative BWT than Vanilla, but their relative ranking changes.Source: §7.2 · pp. 12–13

7.3 Quantitative Representation Drift (RQ2)

  • Both HMCL variants reduce radial, angular, cross-modal, and paired-distance drift across all 15 old tasks; CA has slightly lower mean drift, and every paired comparison favors HMCL.The paired comparisons are significant under an exact two-sided Wilcoxon signed-rank test (p = 6.1 × 10^-5; Bonferroni-adjusted p = 2.4 × 10^-4 over four measures).Source: §7.3 · pp. 13–14, Figure 5 · p. 14

7.4 ImageNet–WordNet Hierarchy (RQ3)

  • On HyCoCLIP-B, both HMCL variants outperform external baselines in ImageNet top-1 accuracy and every WordNet hierarchy metric, while MR leads on the hierarchy measures and CA on full-stream Overall.Figure 6 shows Vanilla contracting depth rings and disrupting branch layout, while HMCL stays closer to task-end geometry after global rotation alignment.Source: §7.4 · p. 15, Figure 6 · p. 15, Table 3 · p. 15

7.5 Representation-Space Case Study (RQ3)

  • In Flickr30K image-to-origin traversals, HMCL examples maintain local semantic continuity along the geodesic, with concepts becoming progressively more generic toward the origin.Examples include “hat → fashion → style” and “relaxation → vacation → peace,” unlike Vanilla’s abrupt off-path transitions.Source: §7.5 · pp. 15–16, Figure 7 · p. 16

7.6 Ablation Study (RQ4)

  • Correcting AdamW’s realized displacement performs better than correcting its raw gradient for both MR and CA; task anchoring adds retention, with Post+Pullback CA highest in Overall.Post+Pullback MR gives the strongest BWT and ImageNet retention. The MR ablation uses stronger task anchoring than its main-table configuration, representing a different stability–plasticity trade-off.Source: §7.6 · p. 16, Table 4 · p. 16

8 Conclusion

The conclusion presents shared geometric admissibility as a stability–plasticity principle for continual adaptation in hyperbolic multimodal models. It reports improved performance and retention, with HMCL-CA leading Overall scores and HMCL-MR leading hierarchy scores.

8 Conclusion

  • Across backbones, both HMCL variants improve final performance and retention over external baselines; CA leads Overall scores, while MR leads ImageNet–WordNet hierarchy scores.The analyses associate these gains with more stable radial and relational structure, and the component study finds geometric correction and task anchoring address complementary sources of change.Source: §8 · p. 17

B.1 Why Hyperbolic Geometry Represents Hierarchies

  • Hyperbolic space’s exponential volume growth supports low-distortion representations of taxonomies, trees, and knowledge graphs.The paper contrasts this with Euclidean volume growth, which is polynomial in radius.Source: §B.1 · p. 25

C Rationale for the Preservation Objectives

  • Preserving within- and cross-modal Lorentzian products retains semantic similarities, while preserving spatial norms maintains specificity and, together, the concept partial order.The cited rationale links entailment relations to these invariants.Source: C Rationale for the Preservation… · p. 27

D.1 Shared Isometry Induced by Gram Preservation

Under the stated rank and continuity assumptions, preserving the old representations’ within- and cross-modal relations determines a common Lorentz isometry. Preserving hierarchy further restricts that isometry to a spatial rotation.

D.1 Shared Isometry Induced by Gram Preservation

  • Preserving within- and cross-modal relations determines one Lorentz transformation shared across modalities, under the theorem’s assumptions.The argument uses full rank and a continuous stage-to-stage path beginning at the identity component.Source: §D.1 · p. 30

D.2 Removing the Boost Component

  • Preserving every old representation’s spatial norm eliminates nontrivial boosts, leaving only a spatial rotation.The result assumes a full-column-rank stacked representation matrix.Source: §D.2 · p. 31

D.3 First-Order Hierarchy Preservation

  • At first order, preserving hierarchy fixes the time-like coordinate and, through the hyperboloid constraint, requires the spatial perturbation to be orthogonal to the embedding.The derivation proceeds from the first-order change of the time-like coordinate.Source: §D.3 · pp. 31–32

E.1 Proof of Proposition 1

The section characterizes admissible parameter changes as first-order motions induced by one shared rotation. It derives closest-admissible and minimal-rotation corrections, then motivates correcting AdamW’s realized displacement rather than assuming its direction remains admissible.

E.1 Proof of Proposition 1

  • First-order admissibility requires every modality’s parameter displacement to correspond to the same skew-symmetric rotation generator.The representation motion is a spatial rotation, leaving spatial norms unchanged to first order.Source: §E.1 · p. 33

E.2 Derivation of Corollary 2

  • HMCL-CA jointly selects the closest admissible modal displacements and one shared rotation, solved uniquely through a Lyapunov equation.The derivation states a regularity condition for uniqueness.Source: §E.2 · p. 34

E.3 Proof of Corollary 3

  • HMCL-MR is the zero-rotation member of the admissible family and projects candidate spatial displacements onto the null space of the protected representation subspace.This is the minimum-rotation limit of the rotation-regularized closest-point construction.Source: §E.3 · p. 34

F Optimizer-Consistent HMCL: Derivations

  • Both variants correct the fully formed optimizer displacement; CA estimates a shared rotation, while MR fixes it to zero.The appendix describes these as a common spatial rotation preserving the three HMCL conditions.Source: F Optimizer-Consistent HMCL: Derivations · p. 36

F.1 A Concrete Adaptive-Optimizer Mismatch

  • AdamW’s positive diagonal preconditioner preserves every admissible direction only under a specific projector-commutation condition; decoupled decay is admissible only when absent or when parameters lie in the admissible subspace.The paper gives a two-dimensional counterexample in which element-wise scaling takes an admissible direction outside its subspace.Source: §F.1 · p. 36

F.2 Lorentz Strain of a Realized Optimizer Candidate

The section analyzes how optimizer steps depart from the shared-rotation tangent family and describes corrections, anchoring, and residual bounds. It also specifies the multi-task classification-retrieval stream used for evaluation.

F.2 Lorentz Strain of a Realized Optimizer Candidate

  • A realized first-order change preserves the three geometric conditions exactly when all modalities share one skew-symmetric generator, with no symmetric strain or residual motion.The rank condition supports uniqueness of the least-squares generator.Source: §F.2 · p. 37

F.3 Closest Projection of the Realized Step

  • For a realized AdamW candidate, the closest identity-isometry correction projects its spatial displacement away from the protected subspace and sets its time-like input displacement to zero.For protected features in the projector’s range, the finite-step update leaves the complete Lorentz representation and its derived relations unchanged.Source: §F.3 · p. 38

F.4 Task-Anchored Feasible Contraction

  • Task anchoring keeps each corrected update admissible and bounds within-task accumulation when realized optimizer displacements have a finite bound.The stated bound depends on the anchor coefficient and the bound on the realized displacement.Source: §F.4 · p. 39

F.5 Residual Drift under Low-Rank PCA

  • Low-rank PCA correction is exact on the retained subspace, while motion outside it is bounded in mean square by the spectral tail, not in the worst case.The anchor and spectral tail jointly bound average residual motion outside the retained subspace.Source: §F.5 · p. 39

F.6 Pseudocode, Optimizer State, and Complexity

  • MR’s correction costs O(dpr) time and stores O(pr) values; CA adds a Lyapunov solve costing O(d^3), reducible to O(r^3) with a rank-r realization.The PCA basis and task-level CA factors are estimated once per stage; optimizer state remains that produced by raw gradients.Source: §F.6 · p. 40

G.1 Dataset Details and Task Stream

  • The unified benchmark contains 16 datasets across image–text classification and bidirectional cross-modal retrieval, interleaved in one fixed sequential order.Non-replay methods train only on the current dataset; replay baselines access earlier tasks through bounded memory buffers.Source: §G.1 · p. 41, §G.1.3 · p. 41

G.1.1 Classification Datasets

  • The classification stream spans coarse and fine-grained recognition, textures, scenes, and language sentiment, with each dataset treated as a separate task.The listed stages include ImageNet-1K, CIFAR-10, Flowers-102, EuroSAT, and SST-2.Source: §G.1.1 · p. 41

G.1.2 Retrieval Datasets

  • COCO and Flickr30K supply paired-image-caption retrieval stages evaluated in both image-to-text and text-to-image directions.Table 7 reports stage-level dataset statistics and evaluation measures.Source: §G.1.2 · p. 41, §G.1.4 · p. 41, Table 7 · p. 41

G.2 Task-Order Robustness

The supplementary analyses test task-order robustness and examine per-task performance, representation drift, retrieval, and weight-decay sensitivity. HMCL retains advantages across these checks, with scope varying by diagnostic and protocol.

G.2 Task-Order Robustness

  • On seq2, HMCL-MR leads Overall and BWT while HMCL-CA leads final retrieval; on seq3, HMCL-CA leads Overall and HMCL-MR leads retrieval, and both beat Vanilla under both orders.No hyperparameters were retuned for either permutation.Source: §G.2 · p. 42, Table 8 · p. 42

G.3 Supplementary Materials for Case Study

  • In the Flickr30K geodesic case study, HMCL shows clearer movement from image-specific descriptions toward general concepts than Vanilla after the full stream.Examples use either a joint captions-and-concepts pool or captions alone.Source: §G.3 · p. 42

H.1 ImageNet–WordNet Protocol and Per-Task Results

  • HMCL variants achieve the highest final score on 15 of 16 tasks; C-FLAT remains strongest on PCAM, while the HMCL gains span classification and retrieval.HMCL-CA leads on ten classification tasks; HMCL-MR leads on ImageNet-1K, both retrieval datasets, Flowers102, and SST-2.Source: §H.1 · p. 45, Table 9 · p. 45

H.2 Modality-Specific Radial Contraction

  • HMCL-MR retains median image and text norms of 1.00× and 0.99×, respectively, while Vanilla retains 0.87× and 0.79×.Vanilla contraction appears in 14 of 15 image-task medians and all 15 text-task medians; HMCL-MR has no task median below 0.9× for either modality. The correction exactly preserves only the retained PCA subspace, so drift can remain outside it.Source: §H.2 · pp. 45–46

H.3 Stage-Wise Stability–Plasticity Diagnostics

  • Old-pair alignment deviations range from −0.074 to 0.002, and the higher-order terms remain nonpositive in the CMCL-protocol diagnostic.The paper identifies this as a theorem-specific consistency check, not a result on the current 16-task HMCL benchmark.Source: §H.3 · p. 46, Figure 9 · p. 46

H.4 Fine-Grained Retrieval on the Modality-Extended Stream

  • The modality-extended protocol adds audio and thermal retrieval tasks and reports COCO and Flickr30K text-to-image and image-to-text Recall@{1, 5, 10} across three backbones.This protocol is reported separately from the unified 16-task ImageNet experiment.Source: §H.4 · pp. 46–47, Table 10 · p. 47

H.5 Reproducibility Notes

  • All methods receive the same epoch-level parameter-update budget, with accumulated minibatch gradients followed by one AdamW step per epoch.For GEM, projection is applied once when the epoch gradient violates a constraint, before that step.Source: §H.5 · p. 47

H.6 Robustness to AdamW Weight Decay

  • Across weight decay values from 0 to 10^-2, HMCL-CA Overall remains 48.563–48.566 and BWT −0.461 to −0.456, and it is strongest in every condition.The ordering among Vanilla, MRpre, and HMCL-CA is unchanged across the tested range.Source: §H.6 · p. 48, Table 11 · p. 48

H.7 Clean Single-Stage Training Time

A clean single-stage timing diagnostic finds Vanilla and HMCL in the same computational regime. The pullback coefficient also shifts the measured stability–plasticity trade-off.

H.7 Clean Single-Stage Training Time

  • The timed fit_task call includes feature transfer, optimization, HMCL correction and pullback, and task-end state updates, but excludes cache loading, evaluation, and checkpoint serialization.The measurement is for ImageNet-1K, the third task, with a matched schedule and epoch-level optimizer-step budget.Source: §H.7 · p. 49, Figure 10 · p. 49
  • 172.4–194.6 seconds span the Vanilla and HMCL timings; average within-backbone ratios to Vanilla are 1.022× for HMCL-MR and 1.037× for HMCL-CA.With one timing run per pair, differences below 10% are not a statistically resolved ranking.Source: Figure 10 · p. 49, §H.7 · p. 49

H.8 Pullback Hyperparameter Analysis

  • Reducing β from 0.20 to 0.125 improves Overall score by 0.6% and mean classification accuracy by 1.2%, but lowers retrieval R@5 by 2.1%.The change also worsens forgetting, BWT, and final ImageNet retention.Source: §H.8 · p. 49

Related papers

Loading 2609.29329v1…