PaperlayerView exact revision on arXiv

Source-linked AI summary

Rank-Aware Hyperbolic Alignment for Vision-Language Dataset Distillation

Jongoh Jeong, Sun-Kyung Lee, Kuk-Jin Yoon

arXiv:2606.29464v1cs.CVcs.AI

TL;DR

VLDD must compress paired image–text data despite low-rank cross-modal correlation and hierarchical semantics that make uniform Euclidean alignment overly restrictive. RAHA addresses this with rank-aware hyperbolic alignment, matching dominant shared structure while regulating residual variation; it improves stability and cross-architecture transfer under tight compression budgets.

  • Problem

    Uniform full-dimensional Euclidean alignment can suppress modality-private information because image–text correlation is concentrated in a low-dimensional shared range with weakly correlated residual variation.

  • Method

    RAHA lifts representations into hyperbolic space and combines hyperbolic contrastive alignment with adaptive range–residual relevance distillation and residual regularization.

  • Results

    RAHA improves stability and cross-architecture transfer under tight compression budgets, while combined range and residual matching yields the best ablation performance.

  • Takeaways & Limitations

    Selective preservation of dominant shared structure and regulation of weaker residual structure is most useful when the synthetic budget is very small.

  • Takeaways & Limitations

    RAHA does not explicitly detect or remove bias, and its range-preserving objective may retain harmful correlations concentrated in dominant singular directions.

Abstract

from arXiv · show

Vision-language dataset distillation (VLDD) compresses a large image-text paired dataset into a small set of synthetic pairs that can efficiently train contrastive vision-language models under strict data and compute budgets. Most existing methods match expert trajectories or cross-modal statistics, yet still enforce full-dimensional alignment in a Euclidean embedding space. This is often overly restrictive due to rank-deficient image--text correlation, with shared semantics concentrated in a low-dimensional range and remaining variation spread across a weakly correlated residual subspace. LoRS relaxes alignment at the similarity level by low-rank factorization, but does not explicitly control dominant alignment capacity and structure in the representation space. We thus propose a rank-aware hyperbolic alignment (RAHA) that combines hierarchical geometry with explicit alignment-capacity control. RAHA lifts multimodal representations to hyperbolic space and optimizes distilled pairs with asymmetric objectives that enforce geodesic alignment in the shared range while regularizing the residual subspace to preserve modality-private diversity and improve transfer robustness. Experiments on benchmarks show that RAHA demonstrates competitive cross-modal retrieval and improved transfer indicators under fixed budgets.

1 Introduction

Vision–language dataset distillation seeks compact, auditable surrogates that retain multimodal training utility while reducing the burdens of web-scale paired data. RAHA addresses the low-rank and hierarchical structure of image–text correlations through rank-aware hyperbolic alignment.

  • Web-scale image–text datasets improve VLM capability but increase privacy, licensing, provenance, and poisoning burdens.
  • Dataset distillation synthesizes small example sets that approximate large-dataset learning signals and can reduce sensitive-data footprints.
  • Existing VLDD methods include trajectory-based and generative approaches, but trajectory matching is costly and may inherit teacher architectural bias.
  • Low-rank image–text correlation concentrates shared semantics in a dominant range, while uniformly aligning residual directions can suppress complementary information and harm transfer.
  • RAHA matches synthetic relevance in the shared range, regularizes residual interactions, and adds hyperbolic contrastive learning to preserve pairwise discriminability.

2 Related Work

Prior dataset-distillation work progresses from unimodal signal matching toward multimodal methods that must preserve cross-modal correspondence. RAHA differs by combining rank-aware decomposition with hyperbolic geometry instead of uniform Euclidean alignment.

  • Unimodal distillation methods match gradients, distributions, trajectories, information criteria, transport objectives, or diffusion-generated data.
  • VLDD must preserve both intra-modal diversity and cross-modal correspondence, while existing approaches address modality collapse and cross-modal statistics with varied supervision signals.
  • RAHA is trajectory-free and selectively aligns shared range structure while preserving modality-private diversity in a hyperbolic representation space.

3 Proposed Method

RAHA distills paired image–text data by combining hyperbolic contrastive learning with rank-aware relevance matching. It decomposes real cross-modal coupling into dominant range and residual subspaces, matching the former and regularizing the latter.

  • 3.1 Preliminaries and Distillation Protocol: RAHA jointly optimizes hyperbolic contrastive loss and rank-aware relevance distillation over compact synthetic image–text pairs.
  • 3.1 Preliminaries and Distillation Protocol: VLDD seeks a synthetic set whose training preserves bidirectional image-to-text and text-to-image retrieval behavior.
  • 3.1 Preliminaries and Distillation Protocol: Synthetic images are optimized in pixel space and captions through continuous token embeddings, enabling differentiable optimization of both modalities.
  • 3.2 Hyperbolic Contrastive Learning: Hyperbolic lifting replaces Euclidean similarity with Lorentz-hyperboloid geodesic distance, supporting hierarchical semantic relations in symmetric contrastive learning.
  • 3.3 Distilling Cross-Relevance via Range and Residual Subspaces: RAHA factorizes real tangent-space cross-covariance, selects an effective rank by captured singular-value energy, and reuses the resulting range bases for real and synthetic features.
  • 3.3 Distilling Cross-Relevance via Range and Residual Subspaces: The final objective combines hITC, range matching, and residual matching with regularization that prevents weak residual interactions from dominating compression.

4 Experiments

Experiments evaluate RAHA on standard image–caption retrieval benchmarks under fixed synthetic budgets, alongside transfer, robustness, and qualitative analyses. RAHA consistently beats coreset selection, remains competitive with trajectory-matching methods, and shows stronger transfer robustness at intermediate budgets.

  • Experimental Setup: RAHA is evaluated on Flickr8k, Flickr30k, and MS COCO using bidirectional image–text retrieval with 100, 200, or 500 synthetic pairs.All methods use the same budget, architecture, and evaluation protocol.
  • Main Results: Across all datasets and budgets, RAHA consistently outperforms coreset selection and remains competitive with trajectory-matching baselines.The comparison includes MTT-VL, TESLA, LoRS, and RepBlend.
  • Main Results: At N=500 on Flickr8k, RAHA achieves the best overall mean recall, while on Flickr30k and COCO it improves with increasing synthetic budget.These results indicate stronger retrieval performance as synthetic capacity increases on more diverse datasets.
  • Qualitative Comparison: RAHA produces cleaner textures, more natural edges, and better-preserved image–text relational content than CovMatch in qualitative comparisons.CovMatch is described as retaining high-frequency artifacts, banding, and occasional mismatched captions.
  • Architectural Transfer and Robustness: RAHA improves cross-architecture transfer robustness at N=100 and raises mean transfer from 7.2 to 8.7 at N=200.Improvements are reported across BERT and DistilBERT targets and across vision backbones, with lower average degradation under image- and text-side perturbations.

5 Auxiliary Discussion

Auxiliary analyses examine RAHA’s component contributions, computational cost, budget dependence, and rank-aware design. The range term provides the strongest single-component gain, while the full controlled objective performs best but becomes more expensive at larger batch sizes.

  • Ablation Study: Adding the range relevance term to hyperbolic contrast improves mean retrieval more than using the residual term alone.The residual branch is weaker in isolation because residual interactions are harder to match without the dominant shared structure.
  • Ablation Study: The full range–residual objective performs best, indicating that residual matching is most effective when explicitly controlled by subspace regularizers.The ablation supports a primary role for the shared range and a controlled secondary role for the residual branch.
  • Discussion: RAHA does not claim uniform dominance at extreme compression, where too few pairs may fail to populate decomposed semantic modes.The discussion states that MTT-style methods can remain strong at the smallest budgets, while RAHA improves as the number of pairs increases.

6 Conclusion

The paper concludes that RAHA uses hyperbolic geometry and rank-consistent relevance matching to improve stability and cross-architecture transfer under tight compression budgets. Its scope remains bounded by teacher-encoder expressivity, caption quality, and dataset hierarchy.

  • Conclusion: RAHA lifts image–text representations into hyperbolic space and enforces rank-consistent cross-modal relevance matching.The method is presented as a geometry-aware objective for vision-language dataset distillation.
  • Conclusion: RAHA improves stability and cross-architecture transfer under tight compression budgets.This is the paper’s stated conclusion about the method’s principal benefits.
  • Limitations: Performance may degrade with domain-shifted or noisy captions, weak hierarchical structure, or limited teacher-encoder expressivity.The paper identifies these as limitations and scope boundaries.

A1 Scope of Vision-Language Dataset Distillation

Vision-language dataset distillation compresses paired data while preserving the cross-modal structure needed for retrieval learning. Existing approaches preserve this structure indirectly or align feature statistics too uniformly, motivating relevance compression that prioritizes shared relations.

  • Scope: Image–text distillation must preserve cross-modal relational structure in addition to within-modality diversity.The distilled set is trained from scratch and evaluated primarily through bidirectional image–text retrieval.
  • Existing Methods: Trajectory-based methods transfer optimization dynamics but require storing long trajectories and do not explicitly identify the most important correspondence components.Distribution-based methods are more storage-efficient but commonly align Euclidean statistics uniformly across representation directions.
  • Problem Formulation: Cross-modal relevance compression prioritizes shared structure that matters for efficient and transferable retrieval learning.The objective is not to reproduce every aspect of the original multimodal feature distribution.

A2 Motivation and Additional Method Details

RAHA combines hyperbolic lifting with tangent-space range–residual decomposition to prioritize information-rich shared structure during distillation. This design reflects hierarchical image–text semantics and the limited capacity of very small synthetic sets.

  • Why introduce hyperbolic lifting?: Hyperbolic lifting provides an inductive bias for hierarchical image–text semantics spanning coarse-to-fine concepts and relations.It is not intended to impose a literal tree, but to accommodate hierarchical variation more naturally than flat Euclidean similarity.
  • Why decompose features in the hyperbolic tangent space?: RAHA performs range–residual decomposition in the hyperbolic tangent space, combining lifted geometry with stable Euclidean linear operations.The tangent space supports centering, SVD, and projection while retaining the geometry induced by hyperbolic lifting.
  • Logarithmic map: The logarithmic map sends hyperboloid points to the tangent plane, where Euclidean linear algebra can be applied.Its curvature-dependent radial reweighting compresses large-norm features more than small-norm features and approaches the identity as c →0.
  • Why prioritize shared structure?: RAHA matches dominant shared directions while regulating weaker residual structure because uniform residual alignment can amplify noise and modality-specific artifacts.The shared component carries stable retrieval-oriented relevance, whereas the residual component may contain low-energy or unstable interactions.
  • Why prioritize shared structure?: Selective alignment is most important at very small synthetic budgets, when the distilled set cannot reproduce every degree of freedom in the real data.RAHA therefore allocates alignment capacity to dominant shared signals and regulates weaker residual structure.
  • Relation to prior methods: RAHA differs from uniform Euclidean statistics matching and low-rank similarity methods by explicitly separating dominant shared directions from weaker residual ones.Its stated advantage is selective allocation of alignment capacity to the most informative shared structure.

A3 Experimental Details

The experiments evaluate RAHA under standardized retrieval and transfer protocols across datasets, budgets, and qualitative settings. RAHA remains competitive at small budgets and shows stronger gains as dataset complexity or synthetic capacity increases.

  • Datasets and evaluation: Experiments use Flickr8k, Flickr30k, and COCO to evaluate behavior across increasing dataset scale and diversity.Flickr8k represents a small, clean setting; Flickr30k is mid-scale; COCO is larger and more diverse.
  • Reproducibility comparison: Under the reproduced protocol, RAHA outperforms CovMatch at 200 and 500 pairs on all three datasets and remains competitive at 100 pairs.The reported pattern is consistent across Flickr8k, Flickr30k, and COCO.
  • Scale and budget effects: At 500 pairs on the larger-scale comparison, the mean score improves from 5.1 to 8.1.The results associate larger gains with broader source distributions and sufficiently expressive synthetic budgets.
  • Prompted classification: RAHA improves prompted Top-1 accuracy over CovMatch by +1.05 pp on CIFAR-100, +1.90 pp on CUB-200-2011, and +2.17 pp on Stanford Cars.On ImageNet-1K, RAHA matches CovMatch with +0.26 pp.
  • Qualitative results: Qualitative comparisons report cleaner visual structure and better fine-grained image–text agreement for RAHA than the strongest Euclidean statistics-based baseline.The comparison attributes this pattern to preserving dominant shared structure while controlling weaker residual interactions.

A4 Further Analyses and Discussion

The analyses examine RAHA’s feature geometry, component contributions, geometry ablations, and computational profile. RAHA improves matched-pair compatibility and global cross-modal organization, while its gains depend on hyperbolic lifting and its explicit range–residual machinery adds batch-level computation.

  • Feature-association visualization: RAHA concentrates matched pairs near zero radial gap and lower hyperbolic distance, indicating more compatible placement at similar hierarchical depth than CovMatch.CovMatch shows broader, negatively shifted radial-gap distributions and higher-distance regimes.
  • Feature-association visualization: RAHA produces a more coherent global cross-modal geometry, with more compatible radial arrangement, stable progression, and less distortion from weak residual interactions.The visualization supports relative coarse-to-fine preservation rather than exact recovery of a semantic tree.
  • Component ablation: The dominant shared range provides the primary retrieval signal, while the residual branch contributes best as a controlled secondary refinement anchored to range supervision.The full objective performs best; stronger range supervision and mild residual compression are favored.
  • Geometry ablation: 1.4/3.0/2.2 IR/TR/RMean is obtained by all-Euclidean geometry, versus 18.1/21.7/19.9 for eITC and 19.0/21.9/20.4 for fully hyperbolic hITC.The same Sinkhorn relevance-matching pipeline is used across geometry variants.
  • Compute profile: RAHA’s full Flickr8k N=100 distillation takes approximately 1.3 GPU-hours, compared with approximately 0.14 GPU-hours for CovMatch, while peak memory remains approximately 9.3 GB for both.The added cost is associated with lifting, tangent-space cross-covariance and SVD, range–residual projection, and Sinkhorn iterations.
  • Compute profile: The compute overhead comes primarily from explicit subspace decomposition and permutation-invariant matching, rather than being fundamental to hyperbolic formulation alone.The analysis identifies SVD and Sinkhorn operations as dominant batch-level costs.

A5 Broader Societal Impact

The societal-impact discussion frames distillation as a way to reduce data and training burdens while emphasizing that compression does not automatically improve fairness, safety, or robustness. RAHA’s range-prioritizing decomposition introduces additional scope and auditing concerns when signal is domain-shifted or harmful correlations occupy preserved directions.

  • Benefits and inherited risks: Compact distilled sets can reduce storage, transfer, and repeated training costs and limit redistribution of raw paired data when privacy, licensing, or provenance concerns apply.These benefits can improve experimentation accessibility and reproducibility.
  • Benefits and inherited risks: Distillation can preserve harmful correlations, stereotypes, geographic imbalance, and annotation artifacts from source data, so changing data form does not automatically change fairness or safety properties.Multimodal image–language interactions may amplify these risks.
  • RAHA-specific considerations: RAHA prioritizes the dominant range subspace, so harmful correlations there may be retained; residual suppression may remove harmful residual correlations only incidentally.The method provides no explicit bias-detection or bias-removal mechanism.
  • Technical limitations: RAHA depends on pretrained encoders and can degrade under strong domain shift, noisy captions, abstract descriptions, or broadly distributed shared signal.Excessive residual suppression may remove task-relevant information when useful signal spans many weak directions.
  • Responsible use: RAHA is positioned as efficient, structured compression rather than a guarantee of fairness, neutrality, or robustness beyond the tested setting.Future work is suggested on bias auditing, provenance constraints, and safer data filtering.
Loading 2606.29464v1…