Source-linked AI summary
MC-RFM: Geometry-Aware Few-Shot Adaptation via Mixed-Curvature Riemannian Flow Matching
Salim Khazem, Ibrahim Mohamed Serouis, Zakaria Ezzahed
TL;DR
Few-shot adaptation methods often perturb frozen visual features in Euclidean space without modeling task-induced geometry. MC-RFM instead learns task-conditioned transport on a mixed-curvature manifold toward support prototypes, and it performs best in a majority of evaluated settings, especially for Transformer backbones and fine-grained tasks.
Problem
Existing parameter-efficient adaptation methods typically treat adaptation as a discrete Euclidean perturbation, despite hierarchical and locally discriminative structure in downstream visual tasks.
Method
MC-RFM projects frozen features onto hyperbolic and Euclidean factors, transports them task-conditionally toward support-set prototypes, and combines flow matching with a hybrid prototype-linear classifier.
Results
MC-RFM is the best-performing method in a majority of settings across seven benchmarks, five frozen backbones, and 1/4/16-shot regimes, with strongest gains on Transformer backbones and fine-grained datasets.
Takeaways & Limitations
Few-shot adaptation benefits from modeling both the geometry and dynamics of representation movement, not only which parameters are updated.
Takeaways & Limitations
Benefits depend on frozen-representation geometry, while support prototypes may be noisy in extreme low-shot regimes and hyperbolic design choices require control.
Abstract
from arXiv · showhide
Parameter-efficient adaptation of pretrained vision models is commonly performed through linear probes, prompts, low-rank updates, or lightweight residual modules. While effective, these methods usually treat adaptation as a discrete Euclidean perturbation of frozen representations, without explicitly modeling the geometry of the task-induced feature displacement. We propose \textsc{MC-RFM}, a mixed-curvature Riemannian flow-matching framework for few-shot adaptation of frozen visual backbones. The key idea is to represent adapted features on a product manifold combining a hyperbolic factor, which captures hierarchy-sensitive semantic structure, and a Euclidean factor, which preserves locally discriminative visual variation. Adaptation is formulated as a task-conditioned continuous transport from frozen features to support-set prototypes, trained with a flow-matching objective and coupled to a hybrid prototype-linear classifier. The method is lightweight, backbone-agnostic, and operates entirely on cached frozen features. Across seven visual recognition benchmarks, five frozen backbones, and 1/4/16-shot regimes, \textsc{MC-RFM} is the best-performing method in a majority of evaluated settings, with the strongest gains on Transformer backbones and fine-grained datasets. Ablations show that the mixed-curvature head, task conditioning, adaptive branch gating, prototype shrinkage, and discriminative supervision each contribute to performance. These results suggest that few-shot adaptation benefits not only from deciding which parameters to update, but also from modeling how representations should move through a geometry matched to the structure of the downstream task.
1 Introduction
MC-RFM reframes few-shot adaptation of frozen visual backbones as task-conditioned continuous transport in a mixed-curvature space. It combines geometry-aware representations, adaptive components, and discriminative supervision while evaluating broadly across benchmarks, backbones, and shot regimes.
- Existing few-shot adaptation methods typically treat updates as discrete Euclidean perturbations of frozen representations.
- MC-RFM projects frozen features into a product manifold combining hyperbolic and Euclidean factors, then transports them toward support-set prototypes.
- The framework uses task conditioning, adaptive representation and classifier balances, and a hybrid prototype-linear head while keeping hyperbolic states stable.
- MC-RFM is evaluated across seven benchmarks, multiple frozen backbones, and 1/4/16-shot regimes, with strongest gains on fine-grained tasks and Transformer backbones.
- Ablations attribute performance to the combination of mixed curvature, adaptive gating, task conditioning, prototype shrinkage, and joint flow-matching/classification supervision.
2 Related work
Related work spans few-shot adaptation of pretrained visual models, geometry-aware representation learning, and flow matching for continuous transport. MC-RFM connects these strands by applying mixed-curvature transport to frozen visual features.
- Few-shot adaptation has progressed from metric- and optimization-based methods toward parameter-efficient adaptation of strong pretrained visual backbones.
- Euclidean representations capture local visual variation, whereas hyperbolic geometry is suited to hierarchical, taxonomic, coarse-to-fine, and part-whole structure.
- Flow matching learns continuous vector fields along prescribed paths without simulation, while Riemannian variants extend the framework to manifold-valued representations.
- Prior flow-matching methods primarily target data-space generative modeling, and related feature-refinement methods remain Euclidean rather than exploiting hierarchical class structure.
3 Proposed method : MC-RFM
MC-RFM adapts frozen features by projecting them onto a hyperbolic–Euclidean product manifold, constructing support-derived prototypes and task context, and learning task-conditioned transport dynamics.
- 3.1 Problem statement: The problem is lightweight task-specific adaptation on cached features from a frozen pretrained visual backbone without updating its parameters.
- 3.1 Problem statement: MC-RFM separates hierarchy-sensitive structure and local visual variation into hyperbolic and Euclidean branches of a product manifold.
- 3.2 Mixed-Curvature Projection: The projection uses a lightweight bottleneck, normalized branch mappings, constrained hyperbolic scaling, and a latent state on the product manifold.
- 3.3 Class Prototypes and Task Context: Class prototypes are computed from support embeddings and shrunk toward the global support prototype to reduce low-shot variance.
- 3.3 Class Prototypes and Task Context: A task encoder summarizes prototype geometry, branch norms, pairwise distances, and class count into context that conditions the transport vector field.
- 3.4 Task-Conditioned Flow Matching: Flow matching transports support representations toward class prototypes using hyperbolic geodesic interpolation and Euclidean linear interpolation.
- 3.4 Task-Conditioned Flow Matching: The vector field receives chart-space hyperbolic states, Euclidean states, time embeddings, and task context, while the ODE state remains on the product manifold.
3.5 Adaptive Branch Gating
MC-RFM uses a task- and feature-adaptive gate to balance hyperbolic and Euclidean branches in both transport and classification, while combining flow matching with discriminative supervision. Its implementation maintains hyperbolic states inside the Poincaré ball and uses chart-space transport whose learned velocity recovers a conditional mean under the stated parameterization.
- Adaptive branch gating: The adaptive gate scales hyperbolic and Euclidean contributions in both the flow-matching objective and classifier, without removing either branch globally.The gate can reduce an unhelpful branch for a particular task or sample, but does not route samples through different ODE architectures.
- Hybrid classification: The hybrid head interpolates between calibrated product-distance prototype inference and a discriminative linear classifier.A task- and feature-dependent mixing coefficient controls the balance between the prototype and linear components.
- Training and inference: The training objective combines flow matching with cross-entropy supervision, whose label smoothing helps address brittleness from noisy support prototypes.At inference, support prototypes and task context are recomputed, while queries are transported with a low-NFE fixed-step ODE solver before hybrid classification.
- Hyperbolic stability: Projection after every hyperbolic update keeps solver states within the interior Poincaré ball, so distances and conformal factors remain away from the singular boundary.The invariant is established by induction over solver steps under the stated positive-curvature and projection assumptions.
- Flow-matching parameterization: In the origin chart, the target is the constant-speed velocity along a straight path from the current representation to the terminal representation, and squared-error training recovers its conditional mean.This parameterization is an approximation because the hyperbolic branch does not use an exact tangent field at every state.
4 Experiments
MC-RFM is evaluated across frozen backbones, datasets, shot regimes, and controlled ablations, with strongest performance patterns on Transformers and fine-grained tasks. Its components generally improve results, while excessive hyperbolic capacity harms performance.
- Experimental setup: MC-RFM is evaluated with five frozen backbones, three shot regimes, seven datasets, fixed splits, and three random seeds.The evaluation covers convolutional and Transformer architectures, 1-, 4-, and 16-shot settings, and varied recognition tasks.
- Main results: MC-RFM outperforms competing approaches in most experimental settings, with improvements reaching approximately 2% and averaging roughly 1% across comparisons.The reported aggregate gains vary across architectures and tasks.
- Architecture and task effects: Transformer backbones surpass the strongest alternative in 75% of settings, compared with 45% for convolutional backbones, whose largest drop reaches −3.07%.For Transformers, positive contributions reach 83.33% of fine-grained experiments.
- Shot-count effects: At 1 shot, MC-RFM has positive contributions in 64.28% of settings, while Transformer results reach 80.95%; at 16 shots, fine-grained Transformer tasks are best in all cases.Convolutional backbones show a net positive effect primarily at 16 shots.
- Ablation analysis: Removing cross-entropy drops performance by −5.6 pp on average, while replacing the mixed-curvature head causes −2.4 pp or −1.5 pp drops.Adaptive gating, feature-adaptive β, prototype shrinkage, and task context each worsen at least 9/10 cells when removed.
- Sensitivity and stability: Performance is relatively stable across curvature and ODE evaluations, but allocating dh/de = 192/64 to the hyperbolic branch consistently degrades results.A lighter split, dh/de = 64/192, is often competitive or better than the default.
5 Discussion
MC-RFM’s gains are especially pronounced for Transformer representations, where smooth geometric transport can realign already structured frozen features. Its effectiveness nevertheless depends on representation geometry and support-prototype quality.
- Interpretation: MC-RFM is particularly effective for Transformer-based representations and fine-grained recognition tasks.The discussion links this pattern to geometric, incremental adaptation rather than a single discrete update.
- Interpretation: The method is most useful when frozen representations provide stable semantics and well-formed neighborhoods, allowing class regions to be realigned without relearning low-level detectors.The paper associates this condition with globally contextualized or hierarchical Transformer features.
- Limitations: MC-RFM can be competitive with simpler variants when downstream classes are already well separated or the task is mostly Euclidean.Its benefits depend on the geometry of the frozen representation and support-set prototypes may be noisy in extreme low-shot regimes.
6 Conclusion
The paper concludes that MC-RFM treats few-shot adaptation as task-conditioned transport on a mixed-curvature product manifold rather than a static Euclidean perturbation. Results favor this formulation particularly for Transformer representations and fine-grained recognition.
- Conclusion: MC-RFM models adaptation as task-conditioned continuous transport on a product manifold combining hyperbolic and Euclidean structure.This replaces a static Euclidean perturbation of frozen features with geometry-aware transport.
- Conclusion: Across diverse datasets, backbones, and shot regimes, MC-RFM is particularly effective for Transformer-based representations and fine-grained recognition tasks.The conclusion attributes gains to the combined effect of mixed-curvature transport, task conditioning, adaptive gating, and discriminative supervision.
A Algorithmic Details
MC-RFM caches frozen features, constructs support prototypes and task context, transports projected states with a conditioned ODE, and classifies transported endpoints using a hybrid head.
- Training workflow: Training caches frozen backbone features, projects support features into mixed-curvature space, and constructs class prototypes and task context.The task context summarizes prototype information before conditioning the vector field.
- Training workflow: Each minibatch is transported toward its class prototype using flow matching, while endpoint classification adds cross-entropy supervision.The objective combines LFM with λclsLCE and updates the adapter, vector field, task encoder, and classifier.
- Inference workflow: At inference, support prototypes and task context are recomputed, query features are projected to M, and a low-NFE fixed-step ODE solver transports them.The hyperbolic component is re-projected to the Poincaré ball after each ODE step.
- Inference workflow: The hybrid head combines prototype-based and linear logits from the transported endpoint before returning the highest-scoring class.This preserves both prototype comparisons and learned linear classification in the final prediction.
B Complete results for all the experiments
Table 3 reports complete few-shot adaptation results as Top-1 accuracy across datasets, backbones, and shot counts, with MC-RFM compared against Euclidean and hyperbolic alternatives. Results span EuroSAT, FGVC, Flowers102, and Food-101 entries shown here, using three-seed mean and standard deviation.
- Table 3 reports Top-1 accuracy as mean ± standard deviation over three seeds, with the best method per row bolded.
- EuroSAT: EuroSAT results cover ResNet-50, ViT-B/16, ConvNeXt, DeiT, and Swin backbones at 1, 4, and 16 shots.
- FGVC: FGVC results cover the same five backbone families across 1, 4, and 16 shots, with MC-RFM generally near or above the Euclidean baseline.
- Flowers102: Flowers102 reports 1- and 4-shot results for five backbones, while 16-shot entries are marked N/A.
- Food-101: Food-101 reports ResNet-50, ViT-B/16, ConvNeXt, DeiT, and Swin results at 1, 4, and 16 shots.
C Numerical Stability Diagnostics
MC-RFM’s ablation and sensitivity runs show bounded mixed-curvature loss ratios without collapse, boundary-risk events, or severe loss imbalance. The diagnostics attribute observed ablation trends to architectural choices rather than numerical instability.
- Across all ablation and sensitivity runs, MC-RFM shows no collapse, boundary-risk events, or severe loss imbalance.
- The boundary margin remains close to one, indicating that hyperbolic states stay safely inside the Poincaré ball.
- Lh/Le increases with larger hyperbolic allocation dh/de = 192/64 and decreases when the CE term is removed.
- The diagnostics indicate that ablation trends are driven by architectural choices rather than numerical instability.
- The median loss ratio Lh/Le remains bounded across datasets, backbones, and variants over three seeds.
D Licensing and usage rights
The work uses publicly released pretrained weights and publicly available benchmark datasets under the licenses or usage terms reported by official repositories or dataset pages. Research-only or non-commercial assets are used solely for non-commercial academic evaluation.
- The study uses publicly released pretrained weights and publicly available benchmark datasets.
- Table 4 summarizes the assets and their corresponding licenses or usage terms from official repositories or dataset pages.
- Assets with research-only or non-commercial terms are used solely for non-commercial academic evaluation.