PACE: Progressive Angular-to-Norm Contrastive Embedding

1 Introduction

• Multimodal embedding models map heterogeneous inputs into a shared dense space, supporting similarity computation across modalities and applications such as retrieval, classification, VQA, and grounding.Universal multimodal embedders aim to generalize across input modalities and task formulations within one model.Source: §1 · p. 1
• Cosine-based contrastive training removes radial degrees of freedom, restricting semantic compatibility to angular information and excluding embedding norms as a relevance signal.Direct dot-product optimization uses both angular and norm information but underperforms cosine-based training and exhibits unstable dynamics.Source: §1 · pp. 1–2
• Angular–norm entanglement and directional anisotropy help explain direct dot-product training’s underperformance and instability, while full-parameter fine-tuning compounds the difficulty.The proposed response is to establish angular geometry in constrained representation and parameter spaces before expanding both.Source: §1 · p. 2
• PACE first combines cosine similarity with LoRA, then switches to dot-product similarity and full-parameter fine-tuning so directions and norms jointly encode semantics.The progressive Stage II initialization yields locally weak cross-Jacobian interactions, providing a mechanism for mitigating angular–norm entanglement.Source: §1 · p. 2
• PACE consistently outperforms strong baselines across multiple backbone scales and diverse multimodal embedding tasks.Its contributions combine a cosine-to-dot-product curriculum, a LoRA-to-full-fine-tuning schedule, and Focal Embedding Loss.Source: §1 · p. 2
• Focal Embedding Loss downweights queries with well-separated positives and emphasizes ambiguous queries with competitive negatives.This confidence-adaptive weighting is intended to reduce redundant updates on saturated examples and direct expanded optimization capacity toward unresolved distinctions.Source: §1 · p. 2

2 Related Work

• Related work improves embedding models through model development, query augmentation, data-side techniques, and similarity design.The field has progressed from vision–language alignment to universal representation learning, including MLLM-based and native multimodal embedders.Source: §2 · p. 2
• Query augmentation methods generate pseudodocuments, add explicit retrieval reasoning, or align generated expansions with a retriever, generally requiring extra LLM inference per query.Examples include Query2doc (Wang, Yang, and Wei 2023), HyDE (Gao et al. 2023), BRIGHT (Su et al. 2025), and ExpandR (Yao et al. 2025).Source: §2 · p. 2
• Data-side techniques include standardized selection, learned filtering, metadata-guided balancing, and task-diverse synthesis; prior difficulty-aware methods mine hard negatives or apply pair-level focal modulation.Query-level reweighting based on positive retrieval probability over the candidate set remains underexplored, which Focal Embedding Loss addresses.Source: §2 · pp. 2–3
• Most multimodal embedders use cosine similarity over normalized representations, excluding norms; Figure 2 diagnoses norm-stratified directions, directional concentration, and a larger full-tuning degradation under direct dot-product training.Unnormalized inner products are established retrieval and recommendation scores, while reliably learning a multimodal geometry that uses both angular and norm information remains underexplored.Source: Figure 2 · p. 3, §2 · p. 3

3.1 Problem Formulation

• The shared encoder maps heterogeneous queries and candidates to dense vectors, with the objective of ranking each positive candidate above its negatives.The resulting similarity ranking supports modality-agnostic retrieval across heterogeneous input compositions.Source: §3.1 · p. 3

3.2 Motivation

• Direct dot-product training entangles angular and norm information, concentrates directions into a narrower region, and degrades more under full-parameter tuning than under LoRA.The matched comparison motivates first establishing angular geometry in a compact parameter subspace before adding radial degrees of freedom and full-parameter updates.Source: §3.2 · p. 3

3.3 PACE

• PACE first pairs cosine similarity with LoRA, then switches to dot-product similarity and full-parameter fine-tuning; Focal Embedding Loss upweights ambiguous queries and downweights confident ones.Cosine-normalized confidence is used across both stages to keep norm scaling from distorting the difficulty estimate.Source: §3.3 · pp. 4–5

3.4 Theoretical Analysis

• Cosine training has no first-order radial embedding gradient; in the common-norm analysis, cross-Jacobian norms vanish as ρ→0, yielding local—but not global—decoupling.The authors state that this local result is consistent with the empirical reduction in angular–norm dependence.Source: §3.4 · p. 5

4.1 Implementation

• Stage I uses rank-32 LoRA and Focal Embedding Loss; Stage II initializes from the Stage I step-2,000 checkpoint and fine-tunes all parameters with dot-product similarity.Both stages use five hard negatives per query; the learning rate decreases from 2 × 10^-5 to 2 × 10^-6.Source: §4.1 · p. 5

Training Data.

• The training corpus contains approximately 2.0 million mmE5-Synthetic instances designed for MMEB-oriented multimodal embedding tasks.The benchmark spans 20 in-distribution and 16 out-of-distribution datasets across classification, VQA, retrieval, and grounding.Source: Training Data · p. 5

Evaluation.

• Evaluation uses MMEB's 20 in-distribution and 16 out-of-distribution test sets and reports Precision@1 with candidates ranked by unnormalized dot product.Precision@1 is the proportion of queries whose correct candidate ranks first.Source: Evaluation · p. 5

4.3 Baselines

• Baselines include zero-shot vision–language encoders and MLLM embedders, plus fine-tuned VLM2Vec and UniME-V2 models.UniME-V2 uses MLLM-derived semantic matching scores for contrastive supervision.Source: §4.3 · pp. 5–6

• PACE scores 59.0 and 67.3 overall on MMEB with the 0.8B and 2B backbones, respectively, exceeding scale-matched fine-tuned baselines by 2.0 and 2.4.The 0.8B model also exceeds the strongest zero-shot baseline by 14.7.Source: §5 · p. 6
• PACE improves per-category scores by 0.6–3.6 with 0.8B and 1.5–2.7 with 2B across classification, VQA, retrieval, and grounding.Source: §5 · p. 6
• PACE has positive margins on both IND and OOD splits.All methods were trained under identical hyperparameter settings for comparison.Source: §5 · p. 6

• PACE yields lower mean cosine similarity for randomly paired queries and candidates than direct dot-product training, indicating more angularly dispersed embeddings.Source: Figure 4 · p. 6, §6 · p. 6
• PACE reduces the negative Pearson correlation between cosine similarity and the standardized log norm product from −0.78 to −0.11.The authors interpret the near-zero correlation as little mutual information between norm and direction, which can carry complementary signals.Source: §6 · p. 6
• Cosine-to-dot and LoRA-to-full schedules outperform their reversed counterparts by 2.9 and 7.3 overall, respectively.Source: Figure 5 · p. 7
• PACE mitigates both angular–norm coupling and directional anisotropy.Source: §6 · pp. 6–7

• Retaining LoRA instead of switching to full-parameter fine-tuning lowers the overall score to 54.1, the largest reported component degradation.Removing Focal Embedding Loss lowers the score from 59.0 to 57.7; retaining cosine similarity in Stage II lowers it to 57.2.Source: §7 · p. 7
• Cosine-to-dot and LoRA-to-full schedules outperform their reversed counterparts by 2.9 and 7.3 overall, respectively.The comparisons support establishing angular geometry and using a compact parameter subspace before expansion.Source: §7 · p. 7
• Stage I checkpoints at steps 2000 and 4000 score 58.3 and 58.5, exceeding the fully converged step-7847 checkpoint’s 56.9.The authors suggest full convergence in the constrained first stage can overspecialize angular geometry and reduce plasticity for Stage II.Source: §7 · p. 7
• Focal Embedding Loss performs best against random, query–positive, and query–negative weighting alternatives.Its query-level retrieval confidence reflects both positive alignment and negative competition.Source: §7 · p. 7, Figure 6 · p. 7

8 Conclusion

• PACE begins with cosine similarity and LoRA, then introduces dot-product similarity and full-parameter fine-tuning to expand both spaces progressively.This coarse-to-fine strategy establishes angular semantic geometry before adding radial and parameter degrees of freedom.Source: §8 · p. 7
• Focal Embedding Loss emphasizes ambiguous queries while reducing redundant updates on well-separated examples.Source: §8 · p. 7
• Geometric analyses show that PACE alleviates angular–norm coupling and directional anisotropy, and experiments across multiple scales and multimodal tasks validate its effectiveness.Source: §8 · p. 7
• The paper presents progressive optimization-space expansion as an effective approach to better exploiting MLLM-based embedders’ representational capacity.Source: §8 · p. 8

A.1 Training Details

• PACE’s first stage uses LoRA with Focal Embedding Loss, then initializes dot-product full-parameter fine-tuning from the Stage I step-2,000 checkpoint.Both backbones share settings; the vision encoder and multimodal projector remain frozen in both stages.Source: §A.1 · p. 10

A.2 Instruction Prompts

• Each instance is encoded independently with last-token pooling under a shared template, with task instructions in the query and media inserted through modality placeholders.The template injects “Represent the user’s input.” as its system-level instruction.Source: §A.2 · p. 10

B Detailed Theoretical Analysis

• The Proposition 1 derivation assumes common-norm query and candidate embeddings and fixed candidates, then analyzes radial–angular gradient components through a block Jacobian.Source: B Detailed Theoretical Analysis · p. 10

C.1 Effect of GradCache

• 60.4 vs 57.9 overall: GradCache raises PACE from 59.0 to 60.4 and VLM2Vec from 55.8 to 57.9, leaving PACE 2.5 points ahead.The enlarged effective batch supplies more informative in-batch negatives.Source: §C.1 · pp. 11–12, Table 6 · p. 12
• PACE’s grounding score falls from 77.2 to 74.5 with GradCache, but remains the highest among all configurations.Source: §C.1 · p. 12

C.4 Sensitivity to the Focal Parameter

• Final scores range from 58.2 to 59.0 across tested γ settings, and γ = 2.0 achieves the highest score of 59.0.Stage II improves every evaluated configuration by 1.8–2.1 points; stronger modulation adds no benefit.Source: Table 7 · p. 12, §C.4 · pp. 12–13

C.2 Additional Geometry Analysis

• Across four resampled subsets, cosine-trained embeddings remain norm-intermixed, whereas direct dot-product training reproduces norm-stratified structure.The repeated pattern supports angular–norm entanglement as systematic rather than a sampling artifact.Source: §C.2 · p. 12, Figure 7 · p. 13

C.3 Analysis of Stage Transition

• 58.3 vs 49.7 at convergence: the mid-step Stage I start trains smoothly, while the fully converged start deteriorates after step 4,500.The fully converged start leaves an 8.6-point gap; the analysis attributes its instability to recovering norm sensitivity while restructuring angular layout.Source: §C.3 · p. 12, Figure 8 · p. 13

C.2 Additional Geometry Analysis

• Under the same visualization protocol, staged cosine→dot training restores intermixed norms, unlike direct dot-product training’s coherent norm-aligned structure.The cosine→dot transition alone alleviates the coupling, with focal weighting disabled in this comparison.Source: §C.2 · p. 12, Figure 9 · p. 13

D External Results

• Table 8 reports per-dataset results for PACE and all compared methods on the MMEB benchmark.Source: D External Results · p. 13, Table 8 · p. 14
• “All” in Table 8 denotes the average over all 36 datasets.Source: Table 8 · p. 14
• An asterisk marks OOD datasets, and † marks Qwen3.5-2B-Base; all other methods use Qwen3.5-0.8B-Base.Source: Table 8 · p. 14