PaperlayerView exact revision on arXiv

Source-linked AI summary

GAPrune: Gradient-Alignment Pruning for Domain-Aware Embeddings

Yixuan Tang, Yi Yang

arXiv:2509.10844v1cs.CL

TL;DR

Domain-specific embeddings require specialized semantics, while large models challenge deployment and existing pruning treats general and domain-specific parameters uniformly. GAPrune combines Fisher importance with cross-domain gradient alignment; it stays near dense-model performance at 50% sparsity and improves benchmark scores after retraining.

  • Problem

    Pruning methods treat parameters uniformly, risking removal of domain-specific knowledge or loss of general linguistic capability, while large models challenge resource-constrained deployment.

    Source: §1 · p. 2

  • Method

    GAPrune estimates parameter importance with Fisher Information, assesses cross-domain behavior using gradient alignment, and combines the signals into a unified importance score.

    Source: §3.2 · p. 4, §3.2.2 · p. 5

  • Results

    At 50% sparsity, GAPrune maintains performance within 2.5% of dense models; retraining improves Qwen3-Embedding-4B by +4.51% on FinMTEB and +1.73% on ChemTEB.

    Source: §4.2 · p. 8, §4.3 · p. 8

  • Takeaways & Limitations

    The dual-criteria strategy is intended to preserve specialized domain representations alongside general semantic capabilities during pruning.

    Source: §3.2.3 · p. 6

Abstract

from arXiv · show

Domain-specific embedding models have shown promise for applications that require specialized semantic understanding, such as coding agents and financial retrieval systems, often achieving higher performance gains than general models. However, state-of-the-art embedding models are typically based on LLMs, which contain billions of parameters, making deployment challenging in resource-constrained environments. Model compression through pruning offers a promising solution, but existing pruning methods treat all parameters uniformly, failing to distinguish between general semantic representations and domain-specific patterns, leading to suboptimal pruning decisions. Thus, we propose GAPrune, a pruning framework that addresses this challenge by considering both domain importance and preserving general linguistic foundation. Our method uses Fisher Information to measure importance and general-domain gradient alignment to assess parameter behavior, then combines these signals using our Domain Alignment Importance (DAI) scoring. Lower DAI scores indicate that the parameter is either less important for the domain task or creates conflicts between domain and general objectives. Experiments on two domain benchmarks, FinMTEB and ChemTEB, show that GAPrune maintains performance within 2.5% of dense models in one-shot pruning at 50% sparsity, while outperforming all baselines. With retraining in 100 steps, GAPrune achieves +4.51% improvement on FinMTEB and +1.73% on ChemTEB, demonstrating that our pruning strategy not only preserves but enhances domain-specific capabilities. Our findings demonstrate that principled pruning strategies can achieve model compression and enhanced domain specialization, providing the research community with a new approach for development.

1 INTRODUCTION

GAPrune addresses the tension between specialized semantic representations and efficient deployment by combining domain-importance and cross-domain-alignment signals. The introduction reports performance near dense models at 50% sparsity and gains after retraining.

1 INTRODUCTION

  • Domain-specific semantics matter for embedding tasks: “liability” has financial connotations beyond its general legal meaning, while “binding” can refer to molecular interactions.The introduction uses these examples to motivate domain adaptation.Source: §1 · p. 1
  • Domain-adaptation approaches scale pretrained models, with BMEmbed synthesizing specialized data and CodeXEmbed spanning 400M to 7B parameters; performance is reported to improve with size.The introduction links this trend to established scaling laws.Source: §1 · pp. 1–2
  • The introduction frames domain adaptation as a deployment trade-off: larger models perform better, but computational efficiency may shape adoption.It cites Qwen3-Embedding download counts as an example of this trade-off.Source: §1 · p. 2
  • Existing pruning methods assess parameters uniformly, risking removal of domain knowledge or loss of general linguistic capabilities when importance is based only on domain samples.The introduction describes both failure modes as producing pruned models that lose specialization or compromise general abilities.Source: §1 · p. 2
  • At 50% one-shot sparsity, GAPrune stays within 2.5% of dense models on FinMTEB and ChemTEB; retraining yields +4.51% on FinMTEB and +1.73% on ChemTEB.The paper presents these results as evidence that pruning can preserve and enhance domain-specific capabilities.Source: §1 · p. 2
  • GAPrune combines Fisher Information for domain importance with cross-domain gradient alignment in its Domain Alignment Importance score.Low DAI scores indicate lower domain importance or conflict between general and domain-specific objectives.Source: §1 · p. 2

2 RELATED WORK

The related-work section covers LLM-based embeddings, domain-specific adaptation, and compression, highlighting the particular challenges of pruning embedding models.

LLM-based Embedding Models

  • Instruction-tuned E5-Mistral-Instruct (Wang et al., 2024) and Qwen3-Embedding (Zhang et al., 2025) support task-specific embedding, but their scale makes efficient deployment important.The review notes that these models typically contain billions of parameters.Source: §2 · p. 2

Domain-Specific Embedding Models

  • Benchmarks including CoIR (Li et al., 2025b), FinMTEB (Tang & Yang, 2025), and ChemTEB (Kasmaee et al., 2025) show domain-specific embeddings outperforming general models on specialized tasks.The review also notes that deploying large domain-specific models in resource-constrained environments remains challenging.Source: §2 · p. 3

Model Compression for Embedding Models

  • Pruning research ranges from magnitude-based methods to SparseGPT (Frantar & Alistarh, 2023) and Wanda (Sun et al., 2024), but these methods target generative LLMs.Embedding models use task-specific measures such as nDCG@10 for retrieval and accuracy for classification (Muennighoff et al., 2023).Source: §2 · p. 3
  • Embedding models show parameter-sensitivity patterns that include higher sensitivity to attention-head removal, as heads capture distinct semantic relationships (Voita et al., 2019).The review identifies domain-aware pruning designed specifically for LLM-based embedding models as underexplored.Source: §2 · p. 3

3 GAPRUNE: GRADIENT-ALIGNMENT PRUNING

GAPrune frames domain-specific pruning as preserving parameters that matter to the target task while accounting for whether they align with general linguistic objectives. It combines importance and cross-domain gradient signals to guide compression.

3 GAPRUNE: GRADIENT-ALIGNMENT PRUNING

  • GAPrune characterizes parameters by domain-specific importance and alignment between general and domain objectives to guide compression while preserving domain knowledge.Source: §3 · p. 3, §3.2.3 · p. 6

3.1 PROBLEM FORMULATION: DOMAIN PRUNING

  • Domain-specific pruning minimizes target-domain performance degradation subject to a sparsity constraint, because parameters can encode either shared semantics or domain-specific patterns.Source: §3.1 · p. 4

3.2 METHOD

  • The method samples general and domain data, estimates parameter importance and cross-domain alignment, then combines these signals into a unified score.Source: §3.2 · p. 4

3.2.1 DATA PREPARATION AND SAMPLING STRATEGY

  • General and domain datasets supply contrastive triplets; k-means selects 5,000 representative samples from each dataset for gradient analysis.Source: §3.2.1 · p. 4

3.2.2 CHARACTERIZING PARAMETER BEHAVIOR

  • Fisher Information estimates parameter importance from loss sensitivity, while gradient alignment compares domain and general objectives; positive alignment suggests shared knowledge and negative alignment indicates conflict.Source: §3.2.2 · p. 5

3.2.3 DOMAIN-AWARE PRUNING STRATEGY

  • DAI combines domain Fisher value, a penalty for general-domain importance, parameter magnitude, and gradient alignment; negative alignment lowers a parameter’s score.One-shot pruning retains the top-k DAI scores.Source: §3.2.3 · p. 6

4 EXPERIMENT

The experiments evaluate GAPrune on FinMTEB and ChemTEB using one-shot pruning and prune-and-retrain settings, then analyze layer-wise importance and embedding geometry. The results cover two embedding architectures and sparsity settings, with the clearest gains appearing at 50% sparsity and after retraining.

4.1 EXPERIMENTAL SETUP

  • The evaluation covers finance and chemical benchmarks, two embedding architectures, 30% and 50% MLP sparsity, and both one-shot and 100-step retraining protocols.FinMTEB includes 8 classification, 2 STS, and 8 retrieval tasks; ChemTEB includes 17 classification and 2 retrieval tasks.Source: §4.1 · p. 7

4.2 ONE-SHOT PRUNING EVALUATION

  • At 50% sparsity, GAPrune stays within 2.5% of dense performance, while random pruning drops 40–60% and General Fisher drops over 30% on FinMTEB.The one-shot comparison covers FinMTEB and ChemTEB; the table reports changes relative to dense models.Source: Table 1 · p. 8, §4.2 · p. 8

4.3 PRUNE-AND-RETRAIN EVALUATION

  • After retraining, Qwen3-Embedding-4B improves by +4.51% on FinMTEB and +1.73% on ChemTEB relative to dense performance.The prune-and-retrain evaluation uses 50% sparsity; GAPrune also consistently outperforms L3 Prune across metrics and performs best on both datasets with E5-mistral-7B-Instruct.Source: §4.3 · p. 8, Table 2 · p. 9

4.4.1 LAYER CORRELATION ANALYSIS

  • GAPrune has low score correlation with Domain Fisher (-0.406) and General Fisher (-0.459), while the two Fisher methods correlate at 0.978.The authors interpret Fisher Information alone as relatively domain-agnostic.Source: §4.4.1 · p. 9
  • Retrieval performance improves significantly in later layers around layer 24, where Fisher-based methods prune more aggressively and GAPrune retains parameters.The analysis evaluates hidden states from Qwen3-Embedding-4B on FinMTEB tasks; the figure compares average importance scores and performance across layers.Source: §4.4.1 · pp. 9–10, Figure 2 · p. 9

4.4.2 GEOMETRIC HYPERSPHERE ANALYSIS

  • GAPrune achieves the best query-positive alignment at 0.51 alignment loss, while retaining 0.52 cross-dimensional correlation and 0.22 cosine similarity, indicating better preservation of semantic relationships.Magnitude pruning matches the dense model’s uniformity (-3.30), but its cross-dimensional correlation is 0.44 and cosine similarity is 0.16.Source: §4.4.2 · p. 10
  • GAPrune uses 1,820 of 2,560 dimensions, compared with 1,605 for Domain Fisher pruning.The authors suggest domain-only pruning may remove parameters important for general knowledge.Source: §4.4.2 · p. 10

5 CONCLUSION

GAPrune combines parameter importance with cross-domain gradient alignment for pruning domain-specific embedding models. The experiments report performance near dense models at 50% sparsity and gains after retraining in 100 steps.

5 CONCLUSION

  • GAPrune combines Fisher Information and gradient alignment to identify parameters important for domain performance and aligned with general objectives.The framework is presented as a domain-aware pruning approach for embedding models.Source: §5 · p. 10
  • At 50% sparsity, one-shot GAPrune remains within 2.5% of dense performance and improves performance after 100 retraining steps.The conclusion states these results hold across architectures and domains.Source: §5 · p. 10
  • The authors conclude that principled domain-aware compression can achieve both efficiency and performance for specialized embedding models.They describe GAPrune as a potential tool for practitioners and the research community.Source: §5 · p. 10

A ALGORITHM SUMMARY

Algorithm 1 describes GAPrune as a three-stage pruning procedure: representative sampling, per-parameter analysis, and DAI-based scoring and pruning. Its parameter analysis is the main computational cost, while later scoring operations are element-wise.

A ALGORITHM SUMMARY

  • The procedure samples 5,000 representative examples from each general and domain corpus before analyzing parameters.The algorithm uses representative sampling in Stage 1.Source: A ALGORITHM SUMMARY · p. 13
  • For each parameter, GAPrune computes gradients and Fisher Information to assess parameter importance and cross-domain alignment.The analysis is described as computing Fisher information and gradient-alignment scores for each parameter.Source: A ALGORITHM SUMMARY · p. 13
  • GAPrune combines the analysis signals into DAI scores, ranks parameters, and retains the top fraction determined by the target sparsity.The procedure returns a pruned model at the requested sparsity.Source: A ALGORITHM SUMMARY · p. 13
  • Parameter analysis costs O(|θ|·N), while gradient-alignment and DAI computations cost O(|θ|).The authors characterize this as single-shot analysis without iterative procedures or retraining.Source: A ALGORITHM SUMMARY · p. 13

B EVALUATION INSTRUCTIONS

The appendix provides evaluation instructions for the FinMTEB and ChemTEB tasks used in the experiments.

B EVALUATION INSTRUCTIONS

C BENCHMARK METRICS

The benchmark metrics use different primary measures for classification, retrieval, and semantic textual similarity tasks.

  • Classification is evaluated primarily with accuracy, the percentage of correctly classified samples.Source: C BENCHMARK METRICS · p. 13
  • Retrieval is evaluated primarily with nDCG@10, which accounts for the relevance and rank of retrieved documents.Source: C BENCHMARK METRICS · p. 14
  • STS is evaluated primarily with Spearman correlation between predicted similarity scores and ground-truth similarity scores.Source: C BENCHMARK METRICS · p. 14

D RUNTIME ANALYSIS

GAPrune's computational requirements are compared with the dense Qwen3-Embedding-4B model, and runtime is measured on FiQARetrieval.

  • 33.4% reduction in computational requirements versus dense Qwen3-Embedding-4B on 4×3090 GPUs.Runtime is also measured on FiQARetrieval, with each GPU consuming 10.0 GB of memory; the passage refers to Table 7 but gives no runtime values.Source: D RUNTIME ANALYSIS · p. 14

E METHOD CORRELATION ANALYSIS

The analysis compares pruning methods by correlating their normalized parameter-importance ranks; GAPrune's correlations differ from those of the two Fisher-based methods.

  • The analysis normalizes each method's importance scores into relative ranks from 0 to 1, then computes Pearson correlations across common parameters.Source: E METHOD CORRELATION ANALYSIS · p. 16
  • GAPrune has negative rank correlations with Domain Fisher (-0.406) and General Fisher (-0.459), indicating complementary information beyond Fisher Information alone.Source: E METHOD CORRELATION ANALYSIS · p. 16, Table 8 · p. 16
  • Domain Fisher and General Fisher have a 0.978 positive correlation, suggesting that Fisher Information may be relatively domain-agnostic.The passage suggests this similarity could help explain why both methods struggle to preserve domain-specific knowledge during pruning.Source: E METHOD CORRELATION ANALYSIS · p. 16, Table 8 · p. 16

F GEOMETRIC METRICS COMPUTATION

The section describes geometric measures used to characterize embedding distributions, query-positive alignment, dimensional relationships, and preservation of the dense embedding structure.

  • Uniformity loss measures how uniformly embeddings are distributed; lower values indicate a more uniform distribution.The expectation is over pairs of distinct embeddings, with temperature t set to 2.0.Source: F GEOMETRIC METRICS COMPUTATION · p. 16
  • Alignment loss measures query-to-positive alignment; lower values indicate better alignment, with power parameter α set to 2.0.Source: F GEOMETRIC METRICS COMPUTATION · p. 16
  • Cross-dimensional correlation averages absolute correlations across dimensions to quantify preservation of dimensional relationships between queries and positive samples.Source: F GEOMETRIC METRICS COMPUTATION · pp. 16–17
  • Effective dimensionality measures how many dimensions capture essential embedding information; higher values indicate better embedding-space utilization.Source: F GEOMETRIC METRICS COMPUTATION · p. 17
  • Average cosine similarity between pruned and dense embeddings measures preservation of the original embedding structure; higher values indicate better preservation.Source: F GEOMETRIC METRICS COMPUTATION · p. 17
Loading 2509.10844v1…