PaperlayerView exact revision on arXiv

Source-linked AI summary

Learn Before Represent: Bridging Generative and Contrastive Learning for Domain-Specific LLM Embeddings

Xiaoyu Liang, Yuchen Peng, Jiale Luo, Wenhao Wang, Haoji Hu, Xincheng Zhou

arXiv:2601.11124v1cs.IRcs.AI

TL;DR

Vertical-domain embedding models struggle when contrastive learning must represent terminology and entities absent from general pretraining, because semantic alignment does not itself acquire domain knowledge. LBR first injects knowledge through information-bottleneck-constrained generative learning with causal attention, then aligns compressed representations contrastively. Across medical, chemistry, and code retrieval, it significantly and consistently improves over strong baselines, while its current objective remains limited for high-level reasoning and complex deduction.

  • Problem

    Contrastive learning-based LLM embeddings struggle in vertical domains because general pretraining lacks specialized terminology and entities, while contrastive learning focuses on semantic alignment rather than knowledge acquisition.

  • Method

    LBR uses information-bottleneck-constrained generative learning to inject domain knowledge and compress semantics with causal attention, followed by contrastive alignment of the compressed representations.

  • Results

    LBR achieves significant and consistent improvements over strong baselines across medical, chemistry, and code retrieval tasks.

  • Takeaways & Limitations

    The results support learning domain knowledge before representation alignment as a paradigm for vertical-domain representation learning.

  • Takeaways & Limitations

    The current generative objective injects factual knowledge but remains challenged by high-level reasoning and complex logical deduction.

Abstract

from arXiv · show

Large Language Models (LLMs) adapted via contrastive learning excel in general representation learning but struggle in vertical domains like chemistry and law, primarily due to a lack of domain-specific knowledge. This work identifies a core bottleneck: the prevailing ``LLM+CL'' paradigm focuses on semantic alignment but cannot perform knowledge acquisition, leading to failures on specialized terminology. To bridge this gap, we propose Learn Before Represent (LBR), a novel two-stage framework. LBR first injects domain knowledge via an Information Bottleneck-Constrained Generative Learning stage, preserving the LLM's causal attention to maximize knowledge acquisition while compressing semantics. It then performs Generative-Refined Contrastive Learning on the compressed representations for alignment. This approach maintains architectural consistency and resolves the objective conflict between generative and contrastive learning. Extensive experiments on medical, chemistry, and code retrieval tasks show that LBR significantly outperforms strong baselines. Our work establishes a new paradigm for building accurate and robust representations in vertical domains.

1 Introduction

Vertical-domain embedding models need domain knowledge before semantic alignment, because contrastive learning alone cannot acquire missing terminology knowledge. LBR addresses this with a two-stage framework that preserves causal attention, compresses semantics through an information bottleneck, and then applies contrastive alignment.

  • Contrastive learning excels on general benchmarks but struggles with vertical-domain terminology and long-tail entities absent from general pretraining.
  • Vertical retrieval failures can reflect missing factual knowledge, such as recognizing Acetylsalicylic acid as the common pain reliever aspirin.
  • The framework responds to architectural inconsistency: bidirectional attention aids contrastive learning but prevents autoregressive next-token prediction for domain-knowledge acquisition.
  • LBR preserves causal attention while using an information bottleneck to reconcile generative knowledge acquisition with contrastive representation learning.
  • LBR is presented as a unified framework that uses domain knowledge acquisition as the foundation for accurate vertical-domain representations.
  • LBR achieves significant and consistent improvements over strong baselines across chemistry, medical, and code retrieval tasks.

2 Related Work

Related work shifts decoder-only LLM representation learning toward bidirectional adaptation, reasoning-oriented embeddings, and reinforcement learning. LBR instead embeds information bottleneck compression directly into the native decoder-only architecture to unify generative and representation learning.

  • Recent embedding methods remove causal masks to restore bidirectional attention, transforming decoder-only LLMs toward globally perceptive encoders.
  • Reasoning-embedding methods use Chain-of-Thought or soft tokens to bridge semantic gaps while preserving computational depth or reducing overhead.
  • 2.3 Representation Enhancement via Information Bottleneck: Information bottleneck methods extract essential features through compression, but existing masked-autoencoding approaches often require asymmetric architectures or auxiliary decoders.
  • 2.3 Representation Enhancement via Information Bottleneck: LBR creates an intrinsic bottleneck through a specialized attention mask in the native decoder-only architecture, forcing global semantics into limited bottleneck tokens.

3 Method

LBR adapts LLMs into domain-specific embedding models through two stages: IB-constrained generative learning first acquires and compresses domain knowledge, then Generative-Refined Contrastive Learning aligns the compressed representations while preserving causal attention.

  • LBR is a two-stage paradigm that combines IB-constrained generative learning with contrastive learning for domain-specific embeddings.The framework also introduces Separation Ratio to guide selection of an optimal training trajectory.
  • 3.1 Preliminaries: The Information Bottleneck Principle: The Information Bottleneck objective compresses source knowledge into bottleneck tokens Z while retaining information needed to predict the generative target Y.X and Y denote source domain knowledge and the generative target, while β controls the compression–prediction trade-off.
  • 3.2 Stage 1: IB-Constrained Generative Learning: A specialized attention mask blocks direct X-to-Y information flow, forcing the model to compress input semantics through the limited capacity of Z.The compression ratio R = |X|/|Z| parameterizes bottleneck strength.
  • 3.2 Stage 1: IB-Constrained Generative Learning: Generative learning maximizes information retained in Z by training autoregressive target prediction, with SFT-style query–answer or self-supervised reconstruction and prefix-suffix variants selected by data availability.Minimizing Lgen preserves information necessary to predict Y; unlabeled variants enable unsupervised domain knowledge acquisition at higher computational cost.
  • 3.3 Stage 2: Generative-Refined Contrastive Learning: Stage 2 extracts the final bottleneck-token hidden state as the sequence representation and aligns query–passage pairs with InfoNCE under the same causal attention architecture.This preserves consistency with Stage 1 while retaining the LLM’s generative capacity and instruction-following strengths.

4 Experiments

LBR is evaluated across medical, chemistry, and code retrieval using standardized vertical-domain datasets, isolated evaluation sets, and comparisons against generative and contrastive baselines. Across these experiments, Information Bottleneck-constrained generative learning improves retrieval while preserving generation and supports stronger causal contrastive representations.

  • Main Results in Domain: LBR consistently achieves the best results across Chemistry, Medical, and Code, with Qwen2.5-1.5B reaching 87.9 average score versus 79.3 for LLM2Vec.The comparison covers specialized-domain datasets and includes standard generative, contrastive, and combined training paradigms.
  • Main Results in Domain: LBR improves Chemistry R@10 from 0.436 to 0.802 over SFT+CL, a +36.6% gain attributed to Information Bottleneck semantic compression.The bottleneck constraint addresses representation collapse observed with naive SFT+CL.
  • Representation and Generation: SFT obtains the highest generation score at 14.32 B-4, whereas CL reaches 75.59 R@10 but loses generative capability, revealing a representation-generation trade-off.SFT improves retrieval over the base model from 41.03 to 54.91 R@10, but next-token prediction alone does not produce robust semantic representations.
  • Analysis of different stages in LBR: Standard GL provides only marginal retrieval gains, while IB-GL yields substantial improvements and combining IB-GL with CL produces the best overall performance.The ablation attributes the advantage to semantic compression, which resolves the conflict between token-level generative optimization and sample-level contrastive learning.

5 Ablation Studies

The ablations show that LBR benefits from causal attention, moderate bottleneck compression, balanced generative–contrastive allocation, and efficient supervised training. Across these settings, the IB-constrained design improves knowledge utilization while controlling computational cost.

  • 5.1 Causal vs. Bidirectional Attention: IB-GL+Causal CL outperforms IB-GL+Bidirectional CL because bidirectional attention disrupts learned autoregressive compression and can bypass internal knowledge use.Causal attention preserves alignment with the bottleneck compression learned during generative learning.
  • 5.2 Bottleneck Compression Ratio: R = 500 yields optimal retrieval performance, with stable results across R ∈ [200, 500], while mild compression is insufficient and R = 1000 loses information.The recommended range narrows to R ∈ [200, 400] for information-dense domains and expands to R ∈ [500, 800] for high-redundancy domains.
  • 5.3 Learning Efficiency and Data Allocation: With a fixed 100k-sample budget, both methods follow inverted U-shaped curves: increasing GL data initially helps, but excessive allocation degrades performance.The result supports complementary roles for generative and contrastive learning rather than relying exclusively on either objective.
  • 5.3 Learning Efficiency and Data Allocation: IB-constrained generative learning consistently outperforms standard generative learning across all allocation ratios, with a steeper improvement slope.The steeper slope indicates more efficient knowledge utilization for retrieval through explicit semantic compression.
  • 5.4 Training Efficiency Analysis: IB-GL with SFT increases training time by only 3% over standard SFT, whereas PT approximately quadruples training time because reconstruction doubles sequence length under quadratic attention.SFT provides slightly better retrieval performance; PT remains a scalable alternative when supervised data is unavailable.

6 Conclusion

LBR introduces a sequential two-stage framework that injects domain knowledge before aligning representations, resolving conflicts between generative and contrastive learning in vertical domains.

  • LBR establishes domain knowledge ingestion before representation alignment as its core principle for vertical domain adaptation.
  • IB-GL uses an Information Bottleneck to compress global semantics into bottleneck tokens, enabling knowledge injection while preserving the native architecture.This initializes the model for subsequent contrastive alignment.
  • Experiments across medical, chemistry, and code domains validate LBR's superiority over standard methods.
  • The framework's sequential design is effective, while jointly optimizing generative and contrastive objectives remains future work.

7 Limitations

LBR’s current evaluation and design leave three main limitations: uncertain generalizability beyond entity-dense domains, limited validation of complex reasoning, and fixed-length semantic compression.

  • Domain scope: Evaluation primarily covers entity-dense domains such as medicine, chemistry, and code, leaving generalization to logic-intensive fields like law and finance unvalidated.The Information Bottleneck principle is theoretically domain-agnostic, but additional empirical validation would strengthen generalizability claims.
  • Cognitive capability: The framework currently validates factual knowledge injection, while high-level reasoning and complex logical deduction remain challenging under the current generative objective alone.Reasoning-specific supervision such as Chain-of-Thought is proposed as a future extension for complex professional tasks.
  • Architectural flexibility: Fixed-length bottleneck tokens may be suboptimal for inputs with variable information density, causing under-compression of complex semantics or over-compression of simple ones.Adaptive or hierarchical bottlenecks are proposed to learn compression intensity from input complexity.
Loading 2601.11124v1…