Source-linked AI summary
C2LLM Technical Report: A New Frontier in Code Retrieval via Adaptive Cross-Attention Pooling
Jin Qin, Zihan Liao, Ziyin Zhang, Hang Yu, Peng Di, Rui Wang
TL;DR
Code embedding must represent long code sequences without departing from causal LLM pretraining, while EOS pooling can create an information bottleneck. C2LLM adds a learnable-query PMA module to pool token representations and adapt embedding dimension. On MTEB-Code, its 7B model ranks first overall, with strong results also reported for the 0.5B model.
Problem
Mean pooling often relies on bidirectional attention unlike causal LLM pretraining, while EOS representations compress a code sequence into one position and can bottleneck long inputs.
Source: §1 · p. 2Method
Results
On 12 MTEB-Code retrieval tasks, C2LLM-7B averages 80.75, surpassing Seed1.6-Embedding and Qwen3-Embedding-8B; C2LLM-0.5B averages 75.46.
Source: §4.2 · p. 5Takeaways & Limitations
Abstract
from arXiv · showhide
We present C2LLM - Contrastive Code Large Language Models, a family of code embedding models in both 0.5B and 7B sizes. Building upon Qwen-2.5-Coder backbones, C2LLM adopts a Pooling by Multihead Attention (PMA) module for generating sequence embedding from token embeddings, effectively 1) utilizing the LLM's causal representations acquired during pretraining, while also 2) being able to aggregate information from all tokens in the sequence, breaking the information bottleneck in EOS-based sequence embeddings, and 3) supporting flexible adaptation of embedding dimension, serving as an alternative to MRL. Trained on three million publicly available data, C2LLM models set new records on MTEB-Code among models of similar sizes, with C2LLM-7B ranking 1st on the overall leaderboard.
1 Introduction
Code embedding models face a trade-off: mean pooling often departs from causal pretraining, while EOS embeddings compress long code sequences into one position. C2LLM uses a learnable-query PMA module with causal LLM representations, and reports strong MTEB-Code results at both model sizes.
1 Introduction
- Code retrieval selects relevant snippets from large codebases and is a step in code-agent workflows that plan, search, and edit code.The introduction also identifies interactive developer search engines as a use case.Source: §1 · p. 2
- Mean pooling often uses bidirectional attention unlike causal pretraining, while EOS pooling compresses sequence structure into one position, a bottleneck for code files that may contain thousands of tokens.The paper frames these constraints as a challenge for code embedding.Source: §1 · p. 2
- C2LLM preserves its backbone's causal attention and uses a learnable-query PMA module to aggregate all token representations into one vector while supporting embedding-dimension adaptation.The paper positions this as an alternative to the trade-off between mean pooling and EOS representations.Source: §1 · p. 2
- C2LLM-7B averages 80.75 on MTEB-Code and ranks first overall; C2LLM-0.5B scores 75.46 and surpasses similar-sized competitors.The models were trained on 3 million publicly available data.Source: §1 · p. 2
2 Related Work
The section contrasts BERT-based code embedding models with newer code-focused LLM approaches, while noting that the MTEB-Code leaderboard is dominated by general-purpose text embedders.
2 Related Work
- BERT-based CodeBERT, GraphCodeBERT, CodeSage, and CodeT5+ do not use code LLMs pretrained on trillions of tokens; BGE-Code and CodeXEmbed are exceptions, but none of these models appear on MTEB-Code.The leaderboard is described as dominated by general-purpose models including Qwen3-Embedding, INF-Retriever, and EmbeddingGemma.Source: §2 · p. 2
- Figure 2 depicts C2LLM as an LLM followed by PMA, a cross-attention pooling module that uses the LLM's last hidden states as keys and values.The illustration omits the multi-head mechanism.Source: Figure 2 · p. 3
3 Model Architecture: Introducing Pooling by Multihead Attention into Embedding Models
C2LLM places a single-query PMA cross-attention layer over the LLM's final hidden states to form a sequence embedding. This pools token information and can reduce the embedding dimension without changing the causal LLM backbone.
3 Model Architecture: Introducing Pooling by Multihead Attention into Embedding Models
- Mean pooling is often paired with bidirectional attention, whereas EOS pooling creates an information bottleneck; NV-Embed's latent attention still requires mean pooling afterward.C2LLM is presented as another solution to this pooling dilemma.Source: §3 · p. 3
- PMA uses one learnable query and the LLM's last hidden states as keys and values to aggregate the token sequence into one embedding, while reducing embedding dimension as an alternative to MRL.Figure 2 illustrates the LLM-plus-PMA architecture.Source: §3 · p. 3
- The input consists of hidden states for l tokens with dimension d_LLM and a learnable query with dimension d_q; projections map them to a lower dimension d.The output embedding dimension is d.Source: §3 · p. 4
- Cross-attention is computed in the lower dimension with residual connections and layer normalization, and the resulting E is the sequence embedding.The paper identifies E as the embedding for the input sequence.Source: §3 · p. 4
- PMA can learn which tokens matter, such as function signatures or key algorithmic logic, while retaining the causal architecture and adding negligible overhead relative to the LLM.Decoupling the LLM hidden dimension from the output dimension supports compact vector-database embeddings without the computationally expensive MRL objective.Source: §3 · p. 4
4 Experiments
Experiments fine-tune 0.5B and 7B Qwen2.5-Coder models with PMA and contrastive learning on 3 million samples. C2LLM reports strong MTEB-Code performance at both scales, including a first-place 7B result.
4.1 Training Settings
- The models fine-tune Qwen2.5-Coder-0.5B-Instruct and Qwen2.5-Coder-7B-Instruct on 3 million samples, using 32-head PMA, LoRA, and Flash Attention 2.The data combines retrieval, code problem-solving, feedback, editing, question-answering, text-to-SQL, and translation sources.Source: §4.1 · p. 4
- Training uses three epochs, a 1024-token maximum sequence length, global in-batch negatives, and seven hard negatives per query; four checkpoints are weighted-merged for the final model.The loss temperature is 0.05, and training batches are grouped by dataset source and programming language.Source: §4.1 · p. 4
- Table 1 provides prompt templates for each dataset used in training.The available passage identifies the table's scope but gives no template content.Source: §4.1 · p. 4, Table 1 · p. 5
4.2 Results
- 80.75 average on MTEB-Code for C2LLM-7B surpasses Seed1.6-Embedding and Qwen3-Embedding-8B across the 12 retrieval tasks.The leaderboard table presents the top 10 models as of the submission date.Source: Table 2 · p. 5, §4.2 · p. 5
- 75.46 average for C2LLM-0.5B exceeds INF-Retriever-7B's 69.70 and all other models below 1B parameters.The reported result places the smaller model in the compute-efficient regime.Source: §4.2 · p. 5
5 Conclusion
C2LLM combines causal LLM pretraining with flexible PMA pooling and sets new MTEB-Code records with its 7B model. It is the fourth CodeFuse Embedding family entry, with plans to expand into multilingual and multi-domain scenarios.
5 Conclusion
- C2LLM combines causal LLM pretraining with flexible PMA pooling, setting new MTEB-Code records with its 7B model.The conclusion describes the model family as bypassing the EOS-versus-mean-pooling dilemma to aggregate code-sequence information.Source: §5 · p. 5
- C2LLM is the fourth CodeFuse Embedding family entry, following D2LLM (Liao et al., 2024), E2LLM, and F2LLM (Zhang et al., 2025b).Source: §5 · p. 5
- The authors plan to expand the series into massively multilingual and multi-domain scenarios.They also state a commitment to promoting open research in LLM-based embedding models.Source: §5 · p. 6