Source-linked AI summary
C-Pack: Packed Resources For General Chinese Embeddings
Shitao Xiao, Zheng Liu, Peitian Zhang, Niklas Muennighoff, Defu Lian, Jian-Yun Nie
TL;DR
General Chinese embedding research lacked competitive models and comprehensive shared resources despite the importance of embeddings for varied NLP and retrieval applications. C-Pack assembles a benchmark, large training datasets, embedding models, and an integrated training recipe. Its BGE models outperform prior Chinese embedding models across C-MTEB, with especially strong gains on retrieval and related tasks.
Problem
General Chinese embedding development lacked competitive models and comprehensive resources for evaluating and training embeddings across varied tasks.
Method
C-Pack combines C-MTEB, C-MTP, BGE models, and an integrated recipe using pre-training, contrastive learning, and task-specific fine-tuning.
Results
BGE outperforms prior Chinese embedding models across C-MTEB, with the largest improvements in retrieval followed by STS, pair classification, and re-ranking.
Takeaways & Limitations
The publicly released resources support direct use of Chinese embeddings, reproduction of the training methods, and continued fine-tuning.
Abstract
from arXiv · showhide
We introduce C-Pack, a package of resources that significantly advance the field of general Chinese embeddings. C-Pack includes three critical resources. 1) C-MTEB is a comprehensive benchmark for Chinese text embeddings covering 6 tasks and 35 datasets. 2) C-MTP is a massive text embedding dataset curated from labeled and unlabeled Chinese corpora for training embedding models. 3) C-TEM is a family of embedding models covering multiple sizes. Our models outperform all prior Chinese text embeddings on C-MTEB by up to +10% upon the time of the release. We also integrate and optimize the entire suite of training methods for C-TEM. Along with our resources on general Chinese embedding, we release our data and models for English text embeddings. The English models achieve state-of-the-art performance on MTEB benchmark; meanwhile, our released English data is 2 times larger than the Chinese data. All these resources are made publicly available at https://github.com/FlagOpen/FlagEmbedding.
1 INTRODUCTION
General Chinese embeddings require broad data, suitable training methods, and comprehensive benchmarks, but competitive Chinese resources remained limited. C-Pack addresses these needs with C-MTEB, C-MTP, BGE models, and an integrated training recipe.
- Motivation: General-purpose embeddings must support varied tasks and applications, making them more challenging to develop than task-specific embeddings.The paper highlights retrieval, ranking, and classification across applications including question answering and language modeling.
- Motivation: Training data must be large, diverse, and cleaned because general embeddings require hundreds of millions of instances across sources.The paper cites web encyclopedias, question-answering platforms, news websites, and social media as relevant sources.
- Motivation: General embedding training depends on a suitable encoder and a compound recipe combining embedding-oriented pre-training, contrastive learning, and task-specific fine-tuning.The paper presents training as a combination of backbone selection and multiple complementary methods.
- Motivation: Chinese general embedding development was constrained by a shortage of competitive models and comprehensive evaluation resources compared with English-centric research.The paper identifies this shortage as a central motivation for C-Pack.
- C-Pack Resources: C-MTEB collects 35 Chinese datasets across 6 task types with unified protocols to evaluate embedding capabilities on a shared basis.Its evaluation pipeline is designed to simplify assessment and measure generality across major capabilities.
- C-Pack Resources: C-MTP provides 100M Chinese text pairs from web corpora and labeled datasets, while BGE offers small, base, and large models with different efficiency-effectiveness trade-offs.The models contain 24M, 102M, and 326M parameters, respectively.
- Results: BGE outperforms prior Chinese embedding models across C-MTEB, with especially notable gains in retrieval, STS, pair classification, and re-ranking.Classification and clustering improvements are less pronounced but remain comparable or slightly better than competitive models.
- Release: C-Pack releases an optimized recipe covering embedding-oriented pre-training, contrastive learning, and task-specific fine-tuning to support reproduction and continued progress.The project also reports broad community adoption and public release of its resources.
2 RELATED WORK
Prior work shows that general text embeddings depend on publicly available data, scalable models, optimized training, and broad benchmarks. C-Pack is motivated by the resource dependence of this field and the limited public availability of corresponding Chinese resources.
- Data: General text embeddings require large, diversified training data curated from web encyclopedias, QA platforms, news websites, and social communities.Earlier curated datasets were often not publicly available before C-Pack.
- Models: Scaling training and model size is associated with greater generalizability across application scenarios, but large-scale training remains burdensome for community users.The public release of BGE is presented as reducing this cost for direct use and downstream improvement.
- Training: Effective general embedding training combines pretraining, negative sampling, and multi-task fine-tuning rather than relying on a single algorithm.C-Pack integrates, optimizes, and pipelines these operations for BGE reproduction and continued fine-tuning.
- Benchmarks: Broad benchmarks are needed to evaluate embedding generality across many capabilities instead of using isolated task-specific evaluations.BEIR broadened retrieval evaluation, while MTEB extended this approach across a wider set of datasets and tasks.
- Conclusion: Because general text embedding is resource-dependent, progress requires coordinated datasets, models, training methods, and benchmarks.The paper presents public release of these resources as crucial for advancing the field.
3 C-PACK
C-Pack provides a Chinese embedding benchmark, large-scale training data, BGE model variants, and a public training recipe. Together, these resources standardize evaluation and support training general-purpose Chinese embeddings across diverse tasks.
- 3.1 Benchmark: C-MTEB: C-MTEB collects 35 datasets across six tasks—retrieval, re-ranking, STS, classification, pair-classification, and clustering—to evaluate Chinese embeddings comprehensively.The benchmark standardizes protocols and provides an evaluation pipeline for fair comparison.
- 3.1 Benchmark: C-MTEB: C-MTEB averages dataset performance within each task and across all datasets to measure task-level capability and overall generality.Results can be written to an output folder for submission to the C-MTEB leaderboard.
- 3.2 Training Data: C-MTP: C-MTP supplies large-scale, diversified paired texts extracted from unlabeled web corpora and other Chinese sources for general embedding training.Examples include title-body, subtitle-passage, question-answer, paraphrase, and related-document pairs.
- 3.3 Model Class: BGE: BGE offers BERT-like Chinese embedding models in large (326M), base (102M), and small (24M) sizes, enabling efficiency-effectiveness trade-offs and further fine-tuning.The models are intended for direct use and as foundations for stronger downstream embeddings.
- 3.4 Training Recipe: The released BGE training recipe combines plain-text pre-training, contrastive learning with unlabeled C-MTP, and multi-task learning with labeled C-MTP.Its components include embedding-oriented pre-training, in-batch negatives with batches up to 19,200, and instruction-based fine-tuning with mined hard negatives.
4 EXPERIMENTS
Experiments evaluate BGE and its components across Chinese and English embedding benchmarks, finding strong generality and improvements from data, model scaling, and training choices. Ablations show that C-MTP, pre-training, large-batch contrastive learning, and instructions each contribute to performance.
- General Evaluation: BGE outperforms existing Chinese text embeddings by large margins, especially on retrieval, STS, pair classification, and re-ranking.Classification and clustering performance remains comparable to or slightly better than competitive models.
- General Evaluation: 63.96 average performance is achieved by the large model, compared with 58.28 for the small model, with improvements across all evaluation tasks.The model-size trade-off supports high-precision use cases with large models and high-throughput scenarios with small models.
- General Evaluation: English BGE models achieve state-of-the-art performance across 56 MTEB datasets and advance the prior state of the art by 1.1 absolute points in total average.The models use the same training recipe as the Chinese models, alongside publicly released English training data.
- Detailed Analysis: C-MTP unlabeled and labeled data both substantially contribute to embedding quality, with unlabeled data supporting strong average and retrieval performance.The labeled data is smaller but higher quality and is used for task-specific fine-tuning after the unlabeled-data stage.
- Detailed Analysis: 63.96 average performance follows 59.0 after fine-tuning BGE-pretrain on C-MTP labeled data, with the largest gains on retrieval, re-ranking, STS, and pair classification.Other tasks preserve or marginally improve performance, indicating broad gains from diversified labeled data.
- Detailed Analysis: Instructions during task-specific fine-tuning improve average performance, with stronger advantages on retrieval, STS, pair classification, and re-ranking.The recipe also uses large batches for contrastive learning, enabled by gradient checkpointing and cross-device embedding sharing.