Baichuan4-Finance Technical Report

1 INTRODUCTION

• Financial documents combine complex numerical data and specialized terminology, requiring domain knowledge and advanced numerical processing and reasoning.The report describes financial LLMs as useful for interpreting implications in financial documents and converting unstructured data into actionable insights and market intelligence.Source: §1 · p. 3
• Baichuan4-Finance comprises a pre-trained base model and an instruction-tuned model for chat and downstream applications, both built on Baichuan4-Turbo.The base model is not yet aligned with human preferences; the chat model is fine-tuned for downstream use.Source: §1 · p. 3
• The development plan targets data quality and mixture, model design, and training strategy across continual pre-training and alignment.The authors identify these factors as influential to final-model performance.Source: §1 · p. 3

2.1 TOKENIZER

• BBPE tokenization produces a vocabulary of 141,056 regular tokens.The report says BBPE has a high compression rate relative to other tokenizers.Source: §2.1 · p. 4

2.2 MODEL ARCHITECTURE

• The architecture uses RMSNorm for pre-normalization to handle training stability.Source: §2.2 · p. 4
• Grouped-query attention replaces multi-head attention to improve inference speed and reduce key-value cache size during decoding.Source: §2.2 · p. 4
• RoPE serves as the model's positional encoding.Source: §2.2 · p. 4

3.1 PRE-TRAINING DATA COMPOSITION

• The pre-training corpus combines general data for world knowledge with financial data containing financial knowledge.The financial corpus includes news, press releases, finance books and journals, and social media posts; Figure 1 presents its composition.Source: §3.1 · p. 4, Figure 1 · p. 6

3.2 DATA QUALITY ENHANCEMENT

• Quality enhancement uses rule-based text cleaning, a logistic-regression classifier, and Pareto resampling that favors documents with higher quality scores.The classifier is trained with high-quality examples and raw unfiltered documents in a 1:1 ratio; the retention rule gives higher-quality-scored documents a higher retention probability.Source: §3.2 · pp. 4–5
• A second filtering stage scores readability, coherence, informativeness, safety, anonymity, and unattainable references, then trains an automatic scoring model to filter data.Source: §3.2 · p. 5
• The pipeline also uses abnormal-loss analysis, specialized filters for code, math, and Markdown, and URL-, document-, and line-level de-duplication.Anomalous-loss data is sampled for analysis; identified common issues are addressed with rules or learned by the XLM-RoBERTa model.Source: §3.2 · p. 5

3.3 DETERMINING THE DATA MIXTURE RATIO BY SCALING LAWS

• A two-stage scaling-law method estimates the performance of larger models across candidate data-mixture ratios using smaller models.It relates model size, dataset sizes, and mixture ratios to validation loss, then models the relation between validation loss and benchmark accuracy; the report uses it to determine the financial-data mixture ratio.Source: §3.3 · pp. 5–6

3.4 CONTINUAL PRE-TRAINING

• Domain self-constraint continual pre-training balances retaining general-model knowledge with learning financial knowledge.For financial documents, its objective combines a KL term and a language-modeling term; the report says this enables learning financial knowledge while maintaining general knowledge.Source: §3.4 · pp. 6–7

3.5 ANNEALING

• During the final 80M tokens of continual pre-training, the learning rate is linearly reduced to zero and the mixture ratio shifts toward high-quality financial sources.Source: §3.5 · p. 7

4.1.1 DEMONSTRATION DATA CONSTRUCTION

• Demonstration construction combines financial documents, manually built seed instructions across capabilities and complexity levels, and Turbo-generated responses reviewed and filtered by annotators.The process comprises finance-material, seed-repository, and large-scale demonstration construction.Source: §4.1.1 · pp. 7–8

4.1.2 TRAINING STRATEGY

• Over 160,000 high-quality demonstrations support supervised fine-tuning across instruction following, financial classification, clustering, and generation.The supervised fine-tuned model is produced using this dataset and the regular supervised fine-tuning process.Source: §4.1.2 · p. 8

4.2.1 PREFERENCE DATA CONSTRUCTION

• Human-ranked answers yield preference pairs for non-unique tasks, while verifier-checked math answers supply pairs for unique-answer tasks after all-correct and all-incorrect samples are removed.The resulting preference dataset is used to train the reward model.Source: §4.2.1 · p. 8

4.2.2 REWARD MODEL TRAINING

• The reward model starts from Baichuan4-Turbo with its language-modeling head replaced by a value head, and outputs a reward score for each prompt-answer pair.Its loss is formulated using prompts and their chosen and rejected answers.Source: §4.2.2 · p. 8

4.3 REINFORCEMENT LEARNING FROM HUMAN FEEDBACK AND AI FEEDBACK

• A PPO-based reward-robust framework addresses instability and imperfections in reward models; optimizing the supervised fine-tuned model with this framework yields Baichuan4-Finance.The model generates predictions for randomly selected preference-data prompts before reinforcement learning.Source: §4.3 · p. 8

5.1 BENCHMARKS

• Evaluation covers public general datasets, FinanceIQ and FLAME, including certification benchmarks and financial scenario applications.FinanceIQ comprises 7,173 multiple-choice questions across 10 certifications; FLAME includes 14 certifications and 10 scenario applications.Source: §5.1 · p. 9

5.2.1 ABLATION STUDY FOR DOMAIN SELF-CONSTRAINT CONTINUAL PRE-TRAINING

• On a 1B Base model, domain self-constraint improves financial performance while preserving general knowledge, unlike regular continual pre-training.Figure 2 presents the comparison across the three training approaches.Source: §5.2.1 · p. 9, Figure 2 · p. 10
• Baichuan4-Finance-Base surpasses its Baichuan4-Turbo backbone by significant margins on the financial benchmark, with comparable general-dataset performance.Figure 3 presents this comparison.Source: §5.2.1 · p. 9, Figure 3 · p. 10

5.2.2 ABLATION STUDY FOR PPO

• With the same training data, SFT plus PPO performs significantly better than SFT alone on eight mathematical datasets.Figure 4 visualizes this comparison.Source: §5.2.2 · p. 11, Figure 4 · p. 11
• The PPO analysis reports that similar shapes for two Figure 5 lines indicate positive correlation and suggest PPO could transfer pass@5 potential to pass@1 hit rate.The lines compare w/o-PPO pass@1 versus pass@5 performance and Baichuan4-Finance pass@1 versus w/o-PPO pass@1 performance.Source: Figure 5 · p. 11, §5.2.2 · p. 12

5.3.1 BASELINES

• For zero-shot comparisons, Baichuan4-Finance is evaluated against GPT-4o and two open-source models on FinanceIQ; FLAME results come from its official institute.The FLAME comparison includes five chat LLMs.Source: §5.3.1 · p. 12

5.3.2 COMPARISON RESULTS

• Baichuan4-Finance shows strong capabilities across financial certifications and application scenarios on FLAME-Cer, FLAME-Sce, and FinanceIQ.The comparison results are presented in Tables 1, 2, and 3, respectively.Source: Table 1 · p. 12, §5.3.2 · p. 12, Table 2 · p. 13

• Baichuan4-Finance-Base outperforms nearly all competitive baselines on financial tasks without compromising general LLM benchmark performance, while Baichuan4-Finance performs especially well in financial applications.Source: §6 · p. 14
• The report highlights a continual pre-training data pipeline and a data-mixture-ratio process based on two scaling laws.Source: §6 · p. 14
• The series includes Baichuan4-Finance-Base and the Baichuan4-Finance chat model, with domain self-constraint pre-training as a key training strategy.Source: §6 · p. 14