PaperlayerView exact revision on arXiv

Source-linked AI summary

MAP-Neo: Highly Capable and Transparent Bilingual Large Language Model Series

Ge Zhang, Scott Qu, Jiaheng Liu, Chenchen Zhang, Chenghua Lin, Chou Leuang Yu, Danny Pan, Esther Cheng, Jie Liu, Qunshu Lin, Raven Yuan, Tuney Zheng, Wei Pang, Xinrun Du, Yiming Liang, Yinghao Ma, Yizhi Li, Ziyang Ma, Bill Lin, Emmanouil Benetos, Huan Yang, Junting Zhou, Kaijing Ma, Minghao Liu, Morry Niu, Noah Wang, Quehry Que, Ruibo Liu, Sine Liu, Shawn Guo, Soren Gao, Wangchunshu Zhou, Xinyue Zhang, Yizhi Zhou, Yubo Wang, Yuelin Bai, Yuhan Zhang, Yuxiang Zhang, Zenith Wang, Zhenzhu Yang, Zijian Zhao, Jiajun Zhang, Wanli Ouyang, Wenhao Huang, Wenhu Chen

arXiv:2405.19327v4cs.CLcs.AIcs.LG

TL;DR

Competitive LLMs remain closed, while many open releases still omit important training details and truly open models lag on several capabilities. MAP-Neo offers a transparent bilingual LLM suite and its development resources; the authors report strong results against transparent models and matches or better performance on some benchmarks against frontier open-weight models.

  • Problem

    Competitive models remain closed, while open releases often omit data-cleaning and pretraining details relevant to reproducibility and trust.

    Source: §1 · p. 4, §11 · p. 23

  • Method

    MAP-Neo releases a bilingual LLM suite with its data-curation processes, Matrix Data Pile, training, and evaluation details.

    Source: §12 · p. 24

  • Results

    MAP-Neo performs significantly better than prior transparent models in math, code, and complex reasoning, and matches or surpasses frontier open-weight models on some math, code, and Chinese-knowledge benchmarks.

    Source: §10.1.2 · p. 22

  • Takeaways & Limitations

    The authors present MAP-Neo as a resource for advancing transparent NLP research and broadening access to advanced LLM technologies.

    Source: §12 · p. 24

    The NEO Scaling Law’s regularization has no lower loss bound as D approaches negative infinity, though the authors report reasonable loss for datasets under hundreds of trillions of tokens with their experimental d values.

    Source: §8.2 · p. 18

Abstract

from arXiv · show

Large Language Models (LLMs) have made great strides in recent years to achieve unprecedented performance across different tasks. However, due to commercial interest, the most competitive models like GPT, Gemini, and Claude have been gated behind proprietary interfaces without disclosing the training details. Recently, many institutions have open-sourced several strong LLMs like LLaMA-3, comparable to existing closed-source LLMs. However, only the model's weights are provided with most details (e.g., intermediate checkpoints, pre-training corpus, and training code, etc.) being undisclosed. To improve the transparency of LLMs, the research community has formed to open-source truly open LLMs (e.g., Pythia, Amber, OLMo), where more details (e.g., pre-training corpus and training code) are being provided. These models have greatly advanced the scientific study of these large models including their strengths, weaknesses, biases and risks. However, we observe that the existing truly open LLMs on reasoning, knowledge, and coding tasks are still inferior to existing state-of-the-art LLMs with similar model sizes. To this end, we open-source MAP-Neo, a highly capable and transparent bilingual language model with 7B parameters trained from scratch on 4.5T high-quality tokens. Our MAP-Neo is the first fully open-sourced bilingual LLM with comparable performance compared to existing state-of-the-art LLMs. Moreover, we open-source all details to reproduce our MAP-Neo, where the cleaned pre-training corpus, data cleaning pipeline, checkpoints, and well-optimized training/evaluation framework are provided. Finally, we hope our MAP-Neo will enhance and strengthen the open research community and inspire more innovations and creativities to facilitate the further improvements of LLMs.

1 Introduction

MAP-Neo is presented as a transparent LLM suite addressing gaps in model capability and disclosure. The paper positions openness as important for democratization and academic research, and describes MAP-Neo as both a model and a resource for building LLMs.

1 Introduction

  • Earlier transparent models generally lagged industry-level LLMs, especially in coding, reasoning, knowledge, and multilingualism, while releases varied in which training resources they disclosed.The cited examples include LLaMA-series weight releases without training-data details and more extensive releases such as LLM360 and Pythia [9].Source: §1 · p. 4
  • MAP-Neo integrates bilingual data-curation and cleaning code, the Matrix Data Pile and SFT/alignment data, architecture details, training code, and intermediate checkpoints.The training release includes tokenizer, base-model, instruction-tuned, and aligned-model code, plus Megatron-LM enhancements and the NEO Scaling Law.Source: §1 · p. 4
  • The report also provides stable-training infrastructure, evaluation code and settings, and optimization recipes and ablation-based lessons.These materials are presented as resources for stable training and benchmarking.Source: §1 · p. 4
  • The authors present MAP-Neo as both a foundational model and a comprehensive handbook for building LLMs, with particular relevance to non-English research communities.They describe the handbook as covering the full workflow of building LLMs from scratch.Source: §1 · pp. 4–5

2 Related Works

  • MAP-Neo-7B achieves the highest scores among transparent LLMs across C-EVAL [46], MMLU, GSM8K [23], MATH [41], and HumanEval [15], while meeting all transparency checks.Table 1 compares open-source models under the same evaluation manner and distinguishes transparent models below the dashed line.Source: Table 1 · p. 5, §2 · p. 5
  • MAP-Neo combines transparency components often missing from other models, and the authors identify OLMo [36] as similar but weaker in coding, knowledge, and mathematical reasoning and limited to English.The paper says this combination supports in-depth analysis and independent validation.Source: §2 · p. 5

3 Tokenizer

This section describes MAP-Neo’s tokenizer and data pipelines, covering corpus construction, filtering, deduplication, document conversion, and collection from web and domain-specific sources. It also reports iterative refinement and evaluation of data selection, as well as the effect of correcting a whitespace-removal setting on code metrics.

3 Tokenizer

  • Disabling whitespace removal corrected unstable code metrics, while the tokenizer uses 64,000 tokens, digit-split numbers, byte-level fallback, and no normalization or dummy prefixes.The authors report that enabling SentencePiece’s default extra-whitespace removal treated code indentation as a single space; fixing the bug in phase two stabilized and significantly improved code metrics, with rapid gains in decay training.Source: §3 · pp. 5–6
  • HQ Chinese text averaged 1.577 characters per token, versus 3.311 for HQ English, and the authors note tokenizer settings can affect pre-training even at similar compression rates.They call for further study of tokenization strategies for subsequent usage scenarios.Source: §3 · p. 6

4 Matrix Data Pile

  • Matrix is a 4.5T-token bilingual corpus; 52.55% comes from Common Crawl, 22.29% from programming code, and the remainder from academic papers, books, and other printed materials.The paper also proposes collecting high-quality mathematics, scientific-exam synthetic, and wiki-based data through an iterative pipeline.Source: §4 · pp. 6–7, §4.4 · p. 12

4.1 Re-processing Pipeline for Open Datasets

  • The English subset reprocesses several open web corpora using source-tailored heuristic filtering, quality annotations where available, and dataset-specific threshold checks.Filtering includes document- and sentence-level checks, duplicate removal, and sensitive-word screening.Source: §4.1 · p. 7, §4.1.1 · pp. 7–8

4.1.2 Deduplication

  • English deduplication combines exact-document removal, MinHash LSH for near duplicates, and paragraph and exact-substring methods for repeated text.The methods target both whole-document and partial-text repetition; paragraph removal can interfere with content analysis.Source: §4.1.2 · p. 8

4.2 Corpora Crawl from Scratch Pipeline

  • The Chinese Matrix subset is 80.6% crawled Chinese web pages, with filtering tailored to HTML artifacts and Chinese text characteristics, including distinctions between characters and words.The rest comes from several open datasets.Source: §4.2 · p. 9, §4.2.1 · pp. 9–10

4.2.2 Deduplication

  • Chinese deduplication uses similar-line matching because repeated crawls and minor extraction losses can make exact-substring removal retain extra words and reduce readability.The method compares lines using edit distance, after quick dissimilarity checks based on length difference and character-set overlap.Source: §4.2.2 · p. 10, §4.2.3 · p. 10

4.3 Document Conversion Pipeline

  • The document-conversion framework separates layout detection, element recognition, ordering, and post-processing so components can be upgraded or replaced independently.It combines tools for layout parsing, formula and text recognition, table reconstruction, and ordering complex layouts; post-processing repairs broken sentences, hyphenated words, and formulas.Source: §4.3 · pp. 11–12

4.4 High-Quality Supplement Data Collection

  • The workflow trains a FastText classifier on seed-dataset samples as positive examples and an equal amount of Common Crawl data as negatives, then ranks crawl data by confidence.It uses three epochs, a 0.1 learning rate, 256-dimensional embeddings, and 3-grams; quantization reduces model size to about 10% of its original footprint.Source: §4.4 · p. 12
  • ChatGPT 3.5 [1] evaluates retained data by URL, annotates data from domains with over 10% field-specific content, and supports iterative classifier refinement.Unconfirmed positives from earlier iterations are added to the training samples before the next cycle.Source: §4.4 · p. 13
  • Common Crawl data comes from English RedPajama V2, with mathematics seeds from OpenWebMath, science seeds from named research and exercise-data domains, and wiki data from wiki websites.The science sources include Chemrxiv, biorxiv, wanjuan-exam, WebInstruct [117], and Web Of Science [55].Source: §4.4 · p. 13

5 Model

MAP-Neo uses a transformer decoder with an 8192-token context and several architectural enhancements. The paper compares dense 2B- and 7B-parameter models built with the listed architecture hyperparameters.

5.1 Model Architecture

  • The models use a transformer decoder with an 8192-token context; Table 5 lists their architectural parameters, including layer, width, head, and feed-forward dimensions.The feed-forward size is 8 × dmodel.Source: §5.1 · p. 13, Table 5 · p. 13
  • The 7B model uses multi-head attention, while the 2B checkpoints use single-key-value-head multi-query attention, which ablations found effective at smaller scales.Both variants use rotary positional embeddings shared between inputs and outputs, RMSNorm, and SwiGLU.Source: §5.1 · p. 13

5.2 Model Scale Hyperparameters

  • The study compares standard dense Transformers at 2B and 7B parameters, trained with the same vocabulary and batch size but different training data.Their hyperparameters are listed in Table 5.Source: §5.2 · p. 13, Table 5 · p. 13

6 Pre-training

MAP-Neo pre-training has a fundamental phase for general text-generation capability and a decay phase that uses higher-quality and more code data. The latter also addresses tokenizer-related code-generation failures.

6 Pre-training

  • Pre-training first uses a fundamental phase on generic text, then a decay phase that adds high-quality and more code data; Figure 5 shows the phase-specific data-mixture ratios.The decay phase uses Stack V2, whereas the fundamental phase repeated Stack V1 twice because Stack V2 was not yet available.Source: §6 · p. 14, Figure 5 · p. 14

6.1 Fundamental Phase: General Ability Acquisition

  • The fundamental phase processes about 3,726 billion tokens to train general text-generation capability on diverse textual data.Its learning rate warms up from 2 × 10^-5 to 2 × 10^-4 over 2k steps, then cosine-decays to 2 × 10^-5 over about 365k steps.Source: §6.1 · p. 14

6.2 Decay Phase: Improvement and Rectification

  • The decay phase follows tokenizer-related code-generation test failures and is designed to use a fixed-version tokenizer while improving code-generation performance.The phase also increases code data and incorporates high-quality data; Figure 5 depicts the data-mixture ratios in the fundamental and decay phases.Source: Figure 5 · p. 14, §6.2 · p. 15
  • The decay phase processes about 778 billion tokens, mainly high-quality instruction data, and raises the code-data proportion from 14.77% to 17.04%.Its learning rate starts at 2 × 10^-4 and exponentially decays over 148k steps.Source: §6.2 · p. 15

7 Alignment

MAP-Neo alignment combines two phases of supervised fine-tuning with three iterations of preference optimization. The first SFT phase builds foundational skills, the second targets chat, and iterative DPO improves reported AlpacaEval2.0 win rates.

7.1 Supervised Fine-tuning

  • Two-phase SFT first builds foundational abilities, then fine-tunes the model on chat-style queries and responses to improve chat ability.The stated aim is to retain foundational skills while producing natural, helpful, contextually accurate responses.Source: §7.1 · p. 15, §7.1.1 · p. 15

7.1.1 Data

  • The foundational SFT phase uses over 2 million instruction examples, including OpenHermes 2.5 without TheoremQA segments, Code-Feedback, and some WebInstructSub data.The TheoremQA segments are excluded to prevent benchmark data leakage.Source: §7.1.1 · p. 15
  • The chat phase uses over 100k multi-turn conversations and adds 5k math and code examples from the first phase to help retain foundational skills.The paper reports improved chat-benchmark performance without compromising foundational abilities.Source: §7.1.1 · p. 15

7.1.2 Training

  • SFT uses next-token prediction with loss masks on system and user inputs; the first phase runs 3 epochs and the second runs 1 epoch.Both phases use sequence length 8192, batch size 512, and the same hyperparameters.Source: §7.1.2 · pp. 15–16

7.2 Iterative DPO

  • AlpacaEval2.0 length-controlled win rate improves with each of three iterative DPO rounds, which generate response pairs, label them with reward models, and train with DPO loss.The pipeline uses Nectar prompts, Starling-RM-34B, and Chinese preference data in the third iteration.Source: Table 7 · p. 16, §7.2 · p. 16

8 Scaling Law of MAP-Neo

The paper proposes NEO scaling law to predict loss and select compute-constrained training configurations for large, diverse pre-training datasets. It reports a close predicted-to-actual loss match for MAP-Neo and better fitting than Chinchilla for the largest MAP-Neo and DeepSeek models.

8.1 Problem Definition

  • Scaling-law fitting aims to minimize validation cross-entropy under compute constraints; the experiments use training loss because the data are described as almost non-repetitive and high-quality.Models with 250M, 460M, and 980M parameters, trained on 1000B tokens, are used to predict the 7.8B model's scaling behavior.Source: §8.1 · p. 17, §8.2 · p. 17, Figure 6 · p. 17

8.2 NEO Scaling Law

  • The observed loss falls faster than Chinchilla predicts, motivating NEO's additional −d · log(D) term for datasets spanning several trillion tokens and diverse corpora.The paper notes that this term has no theoretical lower bound as D approaches negative infinity, but reports d values of 1e-2 to 3e-2 and reasonable losses below hundreds of trillions of tokens.Source: §8.2 · pp. 17–18
  • The NEO scaling law yields significantly better fitting results on both training and testing sets than the compared scaling laws.Table 8 compares parametric fitting by R2 and Huber loss.Source: §8.2 · p. 18, Table 8 · p. 18
  • 0.6591 real training loss was close to the NEO-predicted 0.6618 loss for the 7.8B-parameter model trained on 3.07T tokens.The paper reports that this agreement demonstrates the effectiveness of the NEO scaling law; it also predicts 0.6597 loss for a 10B model trained on 2.5T tokens under 1.5 × 10^23 FLOPs.Source: §8.2 · p. 18

8.3 Generalization of NEO Scaling Law

  • For MAP-Neo-7B and DeepSeek-67B, NEO fits better than Chinchilla, whose predictions underestimate loss at small dataset sizes and overestimate it as model and data scale up.The comparison is shown for DeepSeek models trained on corpora including Chinese, English, and code; the authors suggest NEO may suit large, diverse, high-quality datasets.Source: §8.3 · p. 18, Figure 7 · p. 18

9 Infrastructure

The infrastructure combines distributed data processing for English and Chinese with systems for pre-training and fine-tuning. Training uses distributed parallelism and checkpoint-based recovery on a multi-node H800 cluster.

9 Infrastructure

  • The training system supports pre-training and fine-tuning, while the data-processing system handles English and Chinese datasets.Data processing uses Spark and object storage; the system includes 94 machines for English processing and 14 for Chinese processing.Source: §9 · p. 19
  • Megatron-Core training reaches 7,200 TPS for a 7B model, compared with 6,400 TPS under the same settings without it.The pipeline also removes faulty nodes, prevents overflow on large corpora, and uses selected checkpoints for recovery.Source: §9 · p. 19
  • The 7B model uses 512 H800 GPUs across 64 nodes, with two-GPU tensor parallelism and optimizer-state sharding similar to ZeRO-1.The 2B models use 256 GPUs with tensor model parallelism set to 1; the cluster uses NCCL for distributed communication.Source: §9 · p. 19

10 Evaluations

The evaluation reports that MAP-Neo matches or surpasses similar-sized open-weight industry models on some math, code, and Chinese-knowledge benchmarks, while outperforming other transparent LLMs on several capabilities. The section also describes evaluations of base and aligned models across academic and conversational benchmarks.

10 Evaluations

  • The evaluation reports distinctive performance in code, math, and instruction following across automatic benchmarks of base and chat models.The authors describe the model family as having academic and practical value.Source: §10 · p. 20

10.1.1 Main Results

  • Base-model comparisons use public English and Chinese benchmarks, with perplexity for multiple-choice tasks and generated-text parsing for generation tasks.The authors state that they do not post-process evaluation content.Source: §10.1.1 · p. 22
  • Evaluated models were tested on reasoning (0-shot except MMLU 5-shot), code generation (pass@1, 0-shot), world knowledge (0-shot), SQuAD2.0 (0-shot), MATH/GSM8K average scores (4-shot), and Chinese tasks (5-shot).Source: §10.1.1 · p. 22

10.1.2 Discussions

  • MAP-Neo shows better math, code, and complex-reasoning performance than earlier transparent LLMs such as Amber and Pythia, which the authors describe as using presumably lower-quality data.This discussion links the performance difference to the quality of training data, with the qualifier “presumably” applying to the earlier models’ data.Source: §10.1.2 · p. 22
  • MAP-Neo matches or surpasses similar-sized industry open-weight models on some math, code, and Chinese-knowledge benchmarks, while transparent LLMs generally lag behind.The discussion calls for more participation in transparent LLM development.Source: §10.1.2 · p. 22

10.2.1 Main Results

  • Aligned-model benchmarks assess Chinese alignment, instruction following, real-world performance, human preferences, and Chinese cultural knowledge.The section also evaluates MMLU-Pro with a 5-shot configuration.Source: §10.2.1 · pp. 22–23

10.2.2 Discussions

  • Neo-7B-Instruct significantly improves on chat-related benchmarks over Neo-7B-SFT, including AlignBench, AlpacaEval, Arena-Hard, and CHC-Bench.Table 10 compares aligned models across benchmarks; the authors attribute the improvement to Iterative DPO.Source: Table 10 · p. 21, §10.2.2 · p. 23
  • The authors suggest that limited base-model capabilities may constrain chat-benchmark performance, noting poor results for Amber-7B-Chat and OLMo-7B-Instruct.This is presented as an assumption about the observed performance, not as a demonstrated cause.Source: §10.2.2 · p. 23

11 Societal Impact

The section argues that limited disclosure of data sources and processing methods leaves important parts of LLM development opaque. It presents MAP-Neo’s openness as a possible resource for Chinese-language model development and for organizations facing cost or privacy constraints.

11 Societal Impact

  • Many open models disclose weights but keep data sources, pre-training code, and data-processing methods opaque, limiting reproducibility and understanding.The section notes that training an LLM may be uneconomical for firms with financial constraints or privacy concerns.Source: §11 · p. 23
  • MAP-Neo may complement the scarcity of Chinese LLM corpora by disclosing data sourcing, cleaning, and pre-training processes.The authors suggest these disclosures could reduce deployment and customization costs, especially for a Chinese LLM.Source: §11 · p. 24
  • The authors suggest that open, economical models could help SMEs and universities build customized LLMs with less dependence on large technology firms.They argue that independent control of data and AI processes could support data and AI democratization.Source: §11 · p. 24

12 Conclusion

The conclusion presents MAP-Neo as a fully open-source bilingual LLM suite that shares details from data curation through evaluation and narrows gaps with industry-level models. The paper also describes the M-A-P community’s broader focus on open research in text, audio, and vision.

12 Conclusion

  • MAP-Neo shares data curation, the Matrix Data Pile corpus, model training, and evaluation details to support transparent NLP research.The authors report enhanced reasoning, instruction-following, and coding abilities and say the model narrows the gap with industry-level models.Source: §12 · p. 24

14 Multimodal Art Projection

  • The M-A-P community works on AI-generated content across text, audio, and vision, aiming to encourage open research on model training and data collection.The stated scope also includes developing applications.Source: §14 · p. 25

A Appendix

The appendix identifies English and Chinese heuristic-rule tables, a Megatron-core overflow issue, and intermediate-checkpoint evaluation prompts. It also gives prompt formats and examples for several datasets.

A.1 Details of Heuristic Rules for English Texts

  • Table 11 lists heuristic rules for English texts.The supplied passage identifies the table's scope but does not specify individual rules.Source: Table 11 · p. 36

A.2 Details of Heuristic Rules for Chinese Texts

  • Table 12 lists heuristic rules for Chinese texts.The supplied passage identifies the table's scope but does not specify individual rules.Source: Table 12 · p. 37

A.3 Training Framework Overflow Details

  • An overflow can occur when int64_t num_samples exceeds the int32_t capacity of sample_idx, potentially causing memory leaks and undefined behavior.Figure 8 illustrates the framework code modification.Source: §A.3 · p. 38, Figure 8 · p. 38

A.4 Detailed Prompts in Intermediate Checkpoints Evaluation

  • The evaluation appendix notes that several datasets combine prompts with questions and answer options, while listing prompts for other datasets separately.The listed combined-prompt datasets include BoolQ, PIQA, SIQA, HellaSwag, WinoGrande, ARC, OpenBookQA, CommonsenseQA, MMLU, CEval, and CMMLU.Source: §A.4 · p. 38

BoolQ

  • BoolQ uses a passage-and-question prompt with a Yes/No answer format.Source: BoolQ · p. 38

HumanEval

  • HumanEval prompts request Python code completion, with examples covering functions such as finding similar tuple elements and identifying non-prime numbers.The examples also include a heap-queue task and a generic prompt-and-tests template.Source: HumanEval · pp. 38–39

NaturalQuestions; Triviaqa

SQuAD 2.0

The SQuAD 2.0 appendix shows question-answering prompts that include step-by-step reasoning and an instruction for unanswerable questions. It also supplies worked examples and a separate TheoremQA prompt.

SQuAD 2.0

  • The appendix also includes worked mathematical prompts on domains, determinants, and systems of equations.Source: SQuAD 2.0 · pp. 39–40
  • Worked examples show stepwise solutions for a basketball scoring problem and a study-planning problem with breaks and lunch included.Source: SQuAD 2.0 · pp. 40–41
  • Other SQuAD 2.0 examples include multi-step arithmetic for marbles, frisbees, and cards, and for fruit counts across baskets.Source: SQuAD 2.0 · pp. 41–42
  • SQuAD 2.0 prompts ask the model to think step by step, while a separate format instructs it to answer “impossible to answer” when the passage does not support an answer.Source: SQuAD 2.0 · pp. 39, 42

TheoremQA

  • TheoremQA's prompt asks for a solution with step-by-step reasoning and constrains the final answer to specified forms, such as a number, list, truth value, or option.Source: TheoremQA · p. 42

A.5 Detailed Results

The appendix documents intermediate-checkpoint evaluations and additional scaling-law, compression, and OCR materials. Its scaling-law discussion compares loss predictions and describes data-diversity patterns, while noting that the available evidence is limited.

A.5 Detailed Results

  • Intermediate-checkpoint evaluations use OpenCompass and report results across benchmark datasets and training-token ranges, with Figure 9 covering MMLU, CEval, CMMLU, HellaSwag, GSM8K, and ARC-c.Tables 13–15 cover token ranges from 20B to 3726.33B, including decay-phase models.Source: §A.5 · p. 43, Table 13 · p. 43, Table 14 · p. 44

A.8 Additional Experimental Results in Scaling Law

  • For OLMo, the Chinchilla law fits the observed loss and does not underestimate loss for large models with small data volumes such as MAP-Neo 7B and DeepSeek 67B.The accompanying figure plots the loss curve and Chinchilla and NEO Scaling Law predictions using 1B and 7B loss values.Source: §A.8 · p. 46, Figure 10 · p. 46
  • The scaling-law discussion says data diversity can slow loss reduction below 500 billion tokens for models of 7 billion parameters or more, while above 1.5 trillion tokens it can sustain loss decreases beyond Chinchilla's flattening trajectory.It notes that evidence is limited because few models have been pretrained across multiple large, high-quality corpora.Source: §A.8 · p. 46
  • The appendix lists open-source pretraining datasets and defines “n ROUND” as the number of rounds for each dataset.Source: Table 16 · p. 47, Table 17 · p. 47

A.9 Compression Rate

  • Table 18 presents compression rates by category and dataset.Source: Table 18 · p. 48

A.10 OCR Post Processing

  • Table 19 lists OCR prompt templates with demonstrations in Chinese and English for OCR post-processing.Source: Table 19 · p. 49
Loading 2405.19327v4…