PaperlayerView exact revision on arXiv

Source-linked AI summary

Bee: A High-Quality Corpus and Full-Stack Suite to Unlock Advanced Fully Open MLLMs

Yi Zhang, Bolin Ni, Xin-Sheng Chen, Heng-Rui Zhang, Yongming Rao, Houwen Peng, Qinglin Lu, Han Hu, Meng-Hao Guo, Shi-Min Hu

arXiv:2510.13795v4cs.CVcs.AI

TL;DR

Fully open MLLMs are held back by noisy SFT data, insufficient complex reasoning examples, and opaque curation practices. The paper introduces Honey-Data-15M and the HoneyPipe/DataStudio suite, then trains Bee-8B on the dataset; Bee-8B reaches SOTA among fully open models while competing with semi-open systems. The approach prioritizes scalable, reproducible data quality, though fidelity verification trades recall for precision and cost-effectiveness.

  • Problem

    Fully open MLLMs face noisy SFT datasets, insufficient complex reasoning data, and a lack of transparent, reproducible curation methods.

  • Method

    The paper builds Honey-Data-15M with systematic cleaning and dual-level short- and long-CoT enrichment using HoneyPipe and DataStudio.

  • Results

    Bee-8B establishes a new SOTA among fully open MLLMs and is competitive with, or sometimes surpasses, recent semi-open models.

  • Takeaways & Limitations

    The work provides a transparent, adaptable data-curation suite and supports data quality as a practical path toward stronger fully open MLLMs.

  • Takeaways & Limitations

    Single-pass fidelity verification prioritizes precision and cost-effectiveness over recall, discarding some valid enriched data to preserve scalability.

Abstract

from arXiv · show

Fully open multimodal large language models (MLLMs) currently lag behind proprietary counterparts, primarily due to a significant gap in data quality for supervised fine-tuning (SFT). Existing open-source datasets are often plagued by widespread noise and a critical deficit in complex reasoning data, such as Chain-of-Thought (CoT), which hinders the development of advanced model capabilities. Addressing these challenges, our work makes three primary contributions. First, we introduce Honey-Data-15M, a new SFT dataset comprising approximately 15 million QA pairs, processed through multiple cleaning techniques and enhanced with a novel dual-level (short and long) CoT enrichment strategy. Second, we introduce HoneyPipe, the data curation pipeline, and its underlying framework DataStudio, providing the community with a transparent and adaptable methodology for data curation that moves beyond static dataset releases. Finally, to validate our dataset and pipeline, we train Bee-8B, an 8B model on Honey-Data-15M. Experiments show that Bee-8B establishes a new state-of-the-art (SOTA) for fully open MLLMs, achieving performance that is competitive with, and in some cases surpasses, recent semi-open models such as InternVL3.5-8B. Our work delivers to the community a suite of foundational resources, including: the Honey-Data-15M corpus; the full-stack suite comprising HoneyPipe and DataStudio; training recipes; an evaluation harness; and the model weights. This effort demonstrates that a principled focus on data quality is a key pathway to developing fully open MLLMs that are highly competitive with their semi-open counterparts.

1 INTRODUCTION

Fully open MLLMs face a data-quality gap caused by noisy SFT datasets, insufficient complex reasoning data, and opaque curation practices. The paper addresses these issues with Honey-Data-15M, HoneyPipe, and Bee-8B, whose results establish strong performance among fully open models.

  • Motivation: Fully open MLLMs lag behind proprietary and semi-open systems because competing on data volume is less viable than improving SFT data quality.Open datasets contain factual, structural, formatting, and image-instruction problems, while proprietary curation resources remain private.
  • Contributions: HoneyPipe and DataStudio provide a transparent, adaptable curation methodology intended to move beyond static dataset releases.The paper identifies opaque or undisclosed curation recipes as a major limitation of prior resources.
  • Contributions: Honey-Data-15M combines systematic noise removal with dual-level CoT enrichment tailored to instruction complexity.It contains 12.1M short-CoT pairs for moderate reasoning and 2.9M long-CoT pairs for the most complex instructions.
  • Validation: Bee-8B achieves SOTA among fully open MLLMs and is competitive with, or sometimes surpasses, semi-open models such as InternVL3.5-8B.Ablations also report significant benchmark improvements from the curation process compared with original, unprocessed data.

2 HONEY-DATA-15M AND HONEYPIPE

HoneyPipe transforms diverse raw multimodal data into Honey-Data-15M through deduplication, rule- and model-based filtering, and dual-level CoT enrichment. Short reasoning is applied broadly, while a verified long-CoT loop targets the most complex instructions, producing a 15M-sample dataset.

  • Pipeline Overview: HoneyPipe is an automated, reproducible workflow built from DataStudio that converts raw data into a transparent, dual-level-CoT SFT dataset.Its architecture combines a large-scale foundational path with a specialized long-CoT loop for complex instructions.
  • Stage 1: Data Aggregation and Preparation: Approximately 24M image-text pairs are aggregated and deduplicated only when both perceptual image hashes and textual instruction simhashes match.The process targets redundancy while preserving diversity across community datasets.
  • Data Filtering: Rule-based operators remove formatting and image-quality problems, while Qwen2.5-VL-72B checks image-instruction consistency before enrichment.Together these operators prune flawed samples and produce a cleaner image-instruction foundation.
  • Short CoT Enrichment: Short CoT enrichment uses Qwen2.5-VL-72B/32B to convert simple responses into step-by-step reasoning, producing approximately 12.1M samples.Samples unsuitable for reasoning enhancement, such as OCR or object detection data, bypass this process.
  • Long CoT Enrichment: A second enrichment loop sends complex or failed-verification samples to a more powerful model, then retains approximately 2.9M long-CoT responses that pass fidelity verification.Responses failing final verification are discarded as erroneous, unsolvable, or too costly to annotate.
  • Dataset Composition: Honey-Data-15M comprises 15M curated multimodal samples spanning domains such as General and STEM through its dual-level reasoning design.The short-CoT and long-CoT components provide foundational inference and deeper multi-step reasoning coverage.

3 BEE-8B TRAINING RECIPE

Bee-8B is trained with a five-stage recipe that progresses from vision-language alignment to complex reasoning and output refinement. The recipe uses Honey-Data-15M for large-scale reasoning-oriented SFT and Honey-Data-1M for targeted polishing.

  • Model: Bee-8B is developed as an 8B multimodal model to validate Honey-Data-15M’s ability to foster advanced complex reasoning.Its architecture builds on Qwen3-8B for reasoning and text generation and uses SigLIP2-so400mpatch14-384 for visual understanding.
  • Training Data: Honey-Data-15M combines data from seven major categories, with starred sources containing long Chain-of-Thought responses.Figure 3 reports source-level sample counts in thousands.
  • Training Overview: Bee-8B uses a five-stage training process that builds capabilities from basic perception through complex reasoning and output reliability.The stages are summarized in Table 1.
  • Foundational Training: The first two stages align visual and language representations through projector-only MLP warmup followed by full-parameter vision-language training.The second stage uses image-text pairs and text-only data while preserving the language model’s pretrained cognitive abilities.
  • Instruction Tuning: The third stage applies large-scale SFT to Honey-Data-15M to instill complex reasoning patterns from its dual-level CoT data.The fourth stage uses Honey-Data-1M, a curated subset with a more rational topic distribution, for efficient refinement SFT.

4 EXPERIMENTS

Experiments evaluate Bee-8B against fully open and semi-open MLLMs and use ablations to assess Honey-Data-15M and its curation strategy. Bee-8B performs especially strongly in factual accuracy, complex reasoning, and structured visual tasks.

  • Main Results: Bee-8B improves across benchmarks over existing fully open models and remains highly competitive with recent semi-open MLLMs.The largest advantages are reported in factual accuracy and complex multi-step reasoning; Table 2 contains the benchmark comparisons.
  • General VQA Tasks: 66.8 on MMMU and 71.4 on MMStar demonstrate competitive performance across broad, multi-domain visual question answering.These results are presented as evidence of Bee-8B’s strong general knowledge base.
  • Document, Chart, and OCR Tasks: 86.7 on ChartQA highlights Bee-8B’s performance on structured visual content, including documents, tables, and charts.The paper describes chart analysis as evidence of robust data parsing abilities.
  • Math and Reasoning Tasks: 67.0 on MathVerse is the RL-tuned Bee-8B score, described as top-ranking among the evaluated models.The paper identifies complex math and reasoning as the area with the most significant advancements.
  • HoneyPipe Ablation: The HoneyPipe ablation compares raw Draw data, filtered and short-CoT-enriched Dcurated data, and an otherwise identical Dno-CoT set without new CoT responses.Figure 4 attributes the lift from Draw to Dno-CoT to noise filtering and the larger lift to Dcurated to CoT enrichment.
  • Honey-Data-1M Ablation: Honey-Data-1M outperforms both the Qwen2.5-VL-7B base model and a Random-1M subset across the evaluated comparison.The study evaluates variants fine-tuned from the Stage 2 checkpoint, and the Honey-Data-1M model beats the original base model on nearly half of the evaluated benchmarks.

5 RELATED WORK

The related-work discussion situates MLLMs within the recent shift toward general language models that process visual inputs and open-source multimodal systems.

  • MLLM Development: GPT-4V helped establish the modern wave of MLLMs by showing that a general LLM could process visual inputs.
  • Open-Source MLLMs: LLaVA established a standard open-source paradigm for multimodal language modeling.The supplied discussion introduces LLaVA as part of the open-source community’s development of MLLMs.
  • Motivation: The discussion places MLLMs in the context of applications spanning diverse real-world scenarios.

6 AVAILABILITY

The work commits to releasing the resources needed to reproduce, inspect, train, and evaluate the Bee-8B system. The release includes the dataset, model, curation stack, scripts, configurations, and intermediate training materials.

  • Core Releases: The release includes the complete Honey-Data-15M corpus and the final Bee-8B model weights.
  • Software: HoneyPipe, DataStudio, source code, scripts, configurations, and training and evaluation code are slated for public release.These resources support examination and reproduction of the data curation and modeling pipeline.
  • Training Resources: Intermediate checkpoints for each training stage and specific training-data subsets will also be released for community analysis and research.

7 CONCLUSION

The paper addresses open-source MLLM data-quality gaps with Honey-Data-15M, HoneyPipe, and Bee-8B. Bee-8B reaches SOTA among fully open models, with strong gains in complex math and reasoning attributed to curated CoT data and validated through ablations.

  • 7 CONCLUSION: Bee-8B establishes a new SOTA among fully open MLLMs and competes with, sometimes surpassing, leading semi-open models.The final model is trained on the full Honey-Data-15M dataset.
  • 7 CONCLUSION: The work releases Honey-Data-15M, HoneyPipe, DataStudio, training resources, evaluation tools, and Bee-8B weights to support reproducible open MLLM development.The release covers the dataset, curation methodology, model, and supporting training and evaluation artifacts.
  • 7 CONCLUSION: Bee-8B shows standout performance in complex math and reasoning, which the authors link to targeted dual-level CoT enrichment.The dataset uses short- and long-CoT reasoning structures tailored to instruction complexity.
  • A.1 ABLATION OF DIFFERENT REASONING MODES AND STAGES: Ablations compare Stage 3 SFT, Stage 4 refinement SFT, and Stage 5 reinforcement learning under short- and long-CoT inference modes.The evaluation spans general VQA, document and chart understanding, and mathematical and logical reasoning benchmarks.
  • A.1 ABLATION OF DIFFERENT REASONING MODES AND STAGES: Stage 4 refinement and Stage 5 reinforcement learning progressively improve performance, especially on complex math and reasoning.The authors attribute refinement gains to a curated 1M subset and reinforcement learning gains partly to reduced response repetition.

A.2 SENSITIVITY ANALYSIS OF JUDGE MODELS

The supplementary analyses test whether Bee-8B’s results depend on the evaluator, inference run, human preference alignment, data leakage, or verifier choice. Results are broadly consistent across judges and runs, with minimal contamination and strong inter-model agreement.

  • A.2 SENSITIVITY ANALYSIS OF JUDGE MODELS: Independent GLM and Qwen judges produce highly consistent Bee-8B-SFT and Bee-8B-RL evaluations across 29 benchmarks.The cross-family evaluation was designed to quantify possible Qwen-family self-preference bias.
  • A.2 SENSITIVITY ANALYSIS OF JUDGE MODELS: Bee-8B-RL’s Global Average shifts only -0.3 points, from 70.2 with the Qwen judge to 69.9 with the GLM judge.Differences are most notable on ChartQA, DocVQA, and InfoVQA, where the GLM judge is stricter.
  • A.2 SENSITIVITY ANALYSIS OF JUDGE MODELS: Five independent inference runs yield extremely low standard deviations and stable global averages, indicating consistent and reproducible performance.The analysis reports each run, the mean, and the standard deviation for both SFT and RL models.
  • A.2 SENSITIVITY ANALYSIS OF JUDGE MODELS: Human evaluators preferred enriched responses for Reasoning at 72.74% and Expression Style at 69.92%, while Accuracy had an 83.65% Tie rate.The blind evaluation comprised 532 pairwise comparisons across 38 questions and four dimensions.
  • A.2 SENSITIVITY ANALYSIS OF JUDGE MODELS: Only 29 of 66,682 evaluation samples were potential overlaps under a relaxed threshold, including 2 exact matches.MMMU, MMMU-Pro, MMStar, and CharXiv remained completely free of overlap.
  • A.2 SENSITIVITY ANALYSIS OF JUDGE MODELS: Three verifier models retained 63,203 common samples, representing 96.5% of data retained by the reference model and 91.3% by the chosen verifier.The agreement spans verifier scales from 32B to 235B.

C REPRODUCIBILITY STATEMENT

The authors commit to full reproducibility by releasing the dataset, model artifacts, intermediate training resources, and complete curation code. They also acknowledge inherited source-data biases and possible harmful or inaccurate Bee-8B outputs.

  • C REPRODUCIBILITY STATEMENT: The release will include Honey-Data-15M, Bee-8B weights, intermediate checkpoints, stage-specific training data, HoneyPipe code, and DataStudio code.This package is intended to enable full reproduction of the training and curation process.
  • C REPRODUCIBILITY STATEMENT: Honey-Data-15M may inherit biases from public sources, and Bee-8B may generate inaccurate or harmful content.These risks remain despite the pipeline’s filtering of noise, factual errors, and image-instruction mismatches.

E.1 LIMITATION OF FIDELITY VERIFICATION

HoneyPipe’s single-model Fidelity Verification prioritizes scalable, reproducible precision over complete recall. The paper also notes that LLM-based evaluation introduces judge-dependent uncertainty and unresolved standardization challenges.

  • E.1 LIMITATION OF FIDELITY VERIFICATION: A single-pass verifier may reject logically correct CoT responses when valid variations conflict with the original answer.Examples include formatting differences, unit conversions, and synonyms.
  • E.1 LIMITATION OF FIDELITY VERIFICATION: Fidelity Verification trades recall for precision and cost-effectiveness to remain scalable and accessible at tens-of-millions scale.The authors acknowledge that some valid enriched samples may be discarded.
  • E.1 LIMITATION OF FIDELITY VERIFICATION: Using one verifier is less robust than an ensemble in principle, although the authors report high inter-model agreement for factual-consistency checking.The single strong open-source verifier is presented as a balance between robustness and engineering complexity.
  • E.2 LIMITATION OF EVALUATION PARADIGM: LLM-judge evaluation is nondeterministic and dependent on judge capabilities, with possible subjectivity in semantic equivalence and formatting strictness.The paper identifies standardized objective metrics for reasoning models as an open challenge.

F DETAILED TRAINING STAGE CONFIGURATIONS

Bee-8B is trained through a five-stage curriculum that progresses from visual-language alignment to instruction tuning, refinement, and reinforcement learning. The recipe uses Honey-Data-15M for advanced reasoning, a balanced 1M subset for efficient refinement, and GRPO to improve output reliability.

  • Stage 1: MLP warmup: Approximately one million image-caption pairs train only the MLP projector initially, combining about 560K LLaVA-OneVision and 440K COYO samples while backbones remain frozen.
  • Stage 2: vision-language alignment: The second stage unfreezes all components and combines approximately 12.6M vision-language pairs with 1.43M text-only samples to build multimodal capabilities while preserving language skills.
  • Stage 3: large-scale SFT: Full-dataset SFT on Honey-Data-15M trains advanced instruction following and visual reasoning, including exposure to rarer long-CoT samples across one complete epoch.
  • Stage 4: refinement SFT: Honey-Data-1M is selected with topic quotas, source-quality scores, and an approximate 1:1 long- versus short-chain ratio for targeted and resource-efficient refinement.
  • Stage 5: reinforcement learning: GRPO uses candidate responses from MMK12 and ViRL39K with rule-based format and accuracy rewards to address repetition, incomplete responses, and improper formatting.The reward weights are 0.2 for format and 0.8 for accuracy.

G DATA LICENSING AND GOVERNANCE

Honey-Data-15M uses a layered licensing and governance framework because its constituent datasets retain their original terms. The project also provides notice-and-takedown procedures and documents licensing information while leaving unverifiable entries blank.

  • Licensing structure: Each Honey-Data-15M sub-dataset remains governed by its original license, while project-owned modified prompts, formatting, and generated CoT responses use CC-BY-NC-4.0.
  • Usage obligations: The project requires users to follow source-specific restrictions, including attribution, non-commercial use, permission notices, and share-alike obligations where applicable.
  • Data provenance and privacy: For OCRed textbooks and K-12 content, the project uses materials from public academic datasets rather than independently digitizing physical sources.
  • Governance: Potential copyright or PII concerns are handled through a dedicated notice-and-takedown channel with prompt review and removal of contested content after valid notices.
  • License documentation: The licensing tables enumerate source-specific terms and restrictions across dataset categories, with unverifiable licensing information left blank for accuracy.

H.3 BENCHMARKS

The benchmark suite evaluates Bee-8B across general VQA, table/chart/OCR, and math/reasoning tasks using curated filtering examples and explicit judging procedures. The included cases illustrate how irrelevance, insufficient information, and answer inconsistency are identified.

  • Benchmark organization: Evaluation covers three benchmark groups: General VQA, Table & Chart & OCR, and Math & Reasoning.
  • Noise and irrelevance filtering: The filtering criteria remove image-question pairs that are irrelevant, ambiguous, linguistically invalid, or otherwise contain obvious issues.
  • Fidelity verification: Answers to precise questions are retained when numerically equivalent to the original and filtered when they introduce fundamental numerical conflicts.
  • Representative cases: The evaluation examples include filtering low-quality images, repeated responses, irrelevant image-instruction pairs, and responses that cannot be validated from incomplete visual information.
  • Unanswerable instructions: Pairs are also filtered when questions are unanswerable because images are partially corrupted or lack information required to answer them.

K.2 SHOWCASE OF HIGH-QUALITY ENRICHED DATA

The showcase presents Honey-Data-15M responses across visual understanding, reasoning, and interpretation tasks, with enriched answers adding detailed reasoning and explanations.

  • Long Chain-of-Thought responses provide more detailed reasoning, broader knowledge, and more accurate answers across diverse topics.
  • Visual descriptions combine concise scene identification with detailed accounts of environmental features, text, and contextual interpretation.
  • Responses about people extend basic activity recognition into interpretations of crowd gathering, possible events, traffic management, and decorations.
  • Reasoning examples compare visual properties explicitly, such as matching matte finishes when determining whether objects share material.
  • Mathematical examples show stepwise reasoning from geometric properties and arithmetic to answers including CD = 4 and the smallest summed accuracy for artery.
  • The corpus also includes explanations of ambiguous visual illusions and category-adjustment guidance linking data distributions to application goals and robustness.
Loading 2510.13795v4…