PaperlayerView exact revision on arXiv

Source-linked AI summary

OpenDataArena: A Fair and Open Arena for Benchmarking Post-Training Dataset Value

Mengzhang Cai, Xin Gao, Yu Li, Honglin Lin, Zheng Liu, Zhuoshi Pan, Qizhi Pei, Xiaoran Shang, Mengyuan Sun, Zinan Tang, Xiaoyang Wang, Zhanping Zhong, Yun Zhu, Dahua Lin, Conghui He, Lijun Wu

arXiv:2512.14051v1cs.AI

TL;DR

Post-training datasets lack fair, systematic evaluation of quality and downstream impact, making results difficult to reproduce. ODA combines standardized dataset evaluation with multidimensional scoring; its benchmarking reveals domain-specific performance variations and model-specific preferences.

  • Problem

    Post-training datasets lack a fair, systematic methodology for evaluating quality and downstream impact, making results difficult to reproduce.

    Source: §1 · p. 2, §2.1 · p. 3

  • Method

    ODA uses a standardized pipeline to fine-tune a common pretrained model exclusively on each dataset, evaluate it on downstream benchmarks, and profile data quality across multiple axes.

    Source: §1 · p. 2, §2.3 · p. 6

  • Results

    120 training datasets were evaluated, revealing domain-specific performance variations and model-specific preferences.

    Source: §1 · p. 3

  • Takeaways & Limitations

    ODA shifts research focus from model-centric tuning toward a deeper understanding of data through quantitative and qualitative dataset rankings.

    Source: §2.1 · p. 4

    ODA treats a dataset’s value as demonstrated solely by its ability to concretely improve a model’s capabilities.

    Source: §2.1 · p. 3

Abstract

from arXiv · show

The rapid evolution of Large Language Models (LLMs) is predicated on the quality and diversity of post-training datasets. However, a critical dichotomy persists: while models are rigorously benchmarked, the data fueling them remains a black box--characterized by opaque composition, uncertain provenance, and a lack of systematic evaluation. This opacity hinders reproducibility and obscures the causal link between data characteristics and model behaviors. To bridge this gap, we introduce OpenDataArena (ODA), a holistic and open platform designed to benchmark the intrinsic value of post-training data. ODA establishes a comprehensive ecosystem comprising four key pillars: (i) a unified training-evaluation pipeline that ensures fair, open comparisons across diverse models (e.g., Llama, Qwen) and domains; (ii) a multi-dimensional scoring framework that profiles data quality along tens of distinct axes; (iii) an interactive data lineage explorer to visualize dataset genealogy and dissect component sources; and (iv) a fully open-source toolkit for training, evaluation, and scoring to foster data research. Extensive experiments on ODA--covering over 120 training datasets across multiple domains on 22 benchmarks, validated by more than 600 training runs and 40 million processed data points--reveal non-trivial insights. Our analysis uncovers the inherent trade-offs between data complexity and task performance, identifies redundancy in popular benchmarks through lineage tracing, and maps the genealogical relationships across datasets. We release all results, tools, and configurations to democratize access to high-quality data evaluation. Rather than merely expanding a leaderboard, ODA envisions a shift from trial-and-error data curation to a principled science of Data-Centric AI, paving the way for rigorous studies on data mixing laws and the strategic composition of foundation models.

1 Introduction

Post-training data quality matters to model behavior, yet dataset provenance and quality lack transparent, fair, systematic evaluation. OpenDataArena combines standardized comparisons with multidimensional scoring, lineage tracing, and open tools, then uses large-scale evaluation to examine how dataset value varies.

1 Introduction

  • Post-training datasets lack a standardized, fair evaluation method, leaving what makes a dataset good unresolved and hindering reproducibility and resource allocation.Dataset composition and provenance are often opaque, while dataset creation and selection are frequently ad hoc.Source: §1 · p. 2
  • ODA’s unified training-evaluation pipeline enables apples-to-apples dataset comparisons across models and downstream benchmarks.The study holds model configurations and other process choices constant so the dataset is the sole variable under investigation.Source: §1 · p. 2
  • ODA profiles each dataset across multiple quality axes, including instruction complexity, response quality, and diversity, rather than relying on one performance metric.The platform’s scoring framework covers over 15 intrinsic properties of each dataset.Source: §1 · p. 2
  • Its interactive lineage explorer visualizes dataset genealogy and lets users trace provenance and dissect the sources of aggregated datasets.The platform maps relationships among datasets and their influence flows.Source: §1 · pp. 2–3
  • ODA releases open-source tools, configurations, and results to support reproducibility and further research.The platform is intended to help researchers select datasets, identify high-value seed data, and study links between data characteristics and model performance.Source: §1 · pp. 2–3
  • Across 120 training datasets, 600+ training runs, and 22 benchmarks, ODA finds domain-specific performance variation and model-specific dataset preferences.It also analyzes correlations between fine-grained quality metrics and downstream results, and identifies high-yield datasets.Source: §1 · p. 3

2 The OpenDataArena Platform

OpenDataArena is an ecosystem for systematic, transparent evaluation of post-training datasets, combining standardized benchmarking with multidimensional diagnostic scoring. Its design spans data handling, analysis, visualization, and reproducible tools.

2.1 Design Principle

  • ODA treats dataset value as demonstrated by model capability gains, enforcing identical model, hyperparameter, and evaluation conditions while supporting openness, diverse domains, and modular extension.Source: §2.1 · pp. 3–4

2.2 Platform Overview

  • ODA’s workflow moves datasets through input, evaluation, analysis, and visualization, delivering leaderboards, over 15-property scores, lineage tracing, and a reproducible open-source toolkit.Source: §2.2 · p. 5

2.3 Benchmarking Pipeline

  • For each post-training dataset, ODA fine-tunes a common pretrained model exclusively on that dataset and uses downstream benchmark performance as a proxy for its value.Source: §2.3 · p. 6

2.3.1 Dataset and Benchmark Selection

  • ODA selects over 120 public SFT datasets using impact, recency, size, safety, and format criteria, then evaluates them across 22+ benchmarks spanning general, math, code, and reasoning tasks.Source: §2.3.1 · p. 6

2.3.2 Training and Evaluation Setup

  • Across Llama3.1-8B, Qwen2.5-7B, and Qwen3-8B, fixed training settings and benchmark-specific evaluation supported over 600 training runs and 10,000 evaluation runs.Source: §2.3.2 · pp. 6–7

2.4 Data Scoring

  • ODA profiles instruction-only Q and full Q&A across tens of axes, using model-based evaluation, LLM judges, and heuristics to assess complexity, response quality, and text properties.Source: §2.4 · pp. 7–8

3 Dataset Landscape and Analysis

ODA characterizes the post-training dataset landscape and traces how datasets are derived, reused, and connected. Its analysis finds a highly connected ecosystem, domain-specific lineage patterns, and benchmark contamination that can propagate through dataset reuse.

3.1 Macro-Analysis of the Data Landscape

  • Dataset creation accelerated from early 2024, reaching 18–19 releases per quarter and more than 40 million samples, while domain shares shifted toward Math, Code, and later Science.Math peaked at 46.2% in 2024Q2, and Science rose from 6.2% to 21.6% by mid-2025.Source: §3.1 · p. 8, Figure 4 · p. 9, Figure 5 · p. 9

3.2 Data Lineage: Uncovering Relationships and Redundancy

  • ODA treats lineage as a directed graph of datasets and derivation relations, providing a basis for provenance tracking, redundancy detection, and contamination analysis.The framework maps pathways from datasets to upstream sources through transformation operations.Source: §3.2 · p. 10, §3.2.1 · p. 10

3.2.1 Automated Data Lineage Framework Design

  • To trace lineage despite scattered, inconsistent documentation and nested dependencies, the framework retrieves evidence from multiple sources, consolidates relations, and flags low-confidence edges for expert review.Its DFS-style process builds lineage graphs incrementally, with cross-source checks used to discard unsupported or contradictory relations.Source: §3.2.1 · pp. 10–11

3.2.2 Landscape-Level Redundancy Analysis

  • Across four domains, 70 seed datasets expand into a global lineage graph of 411 unique nodes and 941 edges (≈2.29 edges per node); the eight highest-degree datasets form an extensive relational structure.This small core is repeatedly reused, fused, and reformulated, leading to strong systemic homogenization.Source: §3.2.2 · pp. 11–12
  • Lineage patterns differ by domain: Math has the deepest average lineage at 5.18, Code centers on contest resources, General frequently reuses math data, and Science relies on cross-domain sources.Science has the shallowest average depth at 3.71, reflecting scarcity of specialized native base datasets.Source: §3.2.2 · pp. 12, 14
  • AM-Thinking-v1-Distilled traces to 435 source data nodes and ranks first in Math and third in Code on Qwen2.5-7B.The paper reports that it also ranks first in Math and second in Code on Llama3.1-8B.Source: §3.2.2 · p. 14
  • Lineage tracing identifies training datasets that directly incorporate evaluation benchmarks, and contaminated seeds can pass leakage recursively to downstream datasets.Examples include Omni-MATH in Big-Math-RL-Verified and LiveCodeBench-v5 in DeepCoder-Preview-Dataset.Source: Figure 8 · p. 13, §3.2.2 · p. 14

4 Findings from Benchmarking

ODA benchmarks post-training datasets across models and domains, finding substantial variation in how datasets perform and transfer. Results show that data scale, response quality, and domain-specific characteristics all matter, while dataset rankings and quality trends differ across settings.

4.1.1 Absolute Performance

  • Qwen3 has the highest median scores across domains, while Math and Code show wider performance variation than General and are more sensitive to dataset quality.The evaluation compares distributions across Llama3.1-8B, Qwen2.5-7B, and Qwen3-8B.Source: §4.1.1 · p. 15, Figure 9 · p. 16

4.1.2 Relative Performance Delta

  • Math and Code show wider performance-delta ranges than General, and Qwen3 achieves both high absolute scores and high positive deltas.Llama3.1 can show larger Math gains than Qwen2.5 while still finishing at a lower absolute score.Source: §4.1.2 · pp. 15–16, Figure 9 · p. 16

4.1.3 Temporal Evolution of Dataset Quality

  • Math dataset scores rose from around 35 in 2023 to approximately 56 by 2025Q3, while Code remained volatile and General stayed consistently high.The paper attributes the gradual global improvement largely to specialized-domain progress, especially Math.Source: §4.1.3 · pp. 16–17, Figure 10 · p. 17

4.1.4 Consistency Across Model Generations

  • Dataset rankings are highly consistent between Qwen2.5 and Qwen3 in Math (0.902), but negatively correlated in General (-0.323) and weakly positively correlated in Code and Science (0.28–0.35).The authors interpret these differences as indicating that dataset value varies with domain and model generation.Source: §4.1.4 · pp. 17–18, Table 1 · p. 17

4.2.1 Efficiency Landscape Across Domains

  • Data Efficiency measures performance gain relative to dataset size; Qwen datasets generally occupy higher performance levels than Llama3.1 datasets in General, Math, and Code, while Science results are scattered and model-dependent.The comparison plots efficiency against final performance and includes only datasets with positive efficiency scores on a log-scale x-axis.Source: §4.2 · p. 18, §4.2.1 · p. 18, Figure 11 · p. 19

4.2.2 The Impact of Data Scale on Performance

  • Extreme-efficiency datasets LIMA and LIMO often fall short of the final performance of larger, high-quality datasets, and LIMO degrades performance for Llama3.1 in Math.The authors favor moderate efficiency with sufficient data volume over extreme efficiency at the cost of robustness.Source: §4.2.2 · p. 18

4.3.1 The Primacy of Response Length and Quality

  • Response Length correlates positively with performance, including 0.81 in Math and 0.40 globally, whereas instruction-only complexity is a weak predictor and instruction-response quality metrics correlate positively.The paper reports that longer average responses align with higher performance, particularly in reasoning-intensive domains.Source: §4.3.1 · p. 20, §4.3.2 · p. 20, Figure 12 · p. 21

4.3.3 Domain Divergence: The Unique Case of Code

  • Code reverses several other domains’ correlation patterns: Response Length correlates at -0.29 in Code versus 0.81 in Math, supporting the need for code-specific evaluation criteria.Q Thinking Prob is positively correlated in Code (0.54) and negatively correlated in Math (-0.69).Source: §4.3.3 · pp. 20–21, Figure 12 · p. 21

5 Looking Forward

Because generative AI is evolving rapidly, ODA outlines future directions to keep the platform relevant and support further innovation. These plans span new data modalities, evaluation tracks, valuation methods, scoring dimensions, domains, and shared standards.

Embracing multimodality

  • ODA plans to extend its pipeline beyond text to multimodal datasets, including image-text, for MLLM evaluation.Source: §5 · p. 21

Benchmarking alignment data

  • Dedicated evaluation tracks are planned for RLHF and DPO-style preference datasets to assess alignment with human values and normative expectations.Source: §5 · p. 21

Developing efficient evaluation

  • ODA will investigate training-free or training-light valuation methods, including influence-function estimators and core-set selection, to approximate utility without full-scale fine-tuning.Source: §5 · p. 21

Enhancing data scorers

  • The scoring framework is planned to add measures of long-context dependency, cultural and linguistic diversity, and resilience to adversarial or perturbed prompts.Source: §5 · p. 22

Expanding to vertical domains

  • Benchmark coverage is intended to expand to finance, law, and medicine, where data scarcity, expertise requirements, and quality control pose distinctive challenges.Source: §5 · p. 22

Building a shared standard

  • ODA seeks community and committee collaboration on standardized metrics and invites contributions of datasets, benchmarks, and methods.Source: §5 · p. 22

6 Conclusion

ODA combines standardized benchmarking and multidimensional scoring to evaluate more than 120 heterogeneous datasets, turning data-quality questions into empirical comparisons. Its analyses report findings about data efficiency, response quality, and dataset lineage, while envisioning broader use as data-centric AI infrastructure.

6 Conclusion

  • Carefully curated, information-dense datasets can outperform substantially larger, less structured collections, so data efficiency is not a simple function of scale.Source: §6 · p. 22
  • Rich reasoning traces are more predictive of downstream performance than prompt complexity alone.Source: §6 · p. 22
  • ODA combines a unified training–evaluation pipeline and multidimensional scoring to enable fair, open comparisons across more than 120 heterogeneous datasets.Source: §6 · p. 22
  • Lineage analysis documents a shift from broad instruction tuning toward specialized domains such as mathematics and code.Source: §6 · p. 22
  • ODA envisions extending its infrastructure to multimodal corpora, RLHF preference datasets, and high-stakes vertical domains, with community-driven progress.Source: §6 · p. 22

Contribution List

The contribution list assigns named authors to leaderboard, toolkit, lineage, scoring, and advisory work. The appendix states that its training and evaluation configurations aim to support reproducibility, fairness, and consistent behavior across data sources.

Contribution List

  • Leaderboard construction is credited to Mengzhang Cai, Xin Gao, Honglin Lin, Zheng Liu, Zhuoshi Pan, Qizhi Pei, Xiaoyang Wang, Zhanping Zhong, and Yun Zhu.Source: Contribution List · p. 22
  • Toolkit development is credited to Mengzhang Cai, Yu Li, and Zhanping Zhong.Source: Contribution List · p. 22
  • Data lineage analysis is credited to Xin Gao, Yu Li, Honglin Lin, and Xiaoran Shang.Source: Contribution List · p. 22
  • The scoring system is credited to Xin Gao, Zhuoshi Pan, Qizhi Pei, Mengyuan Sun, Zinan Tang, Xiaoyang Wang, and Yun Zhu; Conghui He and Dahua Lin are listed as advisors.Source: Contribution List · p. 22

A Detailed Benchmarking Settings

  • The appendix provides full training and evaluation configurations intended to ensure reproducibility, fairness across datasets, and consistent model behavior independent of data source.Source: A Detailed Benchmarking Settings · p. 28

A.1 Training Configurations

The appendix describes a standardized SFT training pipeline and points to a table of model- and scenario-specific hyperparameters.

A.1 Training Configurations

  • SFT experiments use the LLaMA-Factory framework (version 0.9.2).Source: §A.1 · p. 28
  • The standardized pipeline is intended to make downstream performance differences arise from dataset quality rather than optimization or hyperparameter variation.Source: §A.1 · p. 28
  • Table 2 summarizes training hyperparameters for different models and scenarios.Source: Table 2 · p. 28

A.2 Evaluation Configurations

The evaluation setup uses OpenCompass and vLLM, with benchmark-specific inference settings and evaluation configurations documented in Tables 3 and 4.

A.2 Evaluation Configurations

  • Evaluations use OpenCompass v0.4.2 and vLLM for high-throughput, deterministic decoding.Table 3 lists inference settings for Llama3.1, Qwen2.5, and Qwen3; Table 4 lists benchmark configurations.Source: §A.2 · p. 28, Table 3 · p. 28, Table 4 · p. 29
  • Table 4 contains benchmark-specific evaluation configurations, including shot samples and metrics.The supplied text does not specify the individual configurations.Source: §A.2 · p. 28, Table 4 · p. 29
  • Table 3 covers inference settings for Llama3.1, Qwen2.5, and Qwen3 models.The passage identifies the table's scope but does not provide its setting values.Source: Table 3 · p. 28

B Comparative Performance Alignment Rankings of Datasets on Qwen2.5 vs. Qwen3 Models

The section compares dataset domain-wise rankings after fine-tuning Qwen2.5-7B and Qwen3-7B under identical settings. It also presents a quality-performance correlation matrix.

B Comparative Performance Alignment Rankings of Datasets on Qwen2.5 vs. Qwen3 Models

  • Qwen2.5-7B and Qwen3-7B are fine-tuned on the same datasets under identical settings for a cross-model comparison.The comparison uses domain-wise dataset rankings in Table 5.Source: B Comparative Performance Alignment… · p. 29
  • Table 5 presents rankings correlation on Qwen2.5 and Qwen3 models.The supplied passage does not provide correlation values or specific ranking outcomes.Source: Table 5 · p. 29
  • The quality-performance matrix reports stronger predictive power for Response-side QA metrics than Instruction-side Q metrics.It displays Spearman correlations between automated quality metrics and domain performance.Source: Figure 13 · p. 30

C Data Scoring Metric Definitions

OpenDataArena scores datasets with metrics applied either to instructions alone (Q) or to full instruction-response pairs (QA). The definitions span model-based, LLM-judged, and heuristic measures.

C.1 Model-based Evaluation

  • Deita Complexity estimates instruction complexity, while Thinking Probability predicts whether an instruction requires deep, multi-step reasoning.Higher Deita Complexity scores indicate more challenging prompts.Source: §C.1 · p. 31
  • Deita Quality scores response quality, helpfulness, and correctness; IFD estimates difficulty adhering to instruction constraints, and Fail Rate tracks model-flagged response failures.These model-based metrics assess the full instruction-response pair.Source: §C.1 · p. 31

C.2 LLM-as-Judge

  • LLM-as-judge metrics assess instruction difficulty and response relevance, clarity, coherence, and completeness.Clarity, coherence, and completeness are evaluated separately for instructions and responses.Source: §C.2 · p. 31
  • LLM-as-judge also rates complexity, correctness, and meaningfulness for instructions and responses.Correctness assesses response factual accuracy and logical soundness given the instruction.Source: §C.2 · p. 31

C.3 Heuristic

  • The heuristic metric measures response length by token or character count.It is calculated using a direct, rule-based method.Source: §C.3 · pp. 31–32

D Visualized Results

The visualizations show dataset lineage relationships and dataset comparisons across model performance and multidimensional quality metrics.

D Visualized Results

  • The ODA website visualizes dataset relationships through lineage analysis and compares datasets using performance and multidimensional quality metrics.Figure 14 is an interactive graph of dataset families and derivation relationships; Figure 15 compares multiple datasets.Source: D Visualized Results · p. 32, Figure 14 · p. 33, Figure 15 · p. 34
Loading 2512.14051v1…