PaperlayerView exact revision on arXiv

Source-linked AI summary

CCQ: A Multi-State Child Care Quality Dataset to Support AI for Children's Health Research

Victor Li, Yuzhang Xie, Ziwei Dong, Qingyang Zhu, Wenjing Ma, Carl Yang, Jinbing Bai, Huiwen Xu, Jiaying Lu

arXiv:2610.05863v1cs.LGcs.AI

TL;DR

Fragmented child care quality data and heterogeneous state rating systems constrain research and prediction across providers. CCQ assembles and curates provider records from twelve states, then benchmarks models for quality rating prediction. Tabular classifiers perform best within states, while cross-state zero-shot transfer is weak and modest target supervision recovers much of the gain from full supervision.

  • Problem

    Fragmented, heterogeneous state rating systems and limited provider-level datasets constrain research and models intended to generalize across states.

    Source: §1 · p. 2

  • Method

    CCQ curates provider-level records from twelve states into textual and numerical releases and benchmarks tabular and language models on within-state and cross-state rating prediction.

    Source: §1 · p. 2, §6 · p. 24

  • Results

    Tabular classifiers on preprocessed data perform best within states, while zero-shot cross-state transfer is near chance and labeling 16% of target providers yields 63% to 83% of full-supervision gains for eight of ten models.

    Source: §5.2.3 · p. 16, §5.4.3 · p. 22, §6 · p. 24

  • Takeaways & Limitations

    Model predictions should support rather than replace on-site assessment because agreement with true ratings is substantial in only a few states and fair in others.

    Source: §6 · p. 24

    Curation agents were evaluated only on Georgia, cleaning scripts for other states were not verified line by line, and benchmark results are reference baselines from a single uniform configuration and random seed.

    Source: §6.1 · p. 24

Abstract

from arXiv · show

High-quality child care in early life is a critical determinant of children's growth and development. Research on child care quality has been constrained by fragmented, non-research-friendly, and privacy-bound datasets. We present CCQ (Child Care Quality), a large-scale, de-identified dataset for applied data science research at the intersection of AI and early childhood health. CCQ integrates 59,372 child care provider records across 12 U.S. states, covering diverse provider types as well as data schemas. To ensure research utility while protecting privacy, we implement an automated, LLM-based curation pipeline that anonymizes, cleans, and standardizes raw state records into two complementary releases: a cleaned textual release and a fully preprocessed tabular release. We also benchmark traditional machine learning models, tabular foundation models, and language models on quality rating prediction and important features analytics. Within a state, tabular classifiers on the preprocessed tables perform best. Across states, zero-shot transfer is near chance, but modest target-state supervision recovers most of the within-state performance, and pretraining on other states benefits finetuned language models. We release both datasets with all code to accelerate AI-driven research on child care quality and ultimately improve children's health and development.

1 Introduction

Child care is a widespread early-life environment associated with children’s health and development, but fragmented and unevenly rated state systems limit research-ready data and prediction across systems. CCQ addresses this gap with a multi-state dataset and benchmarks models for within- and cross-state quality rating prediction.

1 Introduction

  • Child care occupies a substantial share of young children’s time, and care quality is associated with cognitive, language, social, and later academic outcomes.The cited evidence includes children spending an average of 36 hours weekly in routine care and associations with outcomes through adolescence.Source: §1 · pp. 1–2
  • Single-state prediction studies use small samples, while heterogeneous QRIS schemas and costly, domain-specific curation impede a multi-state provider dataset and models that generalize across rating systems.Providers’ quality-relevant features span physical environments, food, policies, activities, and staffing.Source: §1 · p. 2
  • CCQ combines provider-level quality ratings and features from twelve U.S. states in cleaned textual and fully preprocessed numerical releases.The releases are hand-checked for private information and data leakage, with values spot-checked against source portals.Source: §1 · pp. 2–3
  • The benchmark compares classical, tabular, and language models on within-state and cross-state rating prediction, with feature attribution and transfer across differing amounts of target-state labels.Within-state evaluation uses five-fold cross-validation; cross-state evaluation tests transfer to a held-out state.Source: §1 · p. 3, Figure 1 · p. 3
  • CCQ’s LLM-based curation pipeline collects, anonymizes, and cleans state records, while human checks address privacy, leakage, and output values.The contributions include benchmarking ten representative models and examining how transfer changes with more target-state labels.Source: §1 · p. 3

2 Related Work

Prior child care quality datasets either lack provider-level ratings, record few quality-shaping features, or cover only a single system. Prediction studies have likewise used limited single-program samples and have not examined transfer across rating systems.

2.1 Child Care Quality Datasets

  • Existing public datasets variously lack quality-shaping features, provider-level ratings, or multi-state provider coverage; CCQ assembles provider records and ratings across twelve U.S. states.The cited examples include Australian and Indonesian rating data with few features, Alberta and Head Start records without QR ratings, and a system-level QRIS catalog.Source: §2.1 · p. 4

2.2 Child Care Quality Prediction

  • Earlier prediction studies mainly used regression or classical classifiers on small samples from one program, limiting evidence about performance on large, heterogeneous data.The studies described include analyses of Australian, Head Start, Indonesian, and researcher-collected Canadian samples.Source: §2.2 · p. 4
  • Prior studies did not test transfer across rating systems; this benchmark evaluates broader model families on larger data and explicitly studies heterogeneous cross-state transfer.The cross-state setting includes differing schemas across states.Source: §2.2 · p. 4

3 Dataset Creation

CCQ compiles public provider records from twelve state QRIS into two releases through a collection, anonymization, cleaning, and verification pipeline. Its data vary substantially in coverage, rating distributions, feature schemas, and missingness.

3.1 Data Sources

  • CCQ integrates publicly available child-care data across 12 U.S. states, assembling 59,372 records released by state agencies or designated partners; 29,073 providers (49.0%) possess an assigned QR score.Source: §3.1 · pp. 4–5

3.2 Construction Pipeline

  • Georgia serves as the hand-implemented reference for a four-step pipeline—collection, anonymization, cleaning, and verification—adapted for the other eleven states.The reference was chosen because Georgia has the most published fields among the twelve states.Source: §3.2 · p. 5, Figure 2 · p. 6

3.2.1 Collection.

  • Providers are collected into state-specific tables, with a coding agent adapting Georgia’s collection implementation and each state’s collected columns checked by hand.The tables record provider attributes, including an identifier and quality rating.Source: §3.2.1 · p. 5

3.2.2 Cleaning.

  • The raw release preserves textual source features, while the preprocessed release is numerical; both remove private and rating-leakage columns.The curation agents were tested on twenty Georgia tasks, and the adopted code-writing agent receives Georgia’s reference code as an in-context example.Source: §3.2.2 · pp. 5–6, Figure 2 · p. 6

3.2.3 Verification.

  • Across ten reviewed states, two annotators checked four critical fields in a stratified sample of rated providers, excluding state-field pairs where a field was absent.The review included 20 providers per state and 740 checked values per annotator.Source: §3.2.3 · pp. 7–8
  • 97.2% of 1,480 checked released values were judged correct by two annotators, who agreed on 730 of 740 values.The spot-check covered four critical fields in 200 providers across ten states; mismatches were attributed to records changing between collection and verification, though their causes were not individually determined.Source: §3.2.3 · p. 8, Table 3 · p. 8

3.3 Dataset Overview

  • Rating coverage and rating distributions vary widely by state, and feature missingness ranges from nearly complete fields to median fields that are almost entirely empty.The rated share ranges from 7.9% in California to 99.9% in Oklahoma; median-field missingness reaches 99.02% in Oklahoma.Source: §3.3 · pp. 8–9, Figure 3 · p. 9

3.4 Dataset Release

  • The dataset is available as a gated release for non-commercial academic research, with access by request and a data use agreement.The curation and benchmark-reproduction code is also available.Source: §3.4 · pp. 9–10

4 Benchmark Design

The benchmark predicts provider quality ratings under within-state and cross-state deployment settings. It evaluates ordinary supervised prediction within a state and transfer to a target state with differing schemas, rating scales, or distributions.

4 Benchmark Design

  • The benchmark predicts a provider’s state-assigned quality rating in within-state and cross-state settings that mirror agency deployment.Within-state models use a state’s rated providers; cross-state models train on other states whose schemas and rating systems differ.Source: §4 · p. 10

4.1 Within-State QR Prediction

  • Within-state prediction is ordinal classification with training and test data drawn from the same state distribution and sharing its feature schema and label space.Evaluation uses stratified five-fold cross-validation, quadratic weighted kappa, and balanced accuracy to account for ordinal ratings and label imbalance.Source: §4.1 · p. 10

4.2 Cross-State QR Prediction

  • Cross-state prediction must handle schema mismatch, label-space mismatch, and potential distribution shift while learning from source states and some labeled target data.The leave-one-state-out protocol holds a stratified 20% target test set fixed and varies target adaptation data from 0% to 80%, using all source records at every level.Source: §4.2 · pp. 10–11

5.1 Experimental Setup

The experiments compare tabular and textual classifiers on within-state and cross-state quality-rating prediction. The setup defines two data representations, model-training procedures, and metrics for evaluating predictions across states.

5.1.1 Compared Models.

  • The benchmark compares six tabular classifiers with five textual classifiers, including random forests, XGBoost, TabPFN, ModernBERT, and Qwen3-4B variants.Tabular models use the preprocessed representation or embedded raw records; textual models process the raw representation directly.Source: §5.1.1 · p. 11
  • For cross-state prediction, six target-supervision levels are tested; some models fit once on pooled sources and target data, while others pretrain on sources before target fine-tuning.The tested levels are p ∈ {0, 16, 32, 48, 64, 80}.Source: §5.1.1 · p. 11

5.1.2 Implementation Details.

  • All models use a uniform configuration across states and experiments, with class weights or subsampling to address class imbalance.Hyperparameters are tuned only when needed for convergence or consistency; otherwise, library defaults are used.Source: §5.1.2 · p. 11, Table 4 · p. 12

5.1.3 Evaluation Metrics.

  • Quadratic weighted kappa is the primary metric because it penalizes ordinal-rating errors by squared distance; balanced accuracy and Elo scores are also reported.Within-state results average five folds, while cross-state results are point estimates on held-out target test sets.Source: §5.1.3 · pp. 12–13

5.1.2 Implementation Details.

  • Raw provider records are textualized for language models and embedded in 254-token chunks into 384-dimensional inputs for tabular models.The textual representation omits false and blank values, provider identifiers, and ratings; the tabular classifiers also use the preprocessed representation.Source: §5.1.2 · pp. 11–12, Figure 4 · p. 13

5.2 Within-State QR Prediction Experiment

In within-state prediction, tabular classifiers using preprocessed data lead overall, though performance varies across states and metrics. The section also compares model performance with parameter efficiency and qualifies recommendations by state-level agreement and the benchmark’s uniform configuration.

5.2.1 Predictive Performance.

  • Preprocessed tabular classifiers achieve the best QWK in 9 of 12 states and the best balanced accuracy in all 12.The two metrics do not always agree: TabPFN-num’s mean QWK is 38.75, below XGB-num’s 46.34.Source: Table 5 · p. 14, §5.2.1 · p. 14
  • The best state-level QWK spans 54 points, and the leading model varies by state, with four states below 35.The strongest models exceed 70 in OK and MD, while the best scores are 33.63 in WA, 31.67 in MT, 29.99 in NE, and 23.33 in GA.Source: §5.2.1 · p. 14
  • XGB-num leads within-state Elo at 1519.1, followed by TabPFN-num at 1422.2; preprocessed tabular classifiers hold the four highest overall ranks.Every tabular classifier scored higher on preprocessed than raw data, with a 303.5-point average Elo advantage across five model families.Source: §5.2.1 · p. 13, Figure 5 · p. 15

5.2.2 Predictive Performance Versus Model Complexity.

  • TabPFN-num reaches 1422.2 Elo with zero tuned parameters; XGB-num gains 96.8 Elo points with about 5.0 × 10^4 tuned parameters.On raw data, the frontier starts with TabPFN-txt, followed by LR-txt, LLM-LoRA, and MBERT as parameter counts increase.Source: §5.2.2 · p. 14, Figure 6 · p. 15

5.2.3 Discussion.

  • For best within-state prediction, the authors recommend preprocessed tabular models; TabPFN-num is a no-tuned-parameter option, while XGB-num leads Elo and mean QWK.These recommendations rank model families under one uniform benchmark configuration, rather than state-specific tuning.Source: §5.2.3 · p. 16
  • In WA, MT, NE, and GA, even the best model’s QWK remains below 35, limiting practical use for high-stakes decisions in those states.The paper concludes that current models are not yet reliable surrogate rating predictors in every state.Source: §5.2.3 · p. 16

5.3 Driving Factor Analysis

The analysis uses SHAP rankings with a shadow-feature noise floor to identify model-attributed features across states. The results show broad positive and negative patterns, but also state-specific signals and inconclusive attributions for MT.

5.3.1 Analysis Setup.

  • SHAP ranks each feature by mean absolute contribution across providers, while cardinality-matched shuffled shadow features set the noise floor for interpreting real contributions.Only features above the largest mean absolute SHAP value among shadow features are interpreted.Source: §5.3.1 · pp. 16–17

5.3.2 Analysis Results.

  • XGB-reg scores higher than XGB-num in 9 of 12 states and by 0.83 QWK points on average; differences range from −5.83 (MT) to +3.86 (NE).Only MT shows an absolute difference above 5 points.Source: Table 6 · p. 17, §5.3.2 · p. 17
  • 58 of 983 features clear the noise floor overall, ranging from 0 in MT to 9 in KY; among these, 37 push predicted ratings up and 21 push them down.Three have near-zero direction correlations and are better read as undirected; the 58 features span six groups.Source: §5.3.2 · p. 17
  • Institutional formalization is the clearest positive pattern: center-based programs, licensed capacity, and indicators of subsidy or public funding are positive in multiple states.Signals vary by state: operating hours lower predicted ratings in CA, CO, and GA but raise them in OK; MT’s attributions are inconclusive.Source: §5.3.2 · p. 18

5.3.3 Discussion.

  • For MT, attributions are inconclusive, so the authors draw no conclusions: its fidelity gap is −5.83 QWK points, and without class weights its regional indicator clears the noise floor in 29 of 30 seeded runs.MT has only 181 rated providers and 20 binary features in its preprocessed representation.Source: §5.3.3 · p. 18
  • In other states, larger, center-based, publicly funded, and accredited providers tend to receive higher predicted ratings, while violations, enforcement actions, and some less formal arrangements tend to lower them.The authors characterize this as a state-specific tendency, not a rule.Source: §5.3.3 · p. 18
  • Capacity and operational scale are nuanced: larger licensed capacity is generally positive, but weekend care and long hours are negative contributors in some states.Geographic identifiers also exceed the noise floor and may reflect regional factors the dataset only partially captures.Source: §5.3.3 · p. 18

5.4 Cross-State QR Prediction Experiment

Cross-state rating prediction is weak without target labels, but modest target-state supervision makes finetuned language models strong performers. The results also show that transfer benefits depend on training regime and that reconciling rating scales does not ensure successful transfer.

5.4 Cross-State QR Prediction Experiment

  • Cross-state models use the raw representation, are evaluated on each target state’s native scale, and are trained either once on pooled data or sequentially with source pretraining and target finetuning.At p=0, sequential models have no target data for finetuning; Georgia’s five-level source labels are collapsed to three levels when Georgia is the target.Source: §5.4 · p. 20

5.4.1 Predictive Performance.

  • MBERT leads cross-state Elo at 1374.4, ahead of LLM-LoRA at 1324.9 and MBERT-CORN at 1298.6.The 49.5-point gap between MBERT and LLM-LoRA corresponds to a 57.1% expected win rate.Source: §5.4.1 · p. 20
  • At p=0, the best mean QWK is 9.96 (LLM-LoRA), and most models perform near chance in 11 of 12 targets.MD is the exception; the results table reports leave-one-state-out scores on a stratified 20% test set.Source: §5.4.1 · p. 20, Table 7 · p. 21
  • With target supervision, finetuned language models hold the best QWK in at least 10 of 12 targets; MBERT-CORN wins 28 of 60 supervised target-level combinations.At p=80, MBERT-CORN has mean QWK 46.11, versus 45.32 for MBERT and 44.13 for LLM-LoRA.Source: §5.4.1 · p. 20, Table 7 · p. 21

5.4.2 Effect of Target Supervision.

  • The largest mean-QWK gain for every model occurs from p=0 to p=16; this first step accounts for 80% of MBERT-CORN’s total gain and 83% of LLM-LoRA’s.The mean test QWK is plotted across the twelve targets for all ten models.Source: §5.4.2 · p. 20, Figure 8 · p. 22
  • Beyond p=16, eight models’ curves flatten, with additional QWK gains ranging from 5.71 for LLM-LoRA to 9.53 for XGB-txt.LLM-RAG and TabNN-txt are exceptions to the general flattening pattern described for the other models.Source: §5.4.2 · p. 22

5.4.3 Discussion.

  • Zero-shot transfer is generally inadequate, 16% target labels recover most gains, and finetuned language models lead; Georgia remains difficult under scale reconciliation.The authors caution that a state without rated providers should not expect a usable predictor from other states’ data alone.Source: §5.4.3 · pp. 22–23

5.5 Transfer Utility Analysis

The analysis compares models trained on a state’s own data with the same models pretrained on pooled source states and adapted to that state. Pretraining benefits several finetuned language models, but its effect varies by training regime.

5.5 Transfer Utility Analysis

  • The comparison uses the same target-state training pool at 80% in both settings, with the pretrained model additionally trained on pooled source states before adaptation.It reports mean test QWK across twelve states and includes both one-shot-fit and sequentially trained models.Source: §5.5 · p. 23

5.5.1 Analysis Results.

  • Pretraining raises mean QWK for MBERT-CORN by 7.56, LLM-LoRA by 4.61, and MBERT by 4.18, while the four one-shot-fit models all decline.After pretraining, MBERT-CORN, MBERT, and LLM-LoRA have the three highest mean QWK values.Source: §5.5.1 · p. 23, §5.5.2 · p. 23

5.5.2 Discussion.

  • Four of six sequentially trained models gain from pretraining, but LLM-CLS loses 4.65 QWK and TabNN-txt loses 2.45.The authors hypothesize that RAG’s per-class exemplars may transfer better than the fixed class boundaries learned by LLM-CLS.Source: §5.5.1 · p. 23, §5.5.2 · p. 23
  • The authors recommend sequential pretraining followed by adaptation with a finetuned language model such as MBERT, MBERT-CORN, or LLM-LoRA.The authors report that these three models gain more than four mean-QWK points from pretraining, even with the target’s full training pool.Source: §5.5.2 · pp. 23–24

6 Conclusion

The paper presents CCQ and benchmarks quality-rating prediction within and across states. It cautions that benchmark results are reference baselines and that predictions should support, not replace, on-site assessment.

6 Conclusion

  • CCQ contains 59,372 de-identified child care providers across 12 states and offers preprocessed and raw releases for tabular and language models.The authors also release the LLM-based curation pipeline used to produce both releases.Source: §6 · p. 24

6.1 Limitations

  • Benchmark results are reference baselines: curation agents were evaluated only on Georgia, methods used one configuration and random seed, and cross-state results use fixed test sets.The authors also note that feature attributions reflect model reliance, not causal effects on quality.Source: §6.1 · p. 24

6.2 Ethical Considerations

  • The dataset contains aggregated facility-level information, removes or anonymizes directly identifying features, and users are asked not to re-identify providers or use predictions for individual licensing, funding, or enforcement decisions.Records reflect state portals at collection time, so ratings may have changed since.Source: §6.2 · p. 24

Related papers

Loading 2610.05863v1…