PaperlayerView exact revision on arXiv

Source-linked AI summary

What fidelity metrics miss: a structural check on synthetic educational data

Hitoshi Inoue, Koichi Yasutake

arXiv:2609.27265v2cs.CY

TL;DR

Variable-level summary-statistic comparisons are common for evaluating synthetic educational data, but reported confirmation rates do not track those comparisons. The paper checks weekly cohort structure using proximity-graph components and finds substantially less term-wide variation in synthetic than real cohorts.

  • Problem

    Variable-level summary statistics do not capture temporal structure, while reported fidelity scores and confirmation rates do not track each other.

    Source: §1 · p. 1

  • Method

    The paper tracks weekly connected-component counts in learner proximity graphs and compares real and synthetic cohort curves at a common working point.

    Source: §1 · p. 2

  • Results

    2.6–4.9 times as much variation appeared in real curves as in synthetic curves across four cohorts, although synthetic versions reproduced the level and shape of the weekly partition.

    Source: §7 · p. 10

  • Takeaways & Limitations

    Synthetic cohorts reproduce what is needed for questions about who differs in a given week, but not how much a cohort reorganises across the term.

    Source: §5 · p. 8

    Evidence comes from one subject at one institution and four cohorts on one platform; the size of the gap is not claimed to generalise.

    Source: §6 · p. 9

Abstract

from arXiv · show

Secondary use of educational records is increasingly mediated by platforms that share a differentially private synthetic version of a dataset and validate specific findings against the real data on request. The synthetic version is evaluated by comparing summary statistics of each variable, yet reported confirmation rates suggest that such comparisons do not predict which findings survive. We propose a structural check: the number of connected components of a weekly proximity graph over learners, tracked across a term. Across four annual cohorts of lower-secondary study-habit logs, the synthetic versions reproduced the level of this quantity and the shape of the weekly partition, but its variation across the term was between 2.6 and 4.9 times smaller than in the real data at a common working point, without exception, and those changes fell in different weeks: the synthetic cohorts single out the term's examination weeks and the real cohorts do not. We also show that a routine rule for setting the graph threshold makes naive comparisons between two datasets invalid, and illustrate this with an error of our own. The real curves are also distinguishable from marginal-preserving surrogates of themselves in all four cohorts, where three of the four synthetic ones are not, a comparison that needs no real data; these differences trace to what the generator was given.

1. Introduction

The paper asks whether synthetic cohorts differ from real cohorts in temporal structure that variable-level fidelity metrics do not compare. It proposes a weekly graph-based structural check and reports substantially less week-to-week variation in synthetic cohorts.

1. Introduction

  • Summary-statistic fidelity scores did not track confirmation rates, and a misrepresented phenomenon could prevent researchers from formulating questions they might have asked of real data.In a pilot of 25 validation requests from four researchers, an average of 36% of findings from synthetic data were confirmed on real data.Source: §1 · p. 1
  • Common fidelity metrics compare variable-level summary statistics, not whether weekly differences or temporal structure match the real data.The paper leaves open whether that omission matters for the analyses a dataset will be used for.Source: §1 · p. 1
  • The proposed check tracks the number of connected components in a weekly learner-proximity graph across a term.It requires only synthetic data and permutations of that data, without real-data access, a case study, or a researcher panel.Source: §1 · p. 2
  • 2.6–4.9 times less variation appeared in synthetic weekly β0 curves than real curves across four annual cohorts.The paper also reports agreement in the level and shape of the weekly partition, while identifying a threshold-selection issue that can invalidate naive comparisons.Source: §1 · p. 2

2. Background

The background describes a privacy-preserving synthesis-and-validation platform, reviews what existing utility measures capture, and motivates a weekly cohort-structure summary. The paper treats that summary as a coarse structural descriptor, not a measure of learning.

2.1 Sharing educational data through synthesis and validation

  • ReLEAF shares differentially private synthetic cohorts for analysis and runs submitted code against real logs to validate specific findings.Only aggregate statistics are returned after provider review.Source: §2.1 · p. 2

2.2 What utility metrics measure

  • Existing evaluations compare distributional summaries, while epistemic precision measures later confirmation; the two do not track each other.The gap matters because matching those summaries cannot distinguish a generator that captured how variables move together from one that did not, while epistemic precision comes only after findings are formed and submitted.Source: §2.2 · p. 2
  • Prior topological comparisons are static, and proposed sequential-data dimensions do not compare how a population divides at an aligned time point.The paper positions weekly cohort division as an unmeasured quantity among these approaches.Source: §2.2 · pp. 2–3

2.3 Cohort structure as a weekly quantity

  • Counting weekly proximity-graph components summarizes how a cohort divides; tracking the count across weeks describes reorganisation rather than level.The count relates to single-linkage clustering at distance ε, but the paper uses a single-scale count and does not compute persistence.Source: §2.3 · p. 3
  • The component count is used only as a structural summary, not as a measure of learning or an educational interpretation of its level.It is deterministic, defined for disconnected graphs, and describes partitions rather than edge or triangle counts; the threshold-selection rule can undermine comparisons if applied separately to each dataset.Source: §2.3 · p. 3

3. Method

The method measures weekly cohort partitions with proximity-graph components and related descriptors, then compares real and synthetic curves under controlled thresholding. It also tests sensitivity to within-week marginal structure and calibrates what the surrogate comparison detects.

3.1 Structural indicator

  • β0 counts connected components in each week’s learner-proximity graph and is read as the number of distinct behavioural groups.The analysis also records isolated fraction and component concentration; the latter is the sum of squared component shares.Source: §3.1 · p. 3

3.2 Choice of working point, and why it must be controlled

  • The threshold rule targets mean weekly β0 near one quarter of cohort size, so cross-dataset comparisons use a common working point instead of each dataset’s own selected threshold.The target is a convention; the authors report that separate rule-selected thresholds can reverse a contrast, and fix ε = 0.68 from synthetic cohorts alone.Source: §3.2 · p. 4

3.3 Data

  • The study uses four annual ReLEAF cohorts of Japanese lower-secondary study-habit logs and their differentially private synthetic versions, analysed with identical code.The delivered data comprise 117–120 learners over 18 weeks and six time-of-day bands; their shape differs from the platform report’s three cohorts of 17 weeks and four windows. OULAD presentations supply real data only for constructing synthetic controls, not for comparison with the study-habit cohorts.Source: §3.3 · p. 4

3.4 What we compare

  • The primary comparison is the coefficient of variation of weekly β0 at the common working point, measuring movement across weeks rather than the level largely fixed by threshold selection.It is used instead of variance because mean β0 differs across cohorts by more than a factor of two, and instead of range because disclosure rules prohibit returning extreme values.Source: §3.4 · p. 5
  • The surrogate test compares observed and surrogate β0 curves after independently permuting each feature column within each week, preserving each week’s marginals.A large proportion means the observed curve lies inside the surrogate swarm; the test detects structure beyond independent within-week marginals but does not by itself identify between-learner structure. The study also compares partition descriptors and a per-learner correlation without outcome labels.Source: §3.4 · p. 5

3.5 Calibrating the interpretation

  • Calibration injects all-zero learners or scales all values in a week to test whether those mechanisms could account for a real cohort’s surrogate-test result.The study checks whether measured all-zero rates and activity variation are large enough to account for the reported result.Source: §3.5 · p. 5

4. Results

At a common working point, synthetic cohorts broadly reproduce the weekly partition, but substantially understate how much it changes across the term. Surrogate comparisons and controls show that the check can distinguish structural differences, while disclosed statistics leave some aspects of week-to-week structure unconstrained.

4.1 The level and shape of the weekly partition are reproduced

  • Partition descriptors broadly carry over: two cohorts agree within a few percent on every column, and study-volume association remains negative in every cohort and version.The synthetic version is less fragmented by 55% in 2023 and 24% in 2024; study-volume coefficients range from -0.97 to -0.81 for synthetic data and -0.90 to -0.63 for real data.Source: §4.1 · pp. 5–6, Table 1 · p. 6

4.2 The temporal variation is not

  • In every cohort, the real weekly β0 curve varies 2.6 to 4.9 times as much as the synthetic curve, with no overlap across four reference thresholds.Across the four thresholds, synthetic coefficients are 0.05–0.17 and real coefficients are 0.34–0.54; no synthetic value reaches any real value.Source: Table 2 · p. 6, §4.2 · p. 6

4.3 What a marginal-preserving surrogate recovers

  • All four real cohorts separate from surrogates preserving their within-week marginals, while three of four synthetic cohorts do not.Removing all-zero rows leaves real β0 curves unchanged to three significant figures, while weakening the synthetic 2024 result from 0.09 to 0.225.Source: §4.3 · pp. 6–7
  • Measured weekly mean-study-time variation is sufficient to produce a β0 variation gap of the observed size, though the authors do not claim it explains everything.Across datasets, β0 variation is 0.73–1.27 times study-time variation; disclosed statistics reproduce marginal zero rates to within 3%–13% but do not constrain which band and week are jointly zero.Source: §4.3 · p. 7
  • Synthetic cohorts place high study time in each year's examination week, unlike real cohorts, and the weekly β0 rank correlation ranges from -0.22 to 0.57.The synthetic mean is among the two highest of eighteen weeks in the examination week; in real cohorts that week ranks ninth to fifteenth.Source: Figure 1 · p. 7, §4.3 · p. 7

4.5 Can the check be passed?

  • Column permutation is calibrated, while only the Gaussian copula reproduces weekly values, with coefficient of variation within 3.5% of the real one in all four presentations.Across 24 column-permuted replications, test statistics range from 0.010 to 0.860 with mean 0.537, and a uniformity test does not reject.Source: §4.5 · p. 8
  • The multivariate normal separates from its own surrogates more sharply than real data in every presentation, yet misstates β0 level by 28%–143% and variation by 14%–38%.This control shows that separability alone does not establish fidelity.Source: §4.5 · p. 8

5. Discussion

The synthetic cohorts show little week-to-week structural change, which may leave researchers without questions about cohort reorganisation to validate. A synthetic-only check can flag structural differences before real-data validation, but β0’s suitability and the link to low confirmation rates remain untested.

5. Discussion

  • For questions about cohort reorganisation, synthetic data show little week-to-week change, so researchers may never formulate findings to submit for validation.The authors have not tested whether this omission explains low confirmation rates.Source: §5 · p. 8
  • Two of the three released study-habit statistics are marginalisations of one band-by-week matrix, constraining neither week-to-week differences nor alignment across learners.Whether a different summary can constrain those structures under differential privacy without impairing the rest of the analysis is not addressed.Source: §5 · p. 8
  • The check needs only a synthetic dataset and its permutations before release, but measuring the gap’s size still requires real data.Without real data, the surrogate comparison flagged three of four synthetic cohorts; unlike epistemic precision, the check examines one property rather than researchers’ actual conclusions.Source: §5 · pp. 8–9
  • The authors do not establish that β0 is the best descriptor of cohort structure; which descriptor serves the intended purpose remains open.The paper’s conclusion is narrower: real and synthetic cohorts differ along a dimension current fidelity metrics do not compare, and part of the difference can be detected before accessing real data.Source: §5 · p. 9

6. Limitations

Several design and data limitations qualify interpretation of the structural comparisons. The evidence is restricted to four cohorts from one subject, institution, and platform, and the size of the gap is not claimed to generalise.

6. Limitations

  • The surrogate removes both between-learner association and within-learner association among bands, so separation from it cannot identify which drives the difference.Calibration bounds how much zero co-occurrence could account for, but does not settle the question.Source: §6 · p. 9
  • Standardising features by each dataset’s own dispersion means a generator that misstates band spread is not penalised; the comparison concerns shape after standardisation.Observed dispersions differ substantially between cohorts.Source: §6 · p. 9
  • Differences between years cannot be separated from generation conditions, because feedback across ReLEAF’s multi-shot cycles may inform later generations and public information does not identify which feedback was used.The relevant feedback reflected in each analysed synthetic cohort is indeterminable.Source: §6 · p. 9
  • All-zero-row removal occurred only within an approved validation request, and the threshold reselected on filtered data is reported for reference, not as a second working point.Source: §6 · p. 9
  • A zero means no study interval could be measured in a band and week, not that the learner did not study; zero rates therefore should not be read as non-study rates.A single operation log within an hour can leave the estimated study time at zero.Source: §6 · p. 9
  • The platform does not warrant empirical validity of its synthetic cohorts, and reported values should not be read as describing the learners they represent.Source: §6 · p. 9
  • Evidence covers one subject at one institution: four cohorts of 117–120 learners over eighteen weeks each, delivered through one platform; the gap’s size is not claimed to generalise.The check transfers to datasets of this shape, but its informativeness elsewhere requires further datasets and synthesis mechanisms.Source: §6 · p. 9

7. Conclusion

The paper checks cohort structure through weekly connected components in proximity graphs, a dimension absent from variable-level summary-statistic comparisons. Across four cohorts, synthetic data reproduce the level and shape of the weekly partition but substantially understate its movement across the term.

7. Conclusion

  • Variable-level summary-statistic comparisons do not capture whether weeks differ from one another as they do in the real data.Source: §7 · p. 9
  • The structural check counts connected components in a weekly proximity graph over learners to describe how a cohort divides.Source: §7 · p. 10
  • 2.6–4.9 times: real curves varied this much more than synthetic curves across the term at a common working point, in all four cohorts.The synthetic versions reproduced the quantity’s level and the shape of the weekly partition, but not its movement.Source: §7 · p. 10
  • Computing the quantity requires one synthetic dataset and permutations of itself, and can be done before real-data validation; measuring the gap’s size requires real data.Source: §7 · p. 10
Loading 2609.27265v2…