ReLaG: A Scalable Framework Generalizing Random Splits to Data with Latent Relations

• Random splits can place related samples in train and test sets, allowing memorization and producing overly optimistic generalization estimates.The problem is especially salient in biochemical datasets, where evolutionary processes, screening, and dataset bias create sample dependencies.Source: §1 · p. 1
• Existing relation-aware methods are constrained by modality-specific assumptions, computational costs, and often the lack of a principled similarity-threshold selection method.Users may therefore fall back on conventional thresholds that do not suit the task.Source: §1 · p. 2
• ReLaG models samples through group-specific latent variables, infers groups with an approximate proximity graph and community detection, then keeps each community within one split.The framework is modality-agnostic and makes subsets independent under its latent-variable model.Source: §1 · p. 2
• Across molecular and protein datasets, ReLaG achieves minimal leakage comparable to existing methods while its most expensive step runs in O(n log n) rather than O(n^2) time.This scaling enables relation-aware splits at larger dataset sizes.Source: §1 · p. 2
• A label-free procedure selects the threshold where production and training data are most similar at the community level; absent alignment, production data are flagged as out of distribution.Without relational structure, ReLaG reduces to random splitting.Source: §1 · p. 2
• The number of inferred communities, Neff, approximates effective dataset size and is associated with generalization performance, suggesting a basis for diversity-aware experiment prioritization.The proposed use is to prioritize experiments expected to yield new communities.Source: §1 · p. 2

Optimization-based approaches

• Optimization-based methods optimize a split criterion: SIMPD mimics temporal drift, MOOD selects a protocol matching deployment distances, and DataSAIL minimizes cross-subset similarity.Source: §2 · p. 2

Clustering-based approaches

• Clustering-based methods assign clusters to subsets, using molecular fingerprints or UMAP for molecules and k-mer distances or protein families for sequences.Examples include hierarchical clustering, HDBSCAN, SpanSeq, and Pfam-based splits.Source: §2 · p. 3

Graph-based approaches

• Graph-based methods partition similarity graphs using cuts, connected components, or community detection, while thresholds typically determine which samples are connected.GraphPart iteratively reassigns or removes samples to satisfy a specified threshold.Source: §2 · p. 3

Generic methods

• DataSAIL and Hestia are the only generic methods described as cross-modal, but DataSAIL minimizes subset similarity while Hestia estimates performance across a costly threshold sweep.ReLaG instead uses a modality-independent generative model and production data to set resolution for independent subsets.Source: §2 · p. 3

• Evolutionary descent and molecular screening both produce families of related observations, so these datasets are not necessarily i.i.d. at the sample level.The examples include related biological sequences and local molecular variants around active hits or scaffolds.Source: §3 · p. 3
• In the two-level model, group-specific latent variables gi are drawn independently from G, and each sample sij is generated conditionally on its group's gi.Source: Figure 2 · p. 3, §3 · p. 4
• Samples sharing gi are dependent after marginalizing over it, whereas samples from different groups are independent; the relevant grouping depends on resolution τ.Examples of groups include molecular libraries, drug-hit variants, and protein families or superfamilies.Source: §3 · p. 4

4.1 THE RELAG ALGORITHM

• ReLaG infers communities of related samples and assigns each community wholly to one split, rather than splitting individual samples.Its four steps are defining a distance, constructing an approximate proximity graph, detecting communities, and partitioning them.Source: §4.1 · pp. 4–5
• HNSW supports approximate proximity-graph construction; retaining every evaluated pair below the threshold adds edges without additional distance evaluations.Exact construction requires n(n−1)/2 distance evaluations, which can be prohibitive for large datasets or expensive distances.Source: §4.1 · pp. 4–5
• Leiden community detection with a null-model objective identifies regions whose internal edge density exceeds chance expectations; modularity is an alternative when no modality-specific null model is available.The null edge probability is estimated from unrelated pairs using a procedure that removes genuine relatedness while preserving the marginal data distribution.Source: §4.1 · p. 5

4.2 SELECTING THE DISTANCE THRESHOLD

• The threshold controls grouping resolution: larger values produce coarser communities, while smaller values produce finer ones, and the appropriate resolution is unobserved.Figure 4 illustrates that the right threshold makes production communities integrate with the training-community distribution.Source: §4.2 · p. 5, Figure 4 · p. 5
• ReLaG selects a candidate threshold using unlabeled production samples, seeking the resolution at which production and training communities have similar distance distributions.The procedure sweeps thresholds using a graph built over the union of training and production samples.Source: §4.2 · pp. 5–6
• It compares single-linkage distances from pure-training and pure-production communities to training-containing communities, using the Kolmogorov–Smirnov statistic to quantify their distributional difference.A minimal statistic near 0 indicates alignment at the selected resolution; a value near 1 indicates no threshold aligns the distributions and signals production data are out of distribution.Source: §4.2 · p. 6

5.1 SYNTHETIC VALIDATION; 5.2 REAL DATA

• On synthetic data, random splits yielded test PCC 0.86 versus production 0.74, while relation-aware estimates matched production; on real data, random splitting scored higher than relation-aware methods.On synthetic data, ReLaG, Hestia, DataSAIL, and the oracle had near-zero test–production PCC differences; the real-data comparison covered six molecular and two protein datasets.Source: §5.1 · p. 7, §5.2 · p. 7, Figure 6 · p. 7

5.3 THRESHOLD

• Threshold selection returned τ = 0 for approximately i.i.d. MNIST and τ = 0.55 for related DBAASP, but BELKA had minimum KS 0.98 with no alignment.These cases represent no detectable relational structure, recoverable grouping, and production data out of distribution, respectively.Source: Figure 7 · p. 8, §5.3 · p. 8

5.2 REAL DATA

• At n = 3.125M, ReLaG finished in 1.7 hours versus Hestia’s extrapolated 2.1 weeks, an estimated 207× speedup.The Figure 8 caption reports approximately linear scaling for ReLaG and quadratic scaling for Hestia; runs were limited to one day and values above that limit were extrapolated.Source: §5.2 · p. 8, Figure 8 · p. 8

5.3 THRESHOLD

• With no relational structure, singleton communities recover random splitting; with groups, ReLaG splits along their boundaries, while a single group signals no non-leaking split is possible.In that limiting case, ReLaG signals that training and production data are mutually out of distribution.Source: §5.3 · pp. 8–9

5.4 FREE DIVERSITY ESTIMATE

• Removing whole communities reduced performance more than removing the same number of random samples, and the gap correlated with reduced Neff, motivating acquisition of samples forming new communities.Neff is the number of communities and is evaluated as an estimate of effective dataset size.Source: §5.4 · p. 9
• For OOD BELKA, the training graph collapsed to one community, yielding Neff = 1 and indicating that more groups—and thus more diversity—are needed to generalize to other groups.Source: §5.4 · p. 9

• ReLaG infers related-sample groups through an approximate proximity graph and community detection, then preserves inferred group boundaries in train–test splits.Source: §6 · pp. 9–10
• Across molecular and protein datasets, ReLaG provides leakage-aware evaluations comparable to existing generic methods while enabling substantially larger-scale splits.Its label-free threshold procedure adapts splitting resolution to production data, recovers random splitting without relational structure, and flags out-of-distribution production data.Source: §6 · p. 10
• ReLaG remains slower than random splitting, requires a meaningful distance function, and needs unlabeled production data for threshold selection.Without a modality-specific null model, it falls back to a less interpretable modularity objective.Source: §6 · p. 10

• The authors report using generative AI to formulate mathematical equations and to design and provide feedback on research experiments.Source: §7 · p. 10
• They report using generative AI for literature sourcing, figure and code work, paper drafting and editing, brainstorming, and other listed tasks.Source: §7 · p. 10
• The authors say they reviewed AI-assisted work, tested generated code, verified equations independently by at least two co-authors, and checked figures against underlying data.Source: §7 · p. 10

• The complete ReLaG implementation is provided as an open-source Rust library with Python bindings.Source: §8 · p. 10
• The reproduction code automatically downloads and preprocesses datasets, runs experiments, and generates every paper figure.Source: §8 · p. 10
• The repository README lists scripts and commands for reproducing each experiment and figure.Source: §8 · p. 10

A.1.1 NULL MODEL

• p₀(τ) is the probability that two unrelated samples fall within distance τ.The estimate is modality- and problem-specific.Source: §A.1.1 · p. 15
• Monte Carlo null distributions remove relational structure while preserving marginal characteristics, using different constructions for proteins, molecules, and MNIST.Protein sequences are shuffled; molecules are generated as chemically valid SELFIES; MNIST null pairs come from distinct ImageNet classes.Source: §A.1.1 · p. 15
• For rare p₀(τ), a Generalized Pareto Distribution fitted to the smallest 1% of sampled distances extrapolates the lower tail.The resulting estimate is used as the CPM resolution parameter.Source: §A.1.1 · p. 15

A.1.2 DATA PREPARATION

• Evaluation uses eight real datasets—six molecular and two sequence-based—and re-splits pooled samples after discarding predefined train–test splits.The datasets include molecular activity, permeability, mutagenicity, and lipophilicity tasks, antimicrobial peptides, and enzyme optimal-temperature regression.Source: §A.1.2 · p. 16
• Synthetic peptide datasets contain 15,000 sequences from 1,000 families, generated by branching mutation from family-specific ancestors.The annotated and production datasets use different random seeds and share no family; labels are generated from sequence-derived physicochemical descriptors using a nonlinear function.Source: §A.1.2 · p. 16

A.1.3 SPLIT COMPARISON

• All methods use the same data, representations, and MLP; splits allocate 80% to training and validation and 20% to test, with fixed modality-specific thresholds.The thresholds are τ = 0.60 for molecules and τ = 0.50 for proteins; experiments use ten random seeds.Source: §A.1.3 · p. 17

A.2 SCALING

• ReLaG scales more favorably than Hestia on molecules, but Leiden community detection may dominate runtime at very large dataset sizes.The authors attribute its faster growth to complexity scaling with the graph's number of edges.Source: §A.2 · pp. 17–18

A.3.1 ALGORITHM

• Threshold selection sweeps upward from τ = 0 and chooses the evaluated threshold minimizing the KS statistic, preferring the largest threshold in a tie.The sweep stops when all candidate edges are retained or no pure-training or pure-production communities remain.Source: §A.3.1 · p. 18

A.3.2 UNIREF EXPERIMENT

• At UniRef50 scale, ProtSpaM replaces global alignment for distance computation because its approximately linear sequence-length complexity makes threshold search feasible.The authors report that ProtSpaM and global alignment distances agree closely on the same sequence pairs.Source: §A.3.2 · p. 18, Figure 12 · p. 20
• τ = 0 was selected on UniRef50, where training and production relatedness distributions were mostly similar and sequence pre-clustering had removed redundancy.At τ = 0.7, the KS statistic is 0.080; at τ = 0.9, it is 0.135.Source: §A.3.2 · pp. 18–20

A.4 SINGLETONS: NOISE OR SIGNAL?

• Singleton information varied by dataset: they were more informative in three tasks, less informative in three, and statistically indistinguishable in two.The authors count singletons toward Neff only when they show significantly greater per-sample signal than community-matched samples.Source: §A.4 · p. 21, Table 3 · p. 21