PaperlayerView exact revision on arXiv

Source-linked AI summary

When Graph-JEPA Learns the Wrong Thing: Diagnosing and Repairing Category-Conditional Collapse

Gollam Rabby, Sören Auer

arXiv:2608.20516v1cs.LG

TL;DR

The paper asks whether standard health checks can certify usable instance information in joint-embedding predictive representations. It diagnoses and repairs a category-conditional collapse in a Graph-JEPA, then shows that near-ceiling retrieval remains non-diagnostic when the target is structurally reducible.

  • Problem

    Standard linear probing and effective-rank checks can pass even when a representation carries no usable instance information for retrieval.

  • Method

    The paper evaluates Graph-JEPA on masked aspects of typed scientific-reasoning subgraphs using controls, variance-allocation analysis, objective analysis, a loss ablation, and target-validity audits.

  • Results

    14.377 of 14.379 bits after repair falls to 0.307 bits when only the loss is reverted to regression, while ten-cell comparisons show no reliable relationship with a held-out reasoning probe.

  • Takeaways & Limitations

    Representation metrics and retrieval scores can saturate on category information or structurally reducible targets, so reasoning claims require reducibility and target-quality audits.

  • Takeaways & Limitations

    The evaluation target is structurally reducible, and a candidate-side metric improvement can be a re-metrisation rather than structural learning.

Abstract

from arXiv · show

Joint-embedding predictive architectures are selected almost universally by linear probing and effective rank. We report a case where both read healthily while the representation carries zero usable instance information. We repair it, and a second failure appears: the repaired metric saturates on a target carrying no structural information. Our corpus is a scientific-reasoning graph over 57,903 articles, each a subgraph. A Graph-JEPA predicts one masked aspect from a subgraph's remaining aspects, attaining linear-probe accuracy 0.871 and effective rank 18-47, yet retrieval recovers 0.00 of 14.4 bits (MRR 1.9e-4 vs chance 1.99e-4, p=0.98). Three upper bounds on the same pool and code recover nearly everything (+14.28, +14.34, +14.22 bits), ruling out corpus, masking, pool, and metric as causes. We trace this to variance allocation - frozen inputs place 86.05% of variance on subgraph identity and 0.40% on aspect identity, while trained latents place 0.39% and 99.61%. This is a property of the objective's optimum: the degenerate solution is a global minimum of the coupled predictor/EMA-target objective, present already at init. A repaired configuration reaches 14.377 of 14.379 bits, above the 13.865-bit oracle; reverting the loss to regression drops it to 0.307 bits, confirming it. Yet the repair licenses nothing about reasoning: the target is reducible, since intra-subgraph edges are a deterministic function of node census. The oracle reaches 96.4% of the ceiling, and our largest effect is the learning-rate schedule, not architecture. Bits and a reasoning probe show no relation across ten cells. A data-derived target fails a quality gate - 25.96% of nodes are duplicate placeholders, and the rest is more generic than supporting evidence. Rank, probes, and metrics can all saturate on an unsupportive evaluation. We release a harness with a reducibility audit and target gate.

1 Introduction

The paper shows that standard representation-health checks can pass while instance retrieval fails completely, then traces the failure to category-conditional variance allocation and repairs it. The repaired retrieval score is near-ceiling but does not establish reasoning because the target is structurally reducible.

  • Failure diagnosis: 0.871 probe accuracy and effective rank 18–47 coexist with 0.00 of 14.379 recoverable retrieval bits, at permutation p = 0.98.The representation passes conventional checks but fails its intended instance-retrieval task.
  • Failure diagnosis: +14.281, +14.335, and +14.220 bits from frozen-feature averaging, BM25, and a positive-control predictor show that the signal and evaluation harness work.All controls use the same pool, masking, and scoring code.
  • Repair and limits: 14.377 of 14.379 bits after repair falls to 0.307 bits when only the loss is reverted to regression, while the repaired target is structurally reducible.Across ten cells, the optimized metric and held-out reasoning probe show no reliable relationship (Spearman −0.24, n = 10).
  • Contributions: The study contributes a complete probe–task dissociation, a variance-allocation mechanism, formal non-identifiability and reducibility results, and a harness with target audits.The pipeline diagnosis combines controls, query-side localization, a loss ablation, and a data-quality gate.
  • Mechanism: 86.05%→0.39% paper-identity variance and 0.40%→99.61% aspect variance identify the trained representation’s category-conditional collapse.The allocation is measured both across encoders and within one encoder’s trajectory.
  • Mechanism: The coupled predictor/EMA-target objective admits global minima that differ by the entire recoverable budget, so it does not identify whether instance identity is encoded.The degenerate configuration is an objective optimum, not merely an unfavorable data property.

2 Related Work

The related work frames the failure as conditional rather than complete collapse: representations remain high-rank across categories but lose instance information within categories. It also distinguishes this unsupervised masking-induced analogue from supervised class collapse and motivates audits beyond candidate-side geometry.

  • JEPAs and collapse: JEPAs predict masked latent representations from visible context using an EMA target, while Graph-JEPA applies this paradigm to masked graph subgraphs.The paper’s failure concerns a category-conditional version of collapse rather than constant representations over the whole input domain.
  • JEPAs and collapse: The converged latents allocate variance to latent category rather than instance, yet existing probe and rank criteria do not flag the failure.The paper characterizes this configuration as a global optimum of the coupled objective.
  • Graph-JEPA and target validity: Graph-structured evaluation can still be unsupportive when subgraph edges are deterministic from node census, motivating a reducibility audit alongside representation metrics.The paper also raises data-quality concerns for machine-extracted scholarly corpora.
  • Class collapse: Unlike supervised class collapse, this unsupervised analogue receives its collapsing partition from the masking scheme and is total within categories.The paper contrasts it with dimensional collapse and effective-rank summaries, including counterexamples that clear the rpool > 10 threshold.
  • Evaluation and attribution: Candidate-side whitening, anisotropy correction, and metric changes cannot repair a degenerate query bank, making query-side diagnosis necessary.The paper pairs this result with parameter-free baselines for attribution.

3 Setup and Protocol

The setup builds paper-local typed reasoning subgraphs, masks one aspect, and predicts it from the remaining aspects with a heterogeneous Graph-JEPA. Retrieval uses fixed queries, an aspect-matched corpus pool, cosine scoring, bit recovery against a measured ceiling, bootstrap uncertainty, and instrument self-tests.

  • Corpus and graph: 57,903 maskable scientific articles form paper-local subgraphs with typed claim, method, result, evidence, and implication nodes.The broader graph build contains 58,149 records, while the diagnostic harness uses 58,145 non-empty aspect texts.
  • Corpus and graph: Each node uses a frozen sentence encoder, mean pooling is the identity map, and field labels are reserved for probing rather than model input.The graph includes five typed reasoning-node categories and typed scientific-structure edges.
  • Task and configurations: For each maskable subgraph, one present aspect is held out while a heterogeneous GIN, pooling operator, EMA target encoder, and structural context mixer predict its patch embedding.The protocol uses random-walk structural encoding and leak-safe masking.
  • Retrieval protocol: 4,000 fixed-seed queries retrieve from the full aspect-matched corpus pool using cosine similarity in each model’s retrieval space.Lexical systems use a nested 2,000-item subsample because they are approximately 500× costlier per query.
  • Scoring and uncertainty: Bit recovery is measured against a chance baseline of MRR = 1.99 × 10−4 and a measured ceiling, with percentile and paired bootstrap comparisons.The protocol avoids ratios against chance because near-chance MRR makes them sampling-noise dominated.
  • Instrumentation: A 20-assertion self-test must pass before measurement, checking perfect-ranker ceiling recovery, uniform-ranker chance behavior, constant-score AUC, and masking isolation.The probe uses disjoint, label-stratified splits and preregistered thresholds.

4 Why a Constant-per-Category Query Is a Global Optimum

The coupled predictor and EMA-target objective admits category-measurable global minima that discard instance information, and neither geometry nor post-hoc re-metrisation can recover it. The paper distinguishes this learned-target fixed-point mechanism from frozen-target conditional-mean collapse and shows why structural reducibility limits interpretation.

  • Coupled objective: At stationary points, the EMA target branch equals the online parameters, making both arguments of the objective learned rather than treating the target as exogenous.The context includes the visible context and aspect designator, while the target is the held-out subgraph.
  • Coupled objective: Category-measurable query and target branches can achieve zero risk while discarding instance residuals, so the objective does not identify retrievable instance information.Two global optima of the same objective can differ by the entire recoverable budget.
  • Fixed-point mechanism: The frozen-target conditional-mean result is only a special case; here, an oracle recovers +14.281 of 14.385 bits from the context, so zero retrieval reflects the learned-target branch instead.The context is strongly informative about the instance residual, but the target can discard that residual together with the predictor.
  • Why geometry cannot help: A constant query bank makes every injective frame transformation preserve the same ranking, yielding E[MRR] = HN/N and B = 0 exactly.This explains why changing metrics, frames, or geometry cannot repair the degenerate query representation.
  • Structural reducibility: If graph edges are determined by node census, message passing cannot access structural information beyond the feature vectors of each patch’s own nodes.Any performance gain over a fixed training-free statistic is then a re-metrisation of node features rather than structural learning.

5 The Diagnostic Ladder

The diagnostic ladder localizes the failure to the query bank rather than the inputs, pooling, candidate bank, or retrieval metric. Rank restoration and re-metrisation do not improve retrieval, while direct controls preserve substantial recoverable information.

  • Baseline diagnostics: 0.00 bits at probe accuracy 0.871 persists across objectives, aggregators, and depths, showing that standard representation diagnostics can miss the failure.Sum pooling raises the probe to 0.963 while lowering pooled rank to 1.25, with retrieval unchanged.
  • Rank restoration: Whitening raises instance-variance share from 0.39% to 9.86% and input effective rank to 310.9 of 384, but bits remain 0.The probe falls from 0.871 to 0.755 because whitening removes the shared-mean component it exploited.
  • Query-side localization: Five candidate-bank frames move retrieval only from 1.7 × 10−4 to 2.6 × 10−4, buying 0.4 bits against a 14.4-bit deficit.The candidate bank has effective rank 18.1, whereas 4,000 distinct queries have rquery = 1.9, DC energy 0.9999, and self-similarity 0.004.
  • Pooling control: A pooling control preserves rank from rnode = 47.42 to rpool = 47.31, while predictions remain at effective rank 1.97 and the similarity margin is z = −0.001.Across aspects, pooled rank spans 2.05 to 47.31 without moving bits.
  • Pool difficulty: All four systems stay within 1.2% of the ceiling across pool difficulties, while baseline Graph-JEPA recovers 0.0% and the repaired configuration recovers 100.0%.The table compares bits recovered against hard within-category pools of increasing difficulty.

6 Three Upper Bounds: Task, Harness and Objective Are Separable

Three matched controls show that the retrieval task and harness can recover nearly all available identity information; the null is therefore localized to the trained objective and representation.

  • Controls: The controls exclude the corpus, task definition, pool construction, masking, and metric as explanations of the null.They establish that the context determines the target well enough for an identity-preserving global optimum to exist.
  • Interpretation: p1 bounds the harness but does not by itself attribute the gap because it differs from the tested model in both features and predictor stack.A preregistered 2 × 2 factorial is needed to separate those factors.
  • Difficulty: −0.061 bits at the full pool marks the reference predictor’s deficit relative to the oracle, with non-overlapping bootstrap intervals across all four pool sizes.The predictor remains below the oracle as the candidate pool becomes harder.
  • Difficulty: A near-constant deficit from K=10 to 58,145 indicates a large-margin, reducible task whose near-ceiling scores are uninformative about structure.BM25’s MRR falls only 0.9960→0.9910 across that range.

7 The Mechanism: Which Variance Is Encoded, and When

The trained representation reallocates variance from subgraph identity to aspect identity, and the evidence supports a degenerate configuration already present at initialization that the objective does not remove.

  • Endpoint allocation: 86.05%→0.39% is the shift in paper-identity variance, while aspect-identity variance shifts 0.40%→99.61% between frozen inputs and trained latents.The central comparison is measured within one encoder; the endpoint comparison alone does not establish the trajectory.
  • Mechanism: ρ = 0.9961 loses 11.045 of 12.845 bits in simulation, showing that aspect allocation is dominant but not sufficient for total collapse.Total simulated collapse requires ρ ≥ ρ⋆ = 0.99999, so the measured residual is unexplained by allocation alone.
  • Initialization and training: Step-0 measurements support the weaker claim that category-measurable degeneracy precedes training and is not removed by the objective.The objective supplies no gradient that leaves the configuration because the configuration is already a global minimum.
  • Objective intervention: 14.359→0.307 bits when only the loss is changed identifies the loss as the operative lever.The repaired configuration rises from −0.07 bits at step 1 to +14.24 by step 500.

8 What the Task Measures and What We Tried

The retrieval benchmark is strongly lexical and structurally reducible, so high repaired scores do not establish reasoning ability; matched interventions further show that common architectural changes leave the collapse intact.

  • Target audit: 25.7–39.0% of subgraph text is covered by the most frequent 4-grams per aspect, while function words alone classify aspect at 0.937.Removing shared 5-grams reduces BM25 only 0.085 bits, from +14.335 to +14.250.
  • Target audit: The benchmark measures redundancy-based subgraph identification, not scientific reasoning, despite BM25 recovering 99.7% of the ceiling.The paper explicitly makes no claim that performance on this task measures scientific reasoning.
  • Reasoning probe: Bits span 11.808–14.379 and A1 spans 0.711–0.985, with Spearman ρ = −0.24 across ten cells and no reliable relationship.The cells are neither independent nor randomly selected, so the paper claims no trade-off or direction.
  • Interventions: Seven matched intervention cells keep bits at 0 while rpool spans 1.25–2.03 and the probe spans 0.604–0.964.The best attention/InfoNCE cell reaches MRR = 2.3×10^-4, within sampling noise of baseline against a 14.4-bit deficit.
  • Controls: The oracle, BM25, and same-architecture controls recover +14.281, +14.335, and +14.220 bits, respectively, while whitening leaves bits at zero.These controls separate task and harness solvability from the objective’s failure.
  • Limitations: All results come from a single corpus, and the evaluation target is proved census-determined, limiting the empirical scope to this pipeline on this corpus.The data-derived target also has pre-registered quality gates reported despite failure.

9 The Repair Succeeds, and the Metric Stops Meaning Anything

The repaired objective restores retrieval, but the resulting near-ceiling metric is uninformative because the target is structurally reducible and reasoning performance remains decoupled.

  • The repair succeeds: 14.377 of 14.379 bits follows the corrected objective, while reverting only the loss reduces recovery to 0.307 bits.The loss change produces a 14.05-bit swing on one variable.
  • The repair succeeds: +1.337 bits from changing only the learning-rate schedule exceeds the architectural effects measured in the matched sweep.The target-frame main effect is +0.071 bits, while the schedule moves the baseline cell from 12.952 to 14.289 bits.
  • The metric stops meaning anything: Spearman ρ = −0.24 across 10 cells shows no reliable relationship between recovered bits and the reasoning probe.Bits span 11.808–14.379 while A1 spans 0.711–0.985, and the cells are neither independent nor randomly selected.
  • The metric stops meaning anything: 13.865 of 14.379 bits are recovered by a training-free oracle because intra-subgraph relations are deterministic functions of node census.The oracle reaches 96.4% of the ceiling, leaving 0.514 bits of headroom for any trained model.
  • The metric stops meaning anything: Five of nine relations fail the reducibility check, so retrieval scores, probes, and rank statistics cannot detect that the evaluation target is information-free.The audit compares relation cardinalities with endpoint-type counts.
  • The metric stops meaning anything: 25.96% of contradicting-evidence nodes are exact duplicate rows, causing the data-derived replacement to fail its quality gate before training.The largest duplicate group contains 3,031 identical strings, and 31,545 challenge edges point to duplicated rows.

10 Discussion

The paper proposes a diagnostic harness for separating model failure from pipeline and target failure. Its broader conclusion is that evaluation requires explicit oracle, reducibility, query-bank, positive-control, and data-quality checks, while the empirical claims remain corpus-bounded.

  • Diagnostic instruments: A matched training-free oracle separates task difficulty from pipeline capability, while a guarded positive control separates harness failure from model failure.The recommendations require reporting recovered bits against the recoverable ceiling and comparing them with a matched no-training baseline.
  • Scope and open problems: The empirical scope is one corpus with a census-determined evaluation target, so the findings are not presented as claims about JEPAs in general.The paper identifies a non-census-determined graph as the decisive next experiment.
  • Diagnostic instruments: A reducibility audit compares relation cardinalities with node census counts to identify targets that carry no information beyond node existence.A relation matching an endpoint type’s count is information-free under the proposed check.
  • Diagnostic instruments: Query-bank instrumentation, checkpoint-wise variance decomposition, and bidirectional masking tests localize category collapse and context leakage.The proposed measurements include effective rank, mean pairwise cosine, variance allocation at every checkpoint, and perturbation-based invariance checks.
  • Diagnostic instruments: A data-quality gate should precede training on machine-extracted targets, using duplicate-census and genericness checks and treating a fired gate as a result.The paper also recommends monitoring a second metric because the optimized metric and reasoning probe moved independently.
  • Scope and open problems: +1.337 bits is the largest positive effect in the reported sweep, exceeding the disclosed architectural effects.The schedule effect is identified as a nuisance factor that need not be disclosed by convention.

11 Limitations and Conclusion

The paper concludes that healthy probes and rank can coexist with zero instance retrieval, while near-perfect repaired retrieval can still reflect a reducible target rather than learned reasoning. It limits its empirical scope to one corpus and pipeline, discloses data-quality and implementation caveats, and releases preregistered diagnostic infrastructure.

  • Limitations: The empirical claim is limited to this pipeline on one corpus, and leakage controls bound but do not eliminate surface overlap.The bm25-no-ov control retained +14.250 bits.
  • Limitations: The bits/A1 relationship used n = 10 non-independent cells with Spearman −0.24, so the paper draws no directional conclusion.The authors also note that the probe uses 488 coarse labels and that a large permutation p-value indicates failure to reject, not equality.
  • Limitations: 25.96% of one extracted field consisted of placeholder text, limiting the evidential value of the data-derived target.The authors disclose this artifact and state that the remaining content was not sufficient to support the intended evaluation claim.
  • Conclusion: 0.00 of 14.379 recoverable bits was retrieved despite probe accuracy 0.871 and healthy effective rank, while controls recovered nearly the full budget.The positive control recovered +14.220 bits, and BM25 recovered +14.335 bits through the same evaluation harness.
  • Conclusion: 14.377 bits after repair fell to 0.307 when only the loss reverted to regression, a 14.05-bit swing confirming the proposed mechanism on one variable.The repair exceeded the 13.865-bit oracle, but the target was later shown to be structurally reducible.
  • Release: The authors release the graph construction, diagnostic protocols, audits, preregistration, and per-seed logs underlying the reported tables.The harness includes reducibility and data-quality gates, with decision thresholds written before results.

A1 Claim–evidence map

The claim–evidence map makes quantitative provenance explicit by linking each abstract and introduction claim to the table or figure that establishes it. Each value appears once as a source macro, preventing divergence between prose and tables.

  • Claim–evidence map: Each value appears exactly once as a macro in the source, so prose and tables cannot diverge.
  • Claim–evidence map: Every quantitative claim in the abstract and introduction is paired with the table or figure that establishes it.The map records both the claim’s value and its evidential location.

A2 Number provenance

The provenance appendix keeps experimental blocks separate because they measure different objects and support the central results. It also identifies a within-encoder comparison and records a graph-edge count discrepancy without silently resolving it.

  • Number provenance: Blocks A and C share the same cached graph, masking convention, aspect fields, bits measure, and raw records, enabling direct comparison of their reported values.The reported ceilings agree to 0.006 bits and chance levels agree to two significant figures.
  • Number provenance: Table 3’s central variance comparison is measured within the block-A encoder, while block-B features are retained only for the three ceiling computations.The block-A pair required one forward pass on the cached graph and no retraining.
  • Provenance caveats: The schema dump reports 17,631 cites edges versus 11,791 in the stored graph object, so the paper quotes 11,791 and flags the discrepancy.The discrepancy does not affect the reducibility proposition because cites is negligible at either count.

A3 Extended related work

The related-work discussion positions the failure as within-category collapse that evades global rank and non-constantness checks, while the paper’s theoretical and empirical analyses connect objective degeneracy, reducible targets, and evaluation safeguards.

  • Extended related work: Global collapse-avoidance and dimensional-rank analyses miss representations that vary across categories but are constant within each category.The downstream task requires the within-category variation that these checks do not measure.
  • Objective degeneracy: A degenerate predictor/EMA-target pair is a global minimum with zero risk, and an injective alternative can retain the full recoverable information at the same minimum.The two global minima therefore differ by the entire recoverable budget.
  • Mechanism and interventions: 0.4 bits were gained against a 14.4-bit deficit by changing retrieval geometry, whereas changing the objective moved performance by the full recoverable budget.The empirical query-bank rank was 1.9, close to the theoretical three-category bound but not exactly constant.
  • Target reducibility: The evaluation target can be information-free when graph edges are determined by node census, so near-ceiling retrieval does not establish learned structural reasoning.The data-processing argument bounds target information by the node features alone.
  • Collapse diagnosis: The pathology is present at initialisation, and increasing the algebraic target-rank ceiling from 2 to 128 leaves measured target rank between 1.6 and 2.0.
  • Repair analysis: 14.376 bits with an aspect embedding exceeded 14.209 bits with RWSE, while removing the structural cue produced 13.529 bits.The authors attribute the low RWSE rank to the graph’s census-determined structure.
  • Evaluation safeguards: MRR = 1.0000 is treated as a degeneracy signature requiring a leak check, because an exact reading coincided with a +0.1465-AUC edge-type leak.The authors instead headline the 20k, 3-seed result: 14.377 ± 0.002 bits.
Loading 2608.20516v1…