PaperlayerView exact revision on arXiv

Source-linked AI summary

AC-JEPA: Anchored Cross-Modal Joint Embedding Predictive Architecture for Visual-to-Symbolic Translation

Chaofei Hong, Ziyang Chen, Baochen Wang, Christina Yi Jin, Qing Zhang

arXiv:2610.36493v1cs.AIcs.CL

TL;DR

Scientific cross-modal translation must preserve fine-grained structure despite imperfect, costly-to-curate paired data, while existing JEPA formulations largely focus on within-modality prediction. AC-JEPA combines causal SFT with bidirectional masked latent prediction and uses continuous latent targets for pseudo-labeled pairs. On OCSR, it improves over the SFT anchor with clean pairs and gains from latent self-training where direct pseudo-label SFT degrades.

  • Problem

    Scientific cross-modal translation needs to preserve fine-grained target structure despite imperfect, costly-to-curate paired data, while most JEPA formulations focus on within-modality prediction.

    Source: §1 · pp. 1–2

  • Method

    AC-JEPA combines causal SFT with bidirectional masked cross-modal latent prediction, using pseudo-labeled pairs as continuous latent targets rather than hard token targets.

    Source: §1 · p. 2

  • Results

    On OCSR, gold-pair AC-JEPA improves both WildMol Simple and Markush over the SFT anchor, while latent self-training gains with the same pseudo-labeled pairs where direct pseudo-label SFT degrades.

    Source: §1 · p. 3

  • Takeaways & Limitations

    The results support anchored latent prediction as a training strategy for exact cross-modal translation with curated and imperfect pairs.

    Source: §1 · p. 3, §6 · pp. 11–12

    Noise tolerance is conditional: systematic bias may shift representations, controlled-corruption support is limited to the tested OCSR setting, broader tasks need validation, and the added generation and training are not compute-free.

    Source: §5 · p. 11

Abstract

from arXiv · show

Cross-modal translation requires models to capture shared semantics across modalities while preserving the fine-grained structure required by the target modality. Yet in real-world scientific domains, paired data are often imperfect, while obtaining clean pairs at scale is costly. Supervised fine-tuning (SFT) directly optimizes cross-modal translation through categorical token-level supervision, but this objective is sensitive to errors in the target sequences. JEPA offers a complementary representation-space objective, but most existing formulations focus on prediction within a modality rather than translation across modalities. We introduce Anchored Cross-Modal JEPA (AC-JEPA), an SFT-compatible plug-in that predicts masked intermediate states in both cross-modal directions using paired data. The joint-embedding objective targets cross-modal structure predictable from context, while the causal SFT path retains direct token-level supervision for the target sequence. During self-training, AC-JEPA learns from continuous latent targets derived from imperfect pseudo-labeled pairs, offering intrinsic tolerance to local data errors without promoting pseudo-tokens to hard categorical supervision. Using optical chemical structure recognition as a testbed, AC-JEPA trained on clean pairs achieves 76.07% accuracy on the Simple set of the WildMol benchmark, compared with 67.40% for SFT alone. Using the same pseudo-labeled data for both approaches, AC-JEPA-based self-training further raises accuracy to 80.23%, whereas accuracy under direct SFT drops from 67.40% to 63.47%. These results support SFT-anchored joint-embedding prediction as an effective strategy for cross-modal translation under both curated and imperfect supervision.

1 Introduction

Scientific image-to-symbolic translation must recover fine-grained structure and serialize it under a strict grammar, often from imperfect, costly-to-curate pairs. AC-JEPA combines causal SFT with cross-modal latent prediction, and its OCSR evaluation reports gains with clean and pseudo-labeled pairs.

1 Introduction

  • Scientific image-to-symbolic translation must recover local primitives, spatial or topological structure, and long-range dependencies before producing a grammar-valid sequence.In OCSR, molecular diagrams encode atoms, bonds, stereochemistry, and topology, while SMILES is a compact, grammar-sensitive sequence [8, 23].Source: §1 · p. 1
  • SFT learns target grammar through exact token supervision, but hard pseudo-token targets can bias the decoder when generated sequences contain local errors [8] [9].The categorical objective does not directly require intermediate states to recover predictable structure removed from either modality.Source: §1 · p. 1
  • AC-JEPA integrates cross-modal latent prediction into the SFT-compatible generator, using masked paired views to predict intermediate states from remaining multimodal context.Both branches update the shared backbone; latent MSE shapes representations used by causal decoding, and modality-wise SIGReg prevents collapse [5].Source: §1 · p. 2
  • AC-JEPA learns predictable cross-modal structure from pseudo-labeled pairs without converting local pseudo-label errors into categorical supervision.The approach retains the SFT path for direct token-level supervision.Source: §1 · pp. 2–3
  • AC-JEPA improves both WildMol Simple and Markush over the SFT anchor with clean pairs, and latent self-training gains where direct pseudo-label SFT degrades.Both approaches use the same model-generated pairs; the comparison concerns the training route, not additional unique data.Source: §1 · p. 3
  • Local bounds relate decoder-relevant latent residuals to next-token decision stability and characterize clean-output stability under pseudo-label perturbations.These are theoretical contributions alongside the empirical OCSR validation.Source: §1 · p. 3

2 Related Work

Prior multimodal representation and JEPA work learns shared content or predicts latent representations, while exact scientific translation also requires local structure useful to a causal decoder. AC-JEPA places bidirectional masked-state prediction inside the same generator used for symbolic translation.

Multimodal representation learning

  • Global image–text correspondence can support transferable representations, but exact scientific translation can also depend on local, modality-specific structure.AC-JEPA applies local masked-state prediction within the fused representation used for autoregressive decoding, while SFT preserves the task-specific output interface.Source: §2 · p. 3

Latent predictive learning and multimodal JEPA

  • AC-JEPA differs from prior latent-prediction approaches by placing bidirectional masked-state prediction inside the autoregressive image-to-SMILES generator deployed for exact translation.This makes preserving intermediate states useful to the upper causal decoder a compatibility constraint.Source: §2 · p. 3

Self-training with imperfect pseudo-labels

  • Direct pseudo-label SFT treats generated tokens as exact categorical targets, whereas AC-JEPA predicts continuous intermediate states from masked multimodal views using the same pseudo-labeled data.The comparison holds data selection constant across the two training routes.Source: §2 · p. 4

Optical chemical structure recognition

  • OCSR must recover molecular graph structure, stereochemistry, abbreviations, and drawing conventions; this work studies latent prediction as a complementary training strategy for clean and pseudo-labeled pairs.The Markush setting also requires handling variable molecular structures.Source: §2 · p. 4

3 Method

AC-JEPA combines autoregressive SFT and masked latent prediction in one image-to-SMILES generator. Its curriculum anchors exact translation before using latent prediction on gold and pseudo-labeled pairs, and its local analysis relates latent residuals and pseudo-label perturbations to output stability.

3 Method

  • SFT and latent prediction update a shared backbone: autoregressive SFT trains the causal decoder, while JEPA predicts intermediate states and modality-wise SIGReg prevents collapse.The same latent objective can incorporate pseudo-labeled pairs without treating generated SMILES as default token-level targets.Source: §3 · p. 4

3.1 SFT-Anchored Backbone and Latent Interface

  • The latent interface sits after visual and symbolic context fusion but before next-token logits, allowing latent prediction and SFT to shape shared lower layers while preserving causal decoding.The SFT path continues through the remaining decoder and language-model head.Source: §3.1 · p. 4

3.2 Bidirectional Prediction and Latent Regulation

  • Structured masking hides contiguous SMILES spans and rectangular image regions; bidirectional prediction recovers their states from the remaining multimodal context, while SIGReg regularizes clean image and SMILES latents separately.The clean latent is detached only as the MSE target; SIGReg acts on undetached clean latents.Source: §3.2 · pp. 5–6, Figure 2 · p. 5

3.3 From SFT Anchoring to Latent Self-Training

  • The curriculum first trains on synthetic MolParser-7M pairs and Real-SFT 100K, then warms up JEPA modules with the Base SFT backbone frozen before independent gold-pair and self-training runs.Gold-pair training reuses synthetic pairs already seen during pretraining for latent prediction while retaining Real-SFT for token supervision; self-training adds pseudo-SMILES while gold-pair SFT remains the anchor.Source: §3.3 · pp. 6–7, Figure 3 · p. 6

3.4 Local Analysis of Task-Aligned Representations and Noise Tolerance

  • Under a local model, reducing decoder-relevant latent residuals tightens the next-token decision-reversal bound at a fixed positive mean margin.The analysis uses simplifying assumptions and provides a mechanism-level interpretation, not a general guarantee for full nonlinear training dynamics.Source: §3.4 · p. 7
  • The pseudo-label stability bound separates mean signed influence, whose positive values consume clean margin, from variance that decreases across independent pseudo constraints.Conditional centering is sufficient for zero mean first-order influence under the local model.Source: §3.4 · p. 8

4 Experiments

The experiments evaluate AC-JEPA across clean-pair training, pseudo-label self-training, controlled corruption, and design ablations. Results show accuracy gains on WildMol and characterize how training choices affect performance.

4.1 Experimental Setup

  • The Base SFT and AC-JEPA variants share MolParser-7M pretraining and Real-SFT adaptation; self-training adds pseudo-pairs from USPTO-Mol and scientific-paper structure figures to the JEPA stream.The AC-JEPA runs start from the same JEPA-warmed checkpoint but are optimized independently.Source: §4.1 · p. 8, §4.2 · p. 8

4.2 Results and Training Dynamics Across the Curriculum

  • Starting from Base SFT, joint gold-pair AC-JEPA training raised accuracy by 8.67 points on WildMol Simple and 5.80 points on Markush; each curriculum row inherits MolParser-7M pretraining and Real-SFT fine-tuning.Source: §4.2 · p. 8, Table 1 · p. 9
  • 80.23% Simple and 48.28% Markush accuracy: self-training R2 exceeded the independently initialized gold-pair configuration by 4.16 and 6.00 points, respectively.Both self-training rounds retained the SFT anchor on clean data while extending latent prediction with pseudo-pairs.Source: §4.2 · p. 8, Table 1 · p. 9

4.3 Pseudo-Label Noise Analysis

  • Recorded runs did not match total training budgets, so the pseudo-label comparison is evidence from the realized curriculum, not an isolated causal comparison between loss functions.The pseudo-labels contain naturally occurring recognition and SMILES errors.Source: §4.3 · p. 10
  • With controlled SMILES corruption, direct SFT degraded sharply as the corrupted-label fraction rose to 80%, while AC-JEPA retained gains over clean Base SFT on both benchmarks.Only selected MolGrapher labels were corrupted, by perturbing 5–15% of their characters; clean Real-SFT data remained unchanged.Source: §4.3 · p. 10, Table 3 · p. 10

4.4 Ablation Studies

  • Layer 17 was favored among four recorded decoder-layer choices, suggesting states nearer the final decoder output were less effective predictive targets.The table reports preliminary Simple accuracy for the latent-interface choices.Source: §4.4 · p. 10, Table 4 · p. 10
  • Joint image–SMILES masking at ratio 0.35 performed best, while lower or higher ratios reduced Simple accuracy; either single-modality masking still beat Base SFT.The ablation varied masking ratio and masked modality under structured image-block and SMILES-span corruption.Source: §4.4 · p. 11, Table 5 · p. 11
  • Causal predictor attention and a 256-dimensional latent slowed convergence; removing stop-gradient or pooling modality latents under SIGReg caused collapse or accuracy losses.The diagnostics support bidirectional recovery, detached clean targets, and modality-wise regulation rather than one pooled statistic.Source: §4.4 · p. 11

5 Discussion and Limitations

AC-JEPA complements SFT by retaining causal supervision while adding representation-level constraints from gold or pseudo-labeled pairs. Its noise tolerance is conditional on the tested OCSR setting and does not make self-training compute-free.

5 Discussion and Limitations

  • AC-JEPA retains the real-data token anchor while using synthetic pairs for predictive constraints; the mixed-SFT pilot was qualitative, not a systematic comparison of training budgets or sampling ratios.The pilot found mixed synthetic and Real-SFT token supervision did not improve over Real-SFT-only adaptation and was more prone to overfitting.Source: §5 · p. 11
  • Continuous latent targets can soften local pseudo-label errors, but systematic bias may shift representations; the tested noise-tolerance evidence is limited to OCSR, and extra generation and training are not compute-free.Broader visual-to-symbolic generalization requires validation on additional tasks.Source: §5 · p. 11

6 Conclusion

AC-JEPA combines bidirectional masked latent prediction with causal token supervision, improving OCSR with clean pairs and self-training on imperfect pseudo-labels.

6 Conclusion

  • Gold-pair AC-JEPA improves a Base SFT model that already shares complete synthetic MolParser-7M pretraining and Real-SFT adaptation.This supports reusing existing pairs through latent prediction.Source: §6 · p. 11
  • The approach retains real-image token supervision while assigning synthetic-pair reuse to the latent objective.The conclusion contrasts this with small-scale mixed-SFT results that did not improve over Real-SFT-only adaptation and were more prone to overfitting.Source: §6 · p. 11
  • With imperfect pseudo-labels, AC-JEPA self-training improves performance, while the recorded direct pseudo-label SFT run degrades.Source: §6 · p. 12

A.1 Structured Corruption Details

Structured corruption masks symbolic spans and image regions to create prediction positions while excluding prompt, padding, and special tokens.

A.1 Structured Corruption Details

  • Contiguous spans are sampled only within the assistant SMILES region until the prescribed masking ratio is reached.Source: §A.1 · p. 14
  • Rectangular blocks are sampled on the visual patch grid, and masks may remove local motifs or coherent image regions without matching complete chemical substructures.Source: §A.1 · p. 14
  • Prompt, padding, and special tokens are excluded, and clean and corrupted passes use the same prediction positions.Source: §A.1 · p. 14
  • If a modality has no masked positions in a micro-batch, its MSE term is skipped through a graph-valid zero.Source: §A.1 · p. 14

A.2 Target Stabilization and Gradient Routing

A detached clean-view target stabilizes latent prediction while preserving SFT and SIGReg updates through their respective paths.

A.2 Target Stabilization and Gradient Routing

  • The clean branch produces undetached Zclean, then stop-gradient forms the predictive target so MSE does not move the clean target.MSE updates the corrupted-view backbone through layer ℓ, the shared projector, and the predictor.Source: §A.2 · p. 14
  • Stop-gradient does not freeze the shared backbone: gold-pair SFT updates the full visual encoder and causal decoder, while SIGReg uses undetached Zclean.Source: §A.2 · p. 14
  • The no-detach variant is a target-stability diagnostic, since a latent space easy to predict may not remain useful for exact SMILES decoding.It is not the default in this shared-weight autoregressive generator.Source: §A.2 · p. 14

A.3 Modality-wise SIGReg Details

Modality-wise SIGReg regularizes clean image and SMILES latents with separate statistics rather than pooling the modalities together.

A.3 Modality-wise SIGReg Details

  • SIGReg compares one-dimensional random projections of the clean latent distribution with the characteristic function of a standard normal target [5].Source: §A.3 · p. 15
  • For modality m, Zm contains eligible clean-view latents in the JEPA mini-batch, and Nm is their count; random unit directions and frequency knots tk define its penalty.Source: §A.3 · p. 15
  • Image and SMILES positions are pooled across examples separately, active mask tokens are excluded, and the regularizer sums the two modality-specific penalties.Source: §A.3 · p. 15

A.4 Training-Stream Details

The section specifies the unlabeled sources and training-stream protocol, and distinguishes a qualitative mixed-SFT pilot from the controlled label-corruption experiment.

A.4 Training-Stream Details

  • The unlabeled pool for R1 and R2 combines USPTO-Mol molecular images with structure figures extracted from scientific papers.Source: §A.4 · p. 15
  • SFT and JEPA use separate mini-batches from Dsft and Dg jepa, and round 2 reinitializes from C0 rather than continuing round-1 weights.The batch sizes may differ without changing the objectives.Source: §A.4 · p. 15
  • Mixing MolParser-7M synthetic pairs with Real-SFT gave no improvement over Real-SFT-only adaptation and was more prone to overfitting in a small-scale pilot.The pilot is qualitative and does not report a numerical endpoint or matched-compute comparison.Source: §A.4 · p. 15

A.5 Supporting Analysis and Proofs for Task Readout Stability and Pseudo-Label Noise

The appendix gives conditional, local mechanisms linking latent residuals and pseudo-label perturbations to next-token stability. It also characterizes when additional modality evidence tightens a readout bound, without guaranteeing full-sequence accuracy gains.

A.5.1 From Symbolic JEPA Loss to Conditional Residual

  • Under predictor tracking, decoder–JEPA overlap, and preserved SFT margins, reducing symbolic JEPA risk tightens an averaged next-token reversal bound.The result concerns population objectives at two checkpoints, not every stochastic update.Source: §A.5.1 · pp. 16–17

A.5.2 Pseudo-label Residuals and Parameter Influence

  • Pseudo-label influence depends jointly on residual perturbations, training Jacobians, anchored curvature, and clean readout; correlated errors need not have zero bias or variance that falls as 1/n.The curvature condition is local to the analyzed parameter subspace, not global positive definiteness of the network Hessian.Source: §A.5.2 · p. 20
  • Unlike erroneous pseudo-SFT, whose wrong categorical target lowers the local correct-versus-wrong margin, pseudo-JEPA's effect depends on signed alignment and can move either way.The pseudo-JEPA effect is filtered through the latent-to-parameter pathway and clean readout direction; the comparison does not order error magnitudes.Source: §A.5.2 · p. 20

A.5.3 Multimodal Evidence and Local Readout Stability

  • Adding aligned reliable evidence can monotonically tighten the local decision bound when the mean margin is preserved and residual noise does not rise.The mechanism depends on evidence quality, not modality count alone.Source: §A.5.3 · p. 21
  • This conditional local result does not guarantee full-sequence exact-match gains; mismatched, biased, redundant, or ignored modalities may add no reliable evidence.The analysis assumes a fixed local readout and centered independent perturbations.Source: §A.5.3 · p. 21

Related papers

Loading 2610.36493v1…