PaperlayerView exact revision on arXiv

Source-linked AI summary

Explorative Modeling: Unlocking a Third Pretraining Axis and End-to-End Generation

Alexi Gladstone, Heng Ji, Yilun Du

arXiv:2607.27372v1cs.LGcs.AIcs.CLcs.CV

TL;DR

Generative models are typically trained and sampled differently because scalable methods factor generation to handle multimodal distributions. Explorative Modeling instead factors the training loop, and increasing exploration improves performance across images, video, and language while enabling end-to-end generation.

  • Problem

    Scalable generative models factor generation into steps, creating a training–inference mismatch and leaving the question of end-to-end reconstructive training unresolved.

  • Method

    Explorative Modeling generates K candidates during training and updates the model using the candidate closest to each data sample.

  • Results

    Exploration monotonically improves performance across continuous and discrete domains, with gains rising from 7% to 36% as data scales.

  • Takeaways & Limitations

    Exploration provides a third pretraining axis for existing generative models and supports end-to-end generation as a standalone paradigm.

  • Takeaways & Limitations

    Even diffusion and autoregressive models can leave modes uncaptured when individual factored predictions face multiple valid targets, with this limitation worsening as scale increases.

Abstract

from arXiv · show

The deep learning revolution, kicked off by AlexNet, taught us that end-to-end training beats decomposing a problem into hand-designed stages. Generative modeling, however, has remained the exception-despite generative models being remarkably capable, they are still not trained end-to-end. This is because, at its core, generative modeling is about handling distributions with many modes, and existing scalable approaches handle this the same way, by factoring the generation procedure, which prevents end-to-end generation. In this work, we introduce Explorative Modeling, a new paradigm that instead factors the training loop, exploring K candidate matches between model generations and data, and training on the best, so predictions commit to modes rather than blurring them. We find Explorative Models (XMs) useful in two settings. First, increasing exploration adds a third pretraining axis beyond parameters and data for existing generative models-where scaling exploration monotonically improves performance across both continuous and discrete domains (images, video, and language). Notably, gains from exploration increase with scale, climbing from 7% to 36% as data scales and from 13% to 23% as models grow, with efficiency gains more than doubling at 3x the compute. Concretely, exploration improves FLOP efficiency by 4.1x, sample efficiency by 6.2x, parameter efficiency by 47%, lifts the strongest of image-generation recipes to a near-state-of-the-art 1.43 FID on ImageNet without guidance, enables scaling how end-to-end existing models are, and unlocks scaling generalization. Second, XMs enable end-to-end reconstructive generative modeling, matching diffusion on control tasks with 16-256x fewer inference steps. Together, these results establish XMs as both a new pretraining axis for existing generative models and a standalone end-to-end generative modeling paradigm.

1 Introduction

Explorative Modeling addresses generative modeling’s failure to be end-to-end by factoring the training loop rather than generation, exploring K candidate matches and training on the closest. Exploration serves both as a scaling axis for existing generative models and as a standalone approach for end-to-end reconstructive generation.

  • Motivation: Existing autoregressive and diffusion models are trained to predict one step but sampled as recurrent processes over hundreds to thousands of steps, so errors accumulate across inference.Their generation factorization makes training targets nearly unimodal but prevents end-to-end generation.
  • Method: Explorative Modeling explores K possible matches between model generations and data, then trains on the closest to capture multimodal distributions while enabling end-to-end generation.The method factors the training loop instead of decomposing generation into smaller inference steps.
  • Scaling: Exploration adds a third pretraining axis beyond parameters and data by directly scaling generative expressivity and relieving the mode-capture bottleneck of existing models.The introduction reports gains in FLOP, parameter, and sample efficiency that increase with scale.
  • Scaling: As scale increases, models without exploration fall increasingly short of compute-optimal performance, showing that exploration acts like parameters and data under compute-optimal scaling.Increasing parameters or data alone becomes increasingly suboptimal when the other is held fixed; exploration follows the same pattern.
  • Standalone applications: End-to-end XMs match Diffusion Policy on behavior cloning and Diffuser on goal-conditioned world modeling with 16-256× fewer inference steps.They can take as little as a single forward pass instead of hundreds.

2 Background

Generative modeling is difficult because each input can correspond to many valid outputs, or modes, while reconstructive objectives can blur modes when their expressivity is too low. Existing scalable models avoid this by factoring generation into multiple steps, leaving end-to-end reconstructive generation unresolved.

  • Generative modeling and modes: Generative modeling must capture many valid outputs, or modes, rather than learn a single deterministic input-output mapping.A request such as “generate a dog” can have billions or infinitely many valid answers.
  • Generative model families: Reconstructive models map self-produced inputs to explicit data targets and include autoregressive, diffusion, and flow models.Contrastive models compare generated and true data without explicit targets but have struggled with scalability.
  • Limits of factored generation: Existing scalable models factor generation into many nearly unimodal steps, but individual predictions can still miss modes as scale increases and expressivity becomes the bottleneck.Diffusion and autoregression can leave modes uncaptured when a single prediction faces multiple valid targets.
  • Generative expressivity: Generative expressivity is a third capacity beyond parameters and data: the number of distinct modes a training objective allows a model to capture.Unlike parameters and data, this capacity is determined by the training objective itself.
  • Generative expressivity: E = 1 direct regressors produce a single blurred mean even with unlimited parameters and data.When an objective permits fewer modes than the data contains, surplus modes are averaged into predictions that match no real datapoint.
  • End-to-end generation: End-to-end generation uses the same sampling procedure during training and inference, yet scalable reconstructive models remain non-end-to-end because training and inference use different multi-step procedures.Diffusion may train on one denoising step but unroll hundreds at inference, while single-step methods still train against multi-step trajectories.

3 Explorative Modeling Approach

Explorative Modeling trains generative models by exploring multiple candidate matches and optimizing only the best one, shifting reconstruction away from off-manifold averages toward valid data modes. Forward and Reverse XM search over generations or data, respectively, producing complementary recall- and precision-oriented behaviors.

  • Core mechanism: At each training step, XM generates K candidates and backpropagates only through the candidate closest to the data under a reconstruction loss.This best-of-K procedure is implemented as a simple loop in which only the best generation receives gradients.
  • Core mechanism: Best-of-K changes the loss minimizer from the mean of data samples toward the samples themselves, avoiding blurred outputs that fall off the data manifold.The mean of multimodal data may not belong to the data manifold, whereas selecting the closest candidate favors real-looking samples.
  • Distributional interpretation: Explorative Modeling reframes generative modeling as a training-time search for the latent or coupling that best explains the data, equivalent to finding low-loss pairings.The search can be random, though gradient-based search is also possible.
  • Exploration directions: Forward XM fixes a data target, explores K generated candidates, and is mass-covering because every datapoint pulls its nearest generation.Its main weakness is compute: each candidate requires a separate generation.
  • Exploration directions: Reverse XM fixes one generated sample, searches K data targets, and is mode-seeking because each generation is pulled toward its closest data point.Searching over data makes Reverse XM cheap, since each loss calculation requires only one generation regardless of the number of targets.
  • Distributional interpretation: K controls generative expressivity: at K=1, squared-error regression fits a blurred mean, while larger K permits mixtures that capture up to K modes.Forward XM retains a maximum-likelihood interpretation at every K, but increasing K changes the density being fit.

4 Experimentation and Results

Section 4 shows that Explorative Modeling functions both as a scalable pretraining axis for existing generative models and as a standalone end-to-end generative approach. Increasing exploration improves efficiency and performance across modalities and scales, while enabling reconstructive models that match diffusion with far fewer inference evaluations.

  • 4. Experimentation and Results: Explorative Modeling combines exploration with Diffusion/Flow or Jumpy models across continuous and discrete domains, and also serves as a standalone generative approach.The experiments target both a new pretraining axis and standalone generative modeling.
  • 4.1 Pretraining Axis: 4.1: 4.1× fewer FLOPs and 6.2× less data reach the no-exploration baseline’s final image-generation performance.Adding exploration to the RAE recipe improves performance throughout training; the gains also extend to an optimally tuned SiT baseline, where FLOP efficiency improves by as much as 52%.
  • 4.1 Pretraining Axis: Increasing exploration monotonically improves image, video, and language generation performance in both continuous and discrete spaces.These modality experiments vary only the number of explored modes at fixed parameter count.
  • 4.1 Pretraining Axis: More exploration reduces overfitting on a fixed dataset, enabling extra training compute to directly buy generalization.The results establish a compute-generalization tradeoff rather than only improved training-sample efficiency.
  • 4.1 Pretraining Axis: 7% to 36% gains emerge as data scales, while 13% to 23% gains emerge as model size scales, identifying exploration as a missing scaling axis.The FLOP-optimal amount of exploration also grows as training continues, and efficiency gains more than double when compute is tripled.
  • 4.2 Standalone Generative Modeling: 4.2: More end-to-end models benefit most from exploration, making end-to-endness a scalable design property rather than a fixed choice.XJumpy models eventually outperform XDiffusion models as exploration increases, and models with fewer jumps scale better with exploration than models with more jumps.
  • 4.2 Standalone Generative Modeling: 16-256× fewer function evaluations match Diffuser, while a single network forward pass rivals Diffusion Policy instead of 100.Explorative Models match strong diffusion baselines on robotics and world-modeling tasks while shifting generative expressivity cost from inference to training exploration.

5 Discussion

The discussion presents Mode Forcing as a predictive theory: generative expressivity remains limiting, increasingly so at scale, while exploration improves diverse generative recipes and reduces reliance on generation factorization. Explorative Models also support end-to-end reconstruction, matching diffusion baselines with as little as a single forward pass instead of hundreds.

  • Mode Forcing as a Predictive Theory: Exploration improves image, video, and language generation performance across all tested recipes, addressing generative expressivity shortfalls in even the strongest models.Mode Forcing attributes uncaptured modes to generation factorization and treats reliance on guidance as evidence of limited generative expressivity.
  • Mode Forcing as a Predictive Theory: 7% to 36% gains from exploration emerge as data scales, while model scaling increases gains from 13% to 23%.The FLOP-optimal amount of exploration also rises over training, indicating that generative expressivity increasingly becomes the bottleneck at scale.
  • Mode Forcing as a Predictive Theory: As exploration grows, the optimal amount of generation factorization decreases, and more end-to-end models perform better.The discussion interprets this as exploration supplying generative expressivity that would otherwise require factoring the generation procedure.
  • Mode Forcing as a Predictive Theory: Explorative Policy and World Model match diffusion baselines with as little as a single forward pass instead of hundreds.These results confirm that exploration can handle multimodal distributions during training, enabling end-to-end reconstruction.
  • The Benefits of Surplus Generative Expressivity: XMs significantly improve video-generation performance even when the modeled distribution is not very multimodal.In this setting, Jumpy models need only 10 steps, far fewer than modern Diffusion models.

6 Future Works and Broader Impact

The paper identifies future work in extending exploration across generative models, applying end-to-end XMs, improving exploration mechanisms, scaling Reverse XMs, and using exploration beyond pretraining.

  • Exploration as a Scaling Axis for More Generative Models: Exploration may benefit additional generative models, although autoregressive LLMs and language-model evaluation remain difficult cases.The paper demonstrated exploration for Diffusion/Flow, Jumpy, and masked diffusion language models.
  • End-to-end XM Applications: End-to-end XMs are promising for inpainting, super-resolution, and feature-based world models because these settings can have fewer modes.Forward XM is described as especially practical when feature spaces contain far fewer modes than raw observations, while Reverse XM may address highly multimodal settings.
  • Improving and Understanding XMs: Discrete latent embeddings could improve exploration controllability, mode-uniformity, and mode coverage beyond random-noise candidate sampling.The paper used discrete conditioning for masked diffusion language models but not other models.
  • Scaling Reverse XMs: Reverse XMs may scale more gracefully with mode count while adding almost no extra FLOPs, and discrete conditioning can make them essentially free.The proposed mechanism loads a larger batch with K data points per condition so each generation selects its best match.
  • Exploration beyond Pretraining: Exploration could extend beyond pretraining by addressing post-training mode collapse and complementing verifier-based Forward XM approaches.Pass@k rewards and best-of-N-aware fine-tuning are interpreted as Forward XM during post-training, while exploratory pretraining may produce broader base models.

7 Limitations and Conclusion

Explorative Modeling factors the training loop to add exploration as a third pretraining axis and enable end-to-end generative modeling. Its benefits grow with scale, but integration and fully end-to-end generation remain limited by model-family compatibility, evaluation changes, guidance transfer, and multimodality costs.

  • Conclusion: Explorative Modeling factors the training loop rather than the generation procedure, increasing generative expressivity and enabling end-to-end generative modeling.It is introduced as a new paradigm for handling multimodal distributions.
  • Limitations: Exploration is easier to integrate into continuous models than other families because it requires a latent variable for searching over K candidates.Continuous models already condition on noise z, while MDLMs benefited after receiving a learned latent-variable embedding; autoregressive language models were harder to improve.
  • Limitations: Exploration changes the training objective, making distributional metrics such as FID and FVD and downstream metrics such as accuracy more important than directly comparing losses.Guidance techniques also transfer unevenly: autoguidance worked decently, whereas classifier-free guidance helped less than for base models.
  • Limitations: Fully end-to-end Forward XMs require K to grow with the number of modes, making highly multimodal distributions such as image generation currently too expensive.Reverse XMs are presented as a natural solution for handling high distribution multimodality while compute remains constrained.
  • Conclusion: 7% to 36% are the gains from exploration as data scales, while 13% to 23% are the gains as models grow.Across continuous and discrete domains, exploration acts as a third pretraining axis alongside parameters and data, with gains that grow rather than saturate.
  • Conclusion: More than 2x are the efficiency gains at 3× the compute, while exploration’s benefits continue climbing with scale rather than saturating.The reported numbers are therefore described as a floor for benefits at increased model scale.

A Additional Experimentation

Additional experiments show that XRAE converges much faster than the baseline in FID, while exploration also benefits discrete diffusion models and shows early promise for autoregressive language models. In discrete models, these benefits include improved sample efficiency and generalization, mirroring image and video trends.

  • Continuous generation: XRAE converges much faster than the baseline on FID convergence plots.Figure A.1 reports the FID versions of the convergence plots in Figure 4.
  • Discrete generation: Adding exploration to MDLMs improves both sample efficiency and generalization in the discrete domain.These benefits mirror trends observed for image and video generation.
  • Language modeling: Early evidence suggests that exploration improves the data efficiency of autoregressive language models.Detailed reporting is left to future work.

A.1 Exact Results for Exploration Scaling

Table A.1 provides the exact FID and FVD values underlying Figure 7, supporting reproducibility and future comparisons across exploration scaling. Untested exploration levels are marked with dashes.

  • Exact results: Table A.1 reports exact FID and FVD values across exploration scaling, providing the numerical results underlying Figure 7.The table is intended to ease reproduction and comparison in future work.
  • Exact results: Dashes identify exploration levels that were not tested for a given setting.

A.2 Comparing Exploration to Minibatch Optimal Transport Couplings … C Approach Details

Exploration outperforms minibatch Optimal Transport by selecting model-aligned couplings that co-adapt during training, while its implementation supports simple, efficient end-to-end generation. The approach extends to Reverse XM language models and uses flexible candidate-generation, gradient, batching, and data-search strategies.

  • A.2 Comparing Exploration to Minibatch Optimal Transport Couplings: Minibatch OT worsens FID from 46.3 to 54.5 for Small models and from 74.4 to 82.7 for Base models.The comparison uses Flow Matching models on ImageNet-1k after 200k training steps.
  • A.2 Comparing Exploration to Minibatch Optimal Transport Couplings: Exploration selects couplings using the model’s current loss, whereas minibatch OT assigns pairs by within-batch geometry alone.Minibatch OT is a biased approximation of the global coupling, while exploration’s model-aligned choice co-adapts throughout training.
  • A.3 Reverse XM Language Models: Reverse XM language models search training data with a vector database, removing matched points for the epoch to prevent repeated nearest-neighbor selections.This removal is tracked as train coverage percent and plays the role of the entropy term mentioned in the passage.
  • B Additional Intuition: End-to-end XMs generate in a single model pass, and simplest-case training is a short 3-5-line for loop.This contrasts with diffusion, flow, and autoregressive models’ many-step inference procedures and schedules.
  • B Additional Intuition: Exploration searches over noise inputs so trained noise regions act as associative-memory keys retrieving individual data modes.Larger K divides noise into finer regions, allowing modes to receive separate regions rather than blur together.
  • C Approach Details: K explored generations can be folded into the batch dimension and computed in one larger forward pass, considerably speeding training.This parallelizes candidates efficiently on accelerators, and batched Forward XM code is provided.
  • C Approach Details: Fresh candidate draws are used throughout because they are simplest, fair across domains, and less prone to collapse than caching candidates.Caching can provide nearly free candidates for repeated ImageNet classes but works worse for continuous conditioning.
  • C Approach Details: Only the best candidate receives gradients: memory-saving mode re-forwards it with gradients, while FLOP-efficient mode backpropagates only its lowest loss.Memory-saving mode preserves standard activation memory at the cost of one extra forward pass.

D Experimental Details … E.2 Coupling

The paper specifies implementation choices across image, video, behavior-cloning, world-modeling, and language experiments, while defining end-to-end generation and positioning its coupling approach against optimal transport. These details anchor comparisons across generative-modeling settings and motivate the limitations of minibatch OT coupling.

  • D Experimental Details: All image and video generation models use transformers following standard DiT size conventions.Table D.1 summarizes the model sizes for image and video generation.
  • D Experimental Details: Image experiments use class-conditional ImageNet at 256×256, with VAE latents forming 256 tokens per image and batch size 256.The setup otherwise follows SiT, except that horizontal flip augmentation is omitted.
  • D Experimental Details: Video experiments use Something-Something V2 at 128 × 128, model 10 frames conditioned on frames 0, 1, and 9, and use Base 3D transformers.The resolution and model size were chosen because higher-resolution experiments required too much compute and the goal was fair comparison rather than state-of-the-art performance.
  • D Experimental Details: Behavior-cloning runs use XM-10 for Explorative Policy and CNNs for Behavior Cloning, with Diffusion Policy results reproduced under a newer robomimic setup.The reproduced results are not perfectly comparable to the originally reported results.
  • D Experimental Details: Goal-conditioned world-model runs use XM-10 on single-task Maze2D, train for 1M steps, and evaluate with a goal radius of 0.45.The setup reproduces Diffuser results for fair comparison; recurrent blocks are required for end-to-end models’ NFE to exceed one.
  • E.1 End-to-End Generative Modeling: End-to-end generation means training-time sampling, when present, is identical to inference-time sampling; the paper focuses primarily on reconstructive models.Contrastive generative models are already end-to-end but have struggled with scaling relative to reconstructive models.
  • E.2 Coupling: Optimal Transport reduces path crossings geometrically, but minibatch OT is a biased proxy for global coupling whose gains narrow at scale.The authors report that their own OT experiments instead produced worse performance, which they attribute to minibatch bias.

E.3 Explorative Modeling Based Methods · F Additional Theory

Explorative Modeling builds on the recurring best-of-K idea while claiming its significance for generative modeling, and structurally generalizes IMLE. Related methods differ in purpose, including oracle selection for ensembles rather than generative modeling.

  • E.3 Explorative Modeling Based Methods: Explorative Modeling does not claim to invent best-of-K, which has appeared in winner-take-all objectives and inference-time candidate selection.Inference-time candidates may be reranked with a learned reward model, trained verifier, or the model’s own energy.
  • E.3 Explorative Modeling Based Methods: Best-of-K connects Explorative Modeling to earlier winner-take-all training objectives and inference-time best-of-N selection.The supplied passage identifies learned reward models, trained verifiers, and model energies as reranking mechanisms.
  • E.3 Explorative Modeling Based Methods: Multiple Choice Learning trains K predictors under an oracle loss, routing gradients through the lowest-error member for each example.The approach was later extended to single networks with multiple prediction heads.
  • E.3 Explorative Modeling Based Methods: Multiple Choice Learning targets ensemble diversity for downstream oracle selection rather than generative modeling.It shares the structural idea of backpropagating only through the minimum-loss prediction.
  • E.3 Explorative Modeling Based Methods: IMLE draws a pool of model samples, matches each data point to its nearest sample, and trains on those pairs.It uses fast approximate nearest-neighbor search and has theoretical guarantees of recovering MLE under mild conditions.
  • E.3 Explorative Modeling Based Methods: Structurally, Explorative Models generalize IMLE by replacing its shared global sample pool with per-step candidates in end-to-end Forward XM.Both methods minimize the expected distance from each data point to its nearest model sample, according to the supplied passage.

F.1 Generative Expressivity Details

Generative expressivity E measures the mode capacity permitted by a training objective, rather than by a particular dataset. The analysis shows that direct regression collapses to E = 1, while factored objectives can retain broad at-optimum support yet remain limited per prediction.

  • Definition and scope: E counts the strict local maxima of a distribution, or support cardinality for discrete distributions, under the model’s inference-time sampling distribution.The outer supremum makes E an objective-level property, while the inner supremum interprets it as a capacity ceiling.
  • Definition and scope: E = 1 under direct Bregman regression, regardless of parameter count, because its unique minimizer is the conditional mean, a point mass.This is an exact at-optimum result for direct regression.
  • Factored objectives: V^L modes are attainable for autoregression over L tokens with vocabulary size V, while continuous diffusion is unbounded at the loss scale.The passage notes that practical trained models realize far fewer modes, enough for the capacity to remain binding.
  • Factored objectives: Even a two-step sampler can retain arbitrarily many well-separated modes, but without controlling their mass, whereas Proposition 3 guarantees coverage of every mode.Finite sampling steps therefore do not restore a meaningful expressivity cap.
  • Per-Prediction Expressivity: Per-prediction expressivity measures how much target structure one step’s minimizer retains; an MSE step’s ceiling is one mode, while parallel discrete decoding preserves independent per-position conditionals.Factored models are trained as many small reconstruction problems, each with its own ceiling.

F.2 What Forward and Reverse XM Optimize

Forward XM is maximum likelihood over a K-candidate mixture and approaches forward KL as K grows, while Reverse XM targets reverse KL only after entropy correction; without it, Reverse XM collapses. The hard-min implementation instead acts primarily as a coverage objective, while sufficient exploration lets smooth Forward XM represent separated modes.

  • Forward XM: Forward XM performs maximum likelihood at every K over the mixture of the model’s K explored generations, converging to KL(p∗∥pθ) as K →∞.At K=1 it fits a single Gaussian, while larger K permits a multimodal mixture; the optimum matches p∗ exactly only as σ →0 in general.
  • Hard-min XM: The hard min differs from the soft objective by at most log K and becomes a coverage objective as K →∞, rewarding support coverage without fixing mass allocation.This gives it a mode-covering character like forward KL, but it does not determine how probability mass is distributed across the covered support.
  • Reverse XM: Reverse XM minimizes KL(gθ ∥ˆpσ) + H(gθ) at every K and collapses toward a point mass without an entropy bonus.At K=1 the collapse point is the data mean; as K grows, the entropy-corrected objective approaches pure reverse KL and is minimized at gθ = p∗.
  • Objective asymmetry: Forward XM compares raw data with the blurred model, whereas Reverse XM compares the raw model with blurred data; both become opposite-direction raw comparisons as σ →0.Only Forward XM provides a genuine evidence bound; entropy-corrected Reverse XM is variational rather than an ELBO.
  • Mode expressivity: Once K sufficiently outnumbers M ∗ separated modes, smooth Forward XM places nearly all mass at the modes and produces exactly M ∗ local maxima at the loss scale.The stated regime requires pairwise separation ∆≫σ and K(1 −minm wm)K ∆2/σ2 ≲1.

G Frequently Asked Questions

The FAQ clarifies that Explorative Modeling’s novelty is a training-loop factorization that increases generative expressivity without factoring generation. Exploration occurs only during training, improves compute efficiency, helps when latent-variable search addresses expressivity bottlenecks, and avoids collapse through real-data targets or entropy regularization.

  • Novelty: Explorative Modeling’s novelty is recognizing that best-of-K increases generative expressivity without factoring generation.Best-of-K itself is not presented as the central contribution because sampling candidates and keeping the best has appeared previously.
  • Compute and inference: Inference is unchanged because exploration happens entirely during training, although Forward XM incurs added training cost.For Reverse XM, the added training cost is often negligible; the FAQ states the additional cost is worthwhile.
  • Compute and inference: FLOP-matched comparisons find exploring models significantly more compute efficient than models that train longer, with FLOP-optimal exploration growing with scale.The FAQ frames exploration as a scaling axis whose value is judged by compute efficiency, analogous to allocating compute between parameters and data.
  • When exploration helps: Exploration helps when generation is conditioned on a searchable latent variable and helps most when generative expressivity is a bottleneck.Video models also improved significantly despite their generated distribution not being very multimodal.
  • Collapse: Training on model generations does not cause collapse because real data remain the targets, while Reverse XM requires an entropy term because it is mode-seeking.In Forward XM, every datapoint receives a training signal and no part of the data distribution can be dropped; the FAQ identifies Forward XM as maximum likelihood.
Loading 2607.27372v1…