PaperlayerView exact revision on arXiv

Source-linked AI summary

There and Back Again: Bidirectional Diffusion Bridges for Multimodality Translation

Gabe Guo, Elon Litman, Thanawat Sornwanee, Jose Blanchet, Stefano Ermon

arXiv:2608.27885v1cs.LG

TL;DR

Existing multimodality translation methods can be source-unaware and unidirectional, limiting sampling flexibility and preventing inversion. BIT uses bidirectional diffusion bridges between text and images, and is competitive with diffusion and flow baselines while outperforming them on several evaluated tasks.

  • Problem

    Existing approaches may follow paths that do not directly represent the source modality and may be unidirectional, limiting sampling flexibility and preventing inversion between modalities.

  • Method

    BIT is a unified multimodal model that uses a simulatable bridge SDE from text to images and its analytic time reversal from images to captions.

  • Results

    BIT is competitive with noise-to-data diffusion on image-text generation, outperforms it on evaluated data-variation tasks, and achieves the best average rank among evaluated cell-fate models.

  • Takeaways & Limitations

    BIT provides a unified framework for text-to-image generation and image-to-text inversion, with competitive or stronger results across vision-language and scientific evaluations.

  • Takeaways & Limitations

    Theorem 3 applies to state-only round-trip settings and gives no ordering when the text prompt is retained as side information throughout sampling.

Abstract

from arXiv · show

Multimodality translation (e.g., text-to-image) is a core generative AI task. However, existing approaches (1) follow generative paths that do not directly represent the source modality, limiting the flexibility of some sampling algorithms; and (2) are unidirectional, preventing inversion (e.g., image-to-text). We propose BIT: Bidirectional Image-Text Diffusion Bridges. In contrast to previous approaches, BIT starts directly from text and interpolates into images, providing (1) a source-aware generative path that enables diverse and flexible sampling algorithms; and (2) an endpoint-conditioned process that can be traversed from image to text, providing a unified, bidirectional generative framework. BIT is derived through stochastic calculus, yielding SDE forms amenable to simulation and tractable loss functions that scale to high dimensions. Our experiments show that BIT is competitive with denoising-diffusion and deterministic-flow baselines, and outperforms them on several vision--language and natural-science evaluations.

1 INTRODUCTION

Multimodality translation requires models that represent relationships between modalities while supporting stochastic, semantically meaningful paths. BIT addresses limitations of noise-conditioned and unidirectional approaches with a bidirectional diffusion-bridge framework.

  • Motivation: Existing diffusion T2I methods interpolate between images and modality-independent Gaussian noise, while text enters only as drift conditioning.Intermediate states therefore do not resemble the actual text prompts.
  • Motivation: Noise-based trajectories can lose reference information during corruption, limiting algorithms that traverse and reverse paths for semantically related variations.Theorem 3 provides a local comparison motivating data-to-data bridges for source-information retention.
  • Motivation: Unidirectional T2I models cannot reverse their trajectories to produce captions, requiring separate vision-language models for image-to-text translation.This limits cross-modality sampling possibilities.
  • BIT: BIT defines a unified model that generates images from text and captions from images using bidirectional image-text diffusion bridges.The model is designed to capture multimodal translation through a stochastic process rather than a deterministic mapping.
  • BIT: BIT starts from one modality and gradually perturbs it toward paired observations in the other modality, with forward and analytically time-reversed SDEs.Girsanov’s theorem and Doob’s h-transform yield simulatable SDEs and neural-network drift objectives.
  • Results: BIT’s empirical validation finds competitiveness or improvement over noise-to-data baselines on I2T and T2I, stronger performance on evaluated data-variation tasks, and the best average rank among evaluated cell-fate models.The paper evaluates T2I, I2T, and scientific domains including cell-fate modeling.

2 PRELIMINARIES

Diffusion models transform noise into data, whereas diffusion bridges interpolate directly between data endpoints. BIT uses semantically richer paths while representing text continuously for the shared SDE framework.

  • Diffusion Models: Standard diffusion models use an SDE to transform Gaussian noise into data over reverse time.The score function is typically approximated by a neural network.
  • Diffusion Bridge Models: Diffusion bridges instead construct SDEs that interpolate directly from one data endpoint to another.Their chief stated advantage is that intermediate paths are semantically rich.
  • BIT Schematic: BIT’s schematic uses forward- and reverse-time SDEs with neural-network drift to traverse the same text-image generative path.Text begins as invertible token embeddings with zero-padding.

3 PROBABILISTIC MODELING FRAMEWORK

BIT constructs a stochastic bridge with the paired data distribution as its endpoint law, derives forward and reverse simulatable SDEs, and trains their drifts with tractable objectives. It also provides a heuristic bridge analogue of classifier-free guidance.

  • Framework: The target process has endpoint law (X0, X1) ∼pdata, with text and image representations occupying the two endpoints, and must be simulatable in both directions.The construction treats pdata as the joint law of paired endpoints.
  • Framework: BIT starts from a data-independent Brownian base process and changes its path measure so the resulting bridge has the desired joint endpoint law.Conditional paths retain the base Gaussian-bridge kernels, with the terminal endpoint obtained by continuous extension.
  • Forward-Time Model: The forward-time SDE translates X0 to X1 using an h-transform drift estimated by a neural network and a weighted-L2 training objective.The SDE is defined on finite horizons and reaches its endpoint through continuous extension.
  • Caveat: State-only weak solutions can match one-time marginals without identifying transition kernels or guaranteeing conditional round-trip fidelity.The paper evaluates conditional round-trip behavior empirically rather than deriving that guarantee.
  • Reverse-Time Model: The reverse-time SDE is the dual construction, translating images at X1 to text at X0 with an analogous learned drift and objective.Its process also has an almost-sure continuous extension at the terminal reverse time.
  • Classifier-Free Guidance: BIT applies endpoint-conditioned and unconditioned drift extrapolation as a classifier-free-guidance-inspired sampling heuristic.The guidance strength is ω ≥0, but the paper does not claim an analytically identified guided distribution.

4 FROM DISCRETE TO CONTINUOUS TOKEN EMBEDDINGS

BIT converts discrete text into continuous, nearly invertible embeddings so text and image representations can share the SDE framework. Independent token lookup embeddings simplify decoding and dimensionality matching.

  • Requirements: Text embeddings must be nearly invertible and match image dimensionality for the continuous SDE framework.These requirements ensure distinguishable outputs while allowing both modalities to occupy a common space.
  • Construction: BIT embeds each token independently in a fixed lookup table rather than using context-dependent hidden states.This gives each vocabulary token a single representation and simplifies the learning problem.
  • Instantiation: Qwen3-Embedding-8B supplies token embeddings, while a small MLP decodes embeddings into tokens with greater than 99% accuracy, including under noisy embeddings.The image latent size is 4×32×32 = 4096, matched by 64 tokens of 64 dimensions each.

5 COMPARISON TO OTHER MODEL CLASSES

The comparison argues that data-to-data bridges can retain more source information locally than noise-to-data bridges, while SDEs better accommodate multimodal cross-modal distributions than deterministic ODEs. The theorem’s advantage applies to state-only trajectories and extends fully across time in a specified Gaussian setting.

  • Noise-to-Data: Theorem 3 establishes a local mutual-information advantage for the text-to-image data-to-data bridge over the noise-to-image bridge.Under the stated assumptions, there exists τ > 0 such that the data-to-data trajectory retains more information about the target for t ∈ [0, τ).
  • Noise-to-Data: The theorem compares evolving states that must retain source information, not conditional samplers that preserve the text prompt as side information.With retained side information, the theorem gives no ordering; its state-only comparison targets round-trip settings without a suitable auxiliary condition supplied beforehand.
  • Noise-to-Data: For positively correlated jointly Gaussian scalar endpoints under BIT’s interpolation coefficients, strict dominance holds for every t ∈ [0, 1), whereas the extent of τ is empirical for arbitrary distributions.The same theorem applies to image-to-text after flipping time.
  • ODE vs. SDE: Deterministic ODE bridges are unsuitable for multi-peaked cross-modal distributions, while noise-injecting SDEs can model multiple possible outputs.A text prompt may correspond to a range of images, motivating stochastic rather than deterministic distribution-to-distribution modeling.

6 EXPERIMENTS

Experiments compare BIT with noise-to-data diffusion and deterministic flow matching across multimodal generation, stochastic variation, and cell-fate modeling. BIT is competitive in generation, retains fidelity better at higher corruption, and achieves the best average rank among evaluated cell-fate models.

  • Experimental setup: BIT is compared with noise-to-data diffusion and data-to-data flow matching to isolate endpoint choice and transport stochasticity.The controlled baselines use consistent settings and a fixed SDE volatility schedule.
  • Conditional generation: BIT performs competitively with noise-to-data diffusion for text-to-image and image-to-text generation.For text-to-image, BIT is very close on FID and CLIP; for image-to-text, it leads generative perplexity and ranks second on CLIP.
  • Conditional generation: Deterministic data-to-data flow matching performs worst on every metric besides image-to-text CLIP.The paper attributes this to the inability of a bare deterministic source-to-target flow to represent non-degenerate conditional distributions.
  • Cross-modal variation: BIT’s fidelity curve degrades more gracefully than noise-to-data diffusion at higher corruption fractions in both translation directions.Greater corruption increases restoration diversity, with BIT trading off fidelity and diversity while noise-to-data diffusion discards source-state information.
  • Scale and engineering: Classifier-free guidance improves BIT performance for both text and image generation, while larger scale and additional engineering further improve results.The large-scale model uses 1,054,621,840 parameters and is trained jointly in both directions.
  • Scientific domain: The cosine-volatility bridge achieves the best average rank among the evaluated cell-fate models, with an average rank of 2.67.It leads fate kNN accuracy at 0.445, while other bridges lead clone kNN and distributional MMD; the single-run ranks do not establish statistical superiority.

7 RELATED WORKS, DISCUSSION, CONCLUSION

The paper introduces BIT as a bidirectional image-text diffusion bridge that challenges noise-to-data T2I generation. Its endpoint-conditioned construction supports both directions, while controlled experiments find text-to-image interpolation competitive with noise-to-data baselines and better on several downstream tasks.

  • 7 RELATED WORKS, DISCUSSION, CONCLUSION: BIT challenges the traditional assumption that text-to-image generation must start from Gaussian noise.
  • 7 RELATED WORKS, DISCUSSION, CONCLUSION: Its endpoint-conditioned process supports generation from text to images and images to text within one framework.
  • 7 RELATED WORKS, DISCUSSION, CONCLUSION: BIT’s source-aware path enables flexible round-trip sampling without requiring the original endpoint as side information.
  • A CONDITIONAL STOCHASTIC PROCESS DERIVATION: The bridge construction starts from one endpoint, uses finite-horizon h-transform and Girsanov arguments, and obtains a simulatable SDE with a neural-network drift.
  • A CONDITIONAL STOCHASTIC PROCESS DERIVATION: The theoretical derivation does not assert absolute continuity at the terminal time, and singular endpoint laws can invalidate that property.
  • A CONDITIONAL STOCHASTIC PROCESS DERIVATION: The SDE has a continuous extension to the terminal time whose endpoint law is the target distribution.

A.1 PROOF OF FORWARD PROCESS SDE AND OBJECTIVE

The forward-process proof establishes a weighted square-loss objective whose oracle minimizer supplies the canonical drift for BIT’s SDE. The reverse process follows by swapping endpoints and reversing time, with corresponding realizability and version-selection qualifications.

  • A.1 PROOF OF FORWARD PROCESS SDE AND OBJECTIVE: Endpoint behavior is controlled by the weight function, whose integrability makes the weighted time integrals of both score-target moments finite.
  • A.1 PROOF OF FORWARD PROCESS SDE AND OBJECTIVE: The weighted score targets belong to their required weighted L2 spaces under Assumptions 1 and 2.
  • A.1 PROOF OF FORWARD PROCESS SDE AND OBJECTIVE: The Pythagorean identity shows that the conditional-expectation oracle is the unrestricted minimizer of the forward square-loss objective.
  • A.1 PROOF OF FORWARD PROCESS SDE AND OBJECTIVE: Under realizability, every exact parameterized minimizer agrees with the oracle under the weighted training-input law.
  • A.1 PROOF OF FORWARD PROCESS SDE AND OBJECTIVE: The population theorem is an oracle statement; practical training can incur estimation, optimization, and approximation error.
  • A.1 PROOF OF FORWARD PROCESS SDE AND OBJECTIVE: The reverse objective is obtained by time reversal and endpoint swapping, yielding the corresponding reverse oracle and SDE.

B MARGINAL SDES

The marginal-SDE results project endpoint-conditioned drifts onto state-only coefficients while preserving the bridge’s one-time marginals. These are existential weak-solution results, not uniqueness or universal claims about all solutions.

  • B MARGINAL SDES: The projection uses conditional-expectation minimizers and fixed jointly Borel versions as the SDE coefficients.
  • B MARGINAL SDES: The forward projected process starts from the text marginal and ends at the image marginal through continuous path extension.
  • B MARGINAL SDES: The superposition construction establishes existence of selected weak solutions but asserts neither uniqueness in law nor a Markov property.
  • B MARGINAL SDES: A state-only projected SDE admits a weak solution with the same one-time marginals as the canonical endpoint-conditioned process.
  • B MARGINAL SDES: Because the state-only coefficient does not retain the starting text state, the construction motivates sampling and restoration schemes that do not require endpoint knowledge.
  • B MARGINAL SDES: The reverse projected SDE analogously starts from the image marginal and has the text marginal at its terminal endpoint.

D PROOF OF LOCAL MUTUAL INFORMATION ADVANTAGE

The paper proves a local mutual-information advantage for data-to-data bridges over noise-to-data bridges under continuity assumptions, and derives an exact dominance criterion in a tractable Gaussian case. The practical scope is limited because the local interval has no general lower bound and the Gaussian analysis may not match real text-image endpoints.

  • D PROOF OF LOCAL MUTUAL INFORMATION ADVANTAGE: The data-to-data bridge initially contains more mutual information about the target than the noise-to-data bridge.
  • D PROOF OF LOCAL MUTUAL INFORMATION ADVANTAGE: The same local advantage holds for the image-to-text direction under corresponding endpoint and continuity assumptions.
  • D PROOF OF LOCAL MUTUAL INFORMATION ADVANTAGE: The theorem guarantees positivity only on some interval near the initial endpoint and gives no lower bound on that interval’s length.
  • E.1 BACKGROUND: MUTUAL INFORMATION BETWEEN JOINTLY GAUSSIAN VARIABLES: The Gaussian special case is mainly interpretive because real text-image endpoints are unlikely to be mutually Gaussian.
  • E.1 BACKGROUND: MUTUAL INFORMATION BETWEEN JOINTLY GAUSSIAN VARIABLES: For jointly Gaussian scalar endpoints, mutual-information comparisons reduce to comparisons of squared correlations.
  • E.1 BACKGROUND: MUTUAL INFORMATION BETWEEN JOINTLY GAUSSIAN VARIABLES: The Gaussian analysis yields an exact criterion for when the text-to-image bridge dominates the noise-to-image bridge.

E.4 THE POSITIVELY COUPLED REGIME

Under positive endpoint coupling, the paper’s scalar Gaussian analysis establishes full-interval mutual-information dominance for the data-to-data bridge, while explicitly bounding the result’s scope.

  • Positive coupling: Under the stated coefficient and coupling conditions, strict mutual-information dominance holds throughout t ∈ [0, 1).At t = 0, the text endpoint retains positive information about the target while the independent noise endpoint retains none.
  • Assumptions: The coefficient conditions hold for BIT, but positive endpoint correlation is an additional assumption not guaranteed for every empirical representation coordinate.The result applies when scalar text–image endpoint representations are positively correlated.
  • Scope: The analysis uses scalar jointly Gaussian endpoints, so it captures linear dependence rather than the full dependence structure of real text–image pairs.The construction demonstrates that the theorem’s bound can be tight in this idealized setting, not necessarily for the empirical distribution.
  • Scope: The closed forms describe an idealized bridge rather than the learned drift, and do not establish a full-interval theorem for general vector-valued endpoints.These caveats limit direct interpretation of the theoretical result for the learned high-dimensional model.

G TRAINING DETAILS

The experiments use matched transformer capacity and shared training settings across bridge, diffusion, and flow comparisons, while numerical endpoint handling remains approximate.

  • Optimization: All models train for 200,000 steps with global batch size 512 and matched optimizer settings, including AdamW learning rate 1.5 × 10^-4.Score models sample time from a truncated interval, whereas the flow model samples from [0, 1].
  • Score training: Score models use a periodic SDE whose volatility peaks centrally and is smallest at data endpoints, with 30% unconditional and 70% conditional loss usage.All models also use the same image-REPA objective with weight 0.5 at layer 8.
  • Numerical caveat: Finite Euler–Maruyama sampling evaluates score networks at boundary times outside training and does not claim exact endpoint laws for discretized learned samplers.Reported outputs are numerical approximations of the continuous process.
  • Architecture: The controlled comparisons use the same DiTXA-L/2 backbone and approximately equal capacity, with 716,089,872 parameters and 256 transformer tokens.Both image and reshaped text endpoints have shape 4 × 32 × 32.
  • Directions: Separate direction-specific score models are trained for text-to-image and image-to-text because this implementation is more parameter- and compute-efficient.Mathematically, they represent opposite directions of the same endpoint-conditioned transport process.
  • Text decoding: The continuous text endpoint is decoded into discrete tokens with a shared position-independent MLP trained using token-level cross-entropy.The decoder processes recovered 64-dimensional token vectors and uses the vocabulary embedding table.

I DATASET

The dataset and evaluation pipeline support bidirectional generation, stochastic variation, image editing, and image-to-text analysis using paired image–caption data and shared endpoint representations.

  • Dataset: The filtered GPIC training set contains approximately 99 million valid image–caption pairs, with 0.5% reserved for validation.Images are resized and center-cropped to 256 × 256 pixels, and captions are required to be nonempty.
  • Evaluation: Evaluation uses 50K text-to-image samples, 10K image-captioning samples, and 500 samples for stochastic variation experiments.The held-out GPIC testing split is used throughout these evaluations.
  • Metrics: FID, Qwen3-based generative perplexity, and CLIP score measure image quality, text quality, and prompt adherence for cross-modal generation.These metrics are reported for both text-to-image and image-to-text settings where applicable.
  • Variation metrics: Semantic fidelity is measured by source-to-output embedding cosine similarity, while diversity is measured by pairwise similarity among outputs sharing a source.Image embeddings use DINOv2-L and text embeddings use Qwen3-Embedding 8B.
  • Image-to-text inversion: The bridge’s reverse image-to-text output is decoded into a pseudocaption and an on-manifold text endpoint before semantic editing.A token-norm stopping criterion identifies sequence length and padding positions.
  • Scale: A larger model trained with more compute improves overall performance relative to the controlled-scale results.The large-scale model uses a 1.05B-parameter backbone and jointly trains both translation directions.
  • Image editing: Image editing reverses an image into text, rewrites a small number of words, and transports the edited text forward using shared implied noise.Token-wise spherical interpolation creates a spectrum of structurally aligned edits without using the dataset caption.
  • REPA: REPA improves image-to-text CLIP score substantially at the cost of a slight increase in generative perplexity, consistent with stronger semantic alignment but less fine syntax.The comparison uses the same training and inference settings except for REPA.

T QUALITATIVE CROSS-MODAL ROUND-TRIP STOCHASTIC VARIATION RESULTS

BIT’s text endpoint supports semantically related image variation after corruption and enables bidirectional text–image translation, unlike unidirectional or deterministic alternatives.

  • Image stochastic variation: BIT can recover semantically related image variations by reversing corruption because its text endpoint retains source-related information.These are qualitative results corresponding to Figure 3.
  • Text stochastic variation: At 90% corruption, BIT generates semantically related captions while the diffusion baseline goes off-topic.This qualitative comparison is reported for text stochastic variation.
  • Bridge motivation: Diffusion bridges interpolate directly between data endpoints, whereas conventional diffusion paths connect data to source-independent Gaussian noise.The latter can lose source information during corruption, limiting semantically related variation.
  • Bidirectionality: BIT provides bidirectional sampling, while earlier diffusion-bridge formulations described in the paper are unidirectional or omit reverse-time and marginal transitions.This distinction supports image-to-text inversion and broader cross-modality sampling procedures.
  • Stochasticity: Bare deterministic flow models cannot represent non-degenerate conditional distributions without auxiliary randomness.The paper contrasts this limitation with stochastic bridge models and related methods that inject noise through additional components.
Loading 2608.27885v1…