PaperlayerView exact revision on arXiv

Source-linked AI summary

A Language-Informed Loss for Visual Generation

Suchir Agarwal, Keshigeyan Chandrasegaran, Ziyu Chen, Kyle Sargent, Juan Carlos Niebles, Li Fei-Fei, Jiajun Wu

arXiv:2610.23653v1cs.CVcs.LG

TL;DR

Pixel-space generation avoids learned image tokenizers, but pixel loss treats equal-sized errors alike even when their semantic effects differ. The paper introduces LILIE, an invertible caption-trained image encoder used to add language-informed supervision, and reports lower JiT-T2I-g gFD on GPIC with Pixel+LILIE loss.

  • Problem

    Pixel loss treats errors of equal pixel distance alike despite differences in semantic content, motivating study of a richer language-informed training signal.

    Source: §1 · p. 2

  • Method

    LILIE is an invertible image encoder trained for caption recovery from Gaussian-noised features and then used to define a language-informed loss for text-to-image generation.

    Source: §1 · p. 2

  • Results

    Pixel+LILIE lowers JiT-T2I-g gFD from 66.84 to 53.01 on GPIC.

    Source: §5 · p. 10

  • Takeaways & Limitations

    A 32M-parameter invertible encoder supplies a language-informed loss that improves JiT-T2I-g generation quality in the reported GPIC setting.

    Source: §5 · p. 10

    Experiments use GPIC at 256 × 256 with JiT-T2I and only the SmolVLM-256M backbone; combinations with perceptual losses were not tested.

    Source: E LIMITATIONS · p. 22

Abstract

from arXiv · show

End-to-end visual generation with pixel-space loss has recently achieved competitive performance without the need for learned image tokenizers. In this work, we study whether language-informed supervision can provide a richer training signal than pixel loss for visual generation. We introduce LILIE (pronounced "lily"), a Language-Informed and Lossless Image Encoder. LILIE is an invertible image transform trained such that a vision-language model can recover captions from noised features. The encoder uses a normalizing flow transformer trained jointly with a vision-language model via next token prediction. We then use the encoder to extract a simple language-informed loss for text-to-image generation. We train JiT models with LILIE loss, reducing gFD by 14 points (66.84 $\to$ 53.01) against a pixel-only baseline on the GPIC dataset. Our code, encoders, and generative models are available at https://suchiragarwal.com/lilie.

1 INTRODUCTION

The section asks whether language-shaped visual distances can provide richer supervision than pixel loss and introduces LILIE as an invertible, caption-trained encoder. On GPIC, combining LILIE and pixel losses lowers JiT-T2I-g gFD from 66.84 to 53.01.

1 INTRODUCTION

  • Equal pixel distances can correspond to different semantic damage, motivating a language-informed visual distance for generation supervision.The section contrasts a perturbation degrading a subject with one altering a background cloud; both have pixel distance 1000.Source: §1 · p. 2, Figure 2 · p. 2
  • LILIE is an invertible image transform trained so a vision-language model can caption its Gaussian-noised features, preserving visual information while learning caption-relevant features.The VLM and encoder are jointly trained with next-token prediction, then the VLM is discarded and LILIE is used as a standalone transform.Source: §1 · p. 2, Figure 3 · p. 2
  • Combining LILIE and pixel losses lowers JiT-T2I-g gFD from 66.84 to 53.01, versus 66.84 for the pixel-only baseline.Replacing pixel loss entirely with LILIE loss yields gFD 55.78.Source: §1 · pp. 2–3
  • The study also pretrains perceptual-loss generators on GPIC to provide baseline references for evaluating LILIE supervision.The listed perceptual losses are LPIPS, DINOv2, and SigLIP 2.Source: §1 · p. 3

2 RELATED WORK

Related work spans pixel-space generation, feature-space supervision, invertible representations, and language-supervised visual features. LILIE combines caption supervision and an invertible representation while keeping generation in pixel space.

Pixel-space diffusion

  • Pixel-space generators avoid learned image tokenizers; JiT predicts clean images directly from noisy pixels, the generator family used in this work (Li & He, 2026).Other cited approaches include cascaded diffusion and flow matching, neural fields, and patch- and pixel-level processing.Source: §2 · p. 3

Feature-space losses for generation

  • Feature-space losses extend pixel-wise supervision with neural features, while LILIE learns a language-supervised, lossless representation for the loss and retains pixel-space generation.Prior examples include LPIPS and DINOv2 augmentation, denoiser-feature alignment, and pretrained-feature latent spaces.Source: §2 · p. 3

Invertible networks

  • Invertible networks support exact likelihood evaluation and sampling, representation learning, and information-bottleneck objectives; LILIE trains a bijection through caption prediction from corrupted features.LILIE uses the improved TARFlow architecture introduced in BiFlow.Source: §2 · p. 3

Language-supervised visual representations

  • Language-supervised visual representations include contrastive image-text learning and caption-prediction methods; LILIE uses captions as targets for corrupted image features and trains only with caption cross-entropy.Related work also uses language to inform discrete image tokenization and caption-conditioned image encoding and decoding.Source: §2 · p. 3

3 METHODOLOGY

LILIE learns an equally shaped, invertible image representation under language supervision, then uses that frozen representation to augment pixel-space text-to-image training.

3.1 LILIE: A LANGUAGE-INFORMED AND LOSSLESS IMAGE ENCODER

  • Image-caption training uses a noisy, RMS-scaled LILIE representation and a LoRA-finetuned VLM captioning objective to jointly update the encoder and VLM adapters.After training, the VLM is discarded and the encoder and second-moment estimate are retained.Source: §3.1 · p. 4

3.2 TRAINING A T2I GENERATIVE MODEL WITH LILIE

  • An EMA estimate of the feature second moment supplies fixed rescaling for the LILIE-space transform while preserving its bijectivity.The encoder maps images to equally shaped features, and its invertibility prevents representations from discarding visual information.Source: §3 · p. 4, §3.2 · p. 5
  • For text-to-image training, the frozen encoder scores predictions in LILIE-space, with objectives using either LILIE-space supervision alone or combined pixel and LILIE-space losses.The method adds squared error in LILIE-space to the standard pixel-space flow-matching loss.Source: §3 · p. 4, §3.2 · p. 5

4 EXPERIMENTS

The experiments evaluate LILIE losses on GPIC JiT text-to-image models, finding that Pixel+LILIE improves generation metrics over pixel-only training. Ablations test standalone loss use, caption supervision, sampling steps, and encoder scaling.

4.1 EXPERIMENTAL SETUP

  • The study trains encoder and generator models on GPIC at 256 × 256 resolution, selecting noise level σ = 0.25 and loss weight λ = 2 for the main results and ablations.Main results use one epoch of GPIC Full; ablations use 200K generator steps.Source: §4.1 · pp. 5–7

4.2 GENERATION RESULTS

  • Adding LILIE to Pixel lowers gFD from 80.24 to 65.86 for JiT-T2I-XL (14.38 points) and from 66.84 to 53.01 for JiT-T2I-g (13.83 points).Source: §4.2 · p. 7
  • Pixel+LILIE outperforms Pixel at every tested short, medium, and long prompt length on GPIC Test.Figure 5 reports gFD for three sets of 50K prompts, with each model at its optimal CFG scale.Source: Figure 5 · p. 7
  • LILIE-only reduces gFD from 80.24 to 68.88 for JiT-T2I-XL and from 66.84 to 55.78 for JiT-T2I-g; combining it with Pixel yields the best quality.Unlike LILIE, DINOv2, SigLIP 2, and LPIPS losses fail when used without a Pixel loss term.Source: §4.2 · p. 7
  • With random-token captions, JiT-T2I-XL reaches gFD 82.06, versus 80.24 for Pixel and 65.86 with real-caption Pixel+LILIE training.The randomized captions preserve lengths and prompts, while the downstream generator still receives original image-caption pairs.Source: §4.2 · p. 7, Figure 6 · p. 7

4.3 SAMPLING STEPS

  • At 25 Euler steps, Pixel+LILIE reaches gFD 58.23, below Pixel’s 68.67 at 50 steps; Pixel+LILIE has lower gFD at every tested sampling budget.Beyond 50 steps, gains are small, and LILIE alone worsens from gFD 55.78 to 56.63 when increasing from 50 to 75 steps.Source: §4.3 · p. 8, Figure 7 · p. 8

4.4 ENCODER SIZE AND DATA SCALING

  • For JiT-T2I-g, gFD falls from 84.66 with LILIE-µ to 80.87 with LILIE-T and 76.59 with LILIE-S; even the 1.5M-parameter LILIE-µ beats Pixel’s 101.66.All encoders in this model-size ablation train on 100M images; the default LILIE-S setting is highlighted in Table 2.Source: §4.4 · p. 9, Table 2 · p. 9
  • Increasing LILIE-S training from 10M to 100M images lowers JiT-T2I-XL gFD from 97.79 to 91.79 and JiT-T2I-g gFD from 83.01 to 76.59.These generator comparisons use a fixed 200K-step budget; increasing encoder data and training budget does not increase the generator’s training budget.Source: §4.4 · p. 9, Table 3 · p. 9

5 DISCUSSION AND CONCLUSION

The discussion attributes LILIE’s stronger near-data supervision to its language-informed representation and reports improved generation over pixel loss. It also qualifies perceptual-loss comparisons and describes implementation choices for noisy VLM inputs and image preprocessing.

5 DISCUSSION AND CONCLUSION

  • Near real data at t = 0.99, LILIE amplifies residual standard deviation by approximately 4.9× relative to pixels, making small deviations more prominent in supervision.Figure 9 compares standard deviation ratios for LILIE versus pixels on GPIC Nano using the LILIE-S encoder.Source: §5 · p. 9, Figure 9 · p. 9
  • DINOv2 and SigLIP 2 loss comparisons are not controlled experiments because their pretraining data may overlap GPIC Test and GPIC evaluation itself uses DINOv2 features.Their training data and compute also differ from the study’s, and the models violate the GPIC evaluation protocol.Source: §5 · p. 10
  • With LILIE-only loss, JiT-T2I-g gFD falls from 66.84 to 55.78, and Pixel+LILIE reaches 53.01.The encoder is a 32M-parameter invertible model; the discussion characterizes the loss as language-informed and suitable for end-to-end visual generation.Source: §5 · p. 10

A ADDITIONAL TRAINING DETAILS

A.1 ENCODER ARCHITECTURE

The encoder divides images into patches and applies autoregressive affine blocks to produce image-shaped features. Causal transformer layers generate each patch’s coupling parameters from preceding patches.

A.1 ENCODER ARCHITECTURE

  • Each image is split into T patches of dimension D, processed by K autoregressive affine blocks, and reassembled into an image-shaped feature tensor.Successive blocks alternate the patch order.Source: §A.1 · p. 16
  • For each patch, log-scale and shift vectors depend only on preceding patches in the block’s order; the first patch passes through unchanged.Both vectors are clipped to [−1, 1] before the affine map is applied.Source: §A.1 · p. 16, Figure A.1 · p. 17
  • Each block uses L causal transformer layers at bottleneck dimension H to produce coupling parameters, with a shifted output ensuring dependence only on earlier patches.The final head produces 2D values per position; the model-size ablation varies K, H, and L.Source: §A.1 · p. 16, Figure A.1 · p. 17

B EXPERIMENTAL DETAILS

This appendix records the experimental configurations, evaluation protocols, and perceptual-loss baseline implementations. It also identifies the additional benchmark results reported for JiT-T2I-g models.

B EXPERIMENTAL DETAILS

  • Table B.1 reports full GPIC Test-50K generation results, including three-trial mean ± one standard deviation for selected one-epoch Pixel and Pixel+LILIE runs.Models train for 390,600 steps, with selected configurations also trained for 781,200 steps; all use 50 Euler steps and validation-selected CFG.Source: Table B.1 · p. 18

B.1 ADDITIONAL BENCHMARKS

  • GenEval and DPG-Bench results are reported for JiT-T2I-g models trained for 390,600 steps, using 50 Euler steps and CFG values from Table B.1.GenEval uses prompts with the leading “a photo of” prefix removed; its caption reports that this improves scores for all three configurations.Source: §B.1 · p. 19, Table B.4 · p. 20, Table B.5 · p. 20

B.2 IMPLEMENTATION OF PERCEPTUAL LOSS BASELINES

  • Perceptual baselines use LPIPS, DINOv2, or SigLIP 2 feature losses, either alone or combined with pixel loss; combined losses gate perceptual supervision by noise level.The combined loss uses threshold τ = 0.3; weights are 0.1 for LPIPS and 0.01 for DINOv2 and SigLIP 2, while feature-only training uses weight 1 without the pixel term or gate.Source: §B.2 · p. 20

C NOISE LEVEL AND LOSS WEIGHT

The study selects LILIE corruption level and loss weight using validation gFD, comparing noise levels and mixing weights. The selected mixed-objective setting is σ = 0.25 and λ = 2.

C NOISE LEVEL AND LOSS WEIGHT

  • The sweep trains LILIE-S encoders at σ ∈ {0, 0.25, 0.5, 0.75, 1} and evaluates JiT-T2I-XL generators with λ ∈ {0.5, 1, 2} at 200K steps.Generators use 50 Euler steps, with CFG selected from {5.50, 6.25, 7.00, 7.75, 8.50} using validation gFD.Source: C NOISE LEVEL AND LOSS WEIGHT · pp. 20–21
  • σ = 0.25 and λ = 2 gives the lowest validation gFD among mixed objectives and is used as the default for one-epoch GPIC Full runs.Encoders trained without noise yield degenerate generation quality; increasing noise beyond 0.25 does not improve the best mixed result.Source: C NOISE LEVEL AND LOSS WEIGHT · p. 21, Table C.1 · p. 21

D MODEL SIZE AND DATA ABLATIONS

The appendix expands the model-size and training-data ablations and specifies the evaluation protocol for quantitative results.

D MODEL SIZE AND DATA ABLATIONS

  • Table D.1 reports full model-size ablations for LILIE encoders trained on 100M images, with the primary setting highlighted.TARFlow configurations are specified by hidden dimension, blocks, and layers per block.Source: Table D.1 · p. 21
  • Table D.2 reports full training-data ablations for the 32M-parameter LILIE-S encoder, highlighting its 100M-image primary setting.Quantitative results use GPIC Test-50K prompts at 200K training steps and 50 Euler steps; CFG is selected on validation data.Source: Table D.2 · p. 21, D MODEL SIZE AND DATA ABLATIONS · p. 21

E LIMITATIONS

The experiments are limited to one corpus, resolution, and generator family, with further constraints on encoder training and scale. The appendix also describes the caption prompts used for four GPIC caption-length categories.

E LIMITATIONS

  • Experiments cover only GPIC at 256 × 256 with JiT-T2I and SmolVLM-256M; encoder training uses one GPIC Full epoch, without an optimal-budget method.The largest encoder is 32M parameters, below the 86M-parameter DINOv2 and SigLIP 2 base models; combinations with perceptual losses are not tested.Source: E LIMITATIONS · p. 22

F CAPTIONING PROMPTS

  • GPIC captions fall into four length categories—tag, short, medium, and long—and LILIE training pairs each caption type with its own SmolVLM-256M prompt.The prompts call for unordered tags, short captions of 1–2 sentences, and medium captions of 2–3 sentences.Source: F CAPTIONING PROMPTS · p. 22

G SAMPLES

The appendix provides sampling trajectories, held-out random samples, and a flower mosaic for JiT-T2I-g models trained with Pixel, LILIE, and Pixel+LILIE supervision.

G SAMPLES

  • The trajectories compare Pixel, LILIE, and Pixel+LILIE models using the same prompt and initial noise across methods.They show clean predictions at Euler steps 0, 5, 10, and 25, then the final image at step 50, with a fixed 50-step budget.Source: Figure G.1 · p. 23, G SAMPLES · p. 23
  • Random held-out GPIC Test-50K examples show the real image first, followed by Pixel, LILIE, and Pixel+LILIE generated samples.The same ordering is specified for Figures G.2 to G.6.Source: Figure G.2 · p. 24, Figure G.3 · p. 25, Figure G.4 · p. 26
  • Figure G.7 presents selected lily and water lily samples from JiT-T2I-g trained with Pixel+LILIE.Source: G SAMPLES · p. 23, Figure G.7 · p. 29

Related papers

Loading 2610.23653v1…