Text-to-Image Generative Modeling Without Language Pretraining

1 INTRODUCTION

• Language pretraining’s contribution to T2I generation has not been systematically quantified, and its cost is often excluded from estimates of total system-building compute.In representative systems, language pretraining accounts for over 80% of total pretraining FLOPs.Source: §1 · p. 2
• Halcyon trains early-fusion autoregressive models from scratch on GPIC’s 100M image–text pairs and measures language-pretraining utility across three experimental axes.The axes are from-scratch pretraining, controlled data composition and scheduling, and compute comparisons with representative baselines.Source: §1 · p. 2
• 48.0 gFD lets Halcyon-0.52B outperform LlamaGen-XL/8 at 63.3 gFD with 18.2× less total pretraining compute.Halcyon-1.6B also outperforms JiT-T2I-XXL/8++ (35.3 vs. 41.0 gFD) using 275× less total pretraining compute.Source: §1 · p. 2
• Replacing caption dropout with image-to-text supervision enables image captioning while maintaining competitive T2I quality under the same pretraining budget.The replacement uses I2T sequences instead of caption-dropped T2I examples.Source: §1 · p. 2

2 RELATED WORK

• Late fusion uses a separately pretrained language model to condition a visual generator, while early fusion combines text and image tokens in one multimodal model.Early-fusion systems still incorporate text-only pretraining; Halcyon uses paired image–text supervision without it.Source: §2 · pp. 2–3
• In this paper, language pretraining means pretraining on web-scale text-only corpora, excluding image-paired captions.Source: §2 · p. 2
• Systems with language pretraining consume 10^12–10^14 text tokens, often more than their vision budgets, whereas Halcyon’s paired-caption supervision is over two orders of magnitude lower.Figure 2 compares language and vision pretraining tokens; Emu3 and Tuna-2 values are estimates.Source: Figure 2 · p. 2, §2 · p. 3
• Language pretraining’s value for T2I has been debated, with reported benefits ranging from scaling text encoders to marginal or limited gains from pretrained text models.Imagen reported benefits from scaling pretrained text encoders, while Parti found only marginal loss reduction and Zhang et al. found limited benefits from pretrained-language-model initialization.Source: §2 · p. 3

3 APPROACH

• Halcyon is a decoder-only Transformer with a joint text–image vocabulary, and all model components are trained from scratch.The components include the text tokenizer, image tokenizer, and multimodal generative model.Source: §3 · p. 3

3.1 TOKENIZATION

• The text and image tokenizers are trained from scratch, and their tokens join special tokens in a shared vocabulary with a shared embedding table and output head.The vocabulary contains 16,384 text tokens, 64,000 image tokens, and 256 special tokens; each 256 × 256 image is represented by 1,024 image tokens.Source: §3.1 · p. 3

3.2 SEQUENCE FORMAT AND TRAINING OBJECTIVE

• Four sequence formats cover text-only, image-only, T2I, and I2T prediction, using weighted next-token cross-entropy over non-padding text and image tokens.I2T predicts image tokens first and then caption tokens conditioned on the image; caption dropout on T2I examples trains the unconditional branch used for classifier-free guidance.Source: §3.2 · p. 3, Figure 3 · p. 4

3.3 PRETRAINING DATA COMPOSITION AND SCHEDULING

• The experiments control text-only, T2I, and I2T composition and scheduling under a fixed pretraining budget.Configurations include T2I-only training, staged text-only→T2I or I2T→T2I, and interleaved T2I/I2T; one variant replaces caption dropout with I2T while preserving conditional and unconditional image-training tokens.Source: §3.3 · p. 4, Figure 4 · p. 4

4.1 EXPERIMENTAL SETUP

• Halcyon configurations are trained for one epoch on GPIC’s approximately 100M image–text pairs and evaluated at 256×256 on GPIC, GenEval, and DPG-Bench.The study also reports token-level NLL and image-caption alignment.Source: §4.1 · pp. 5–6
• The GPIC-trained image tokenizer outperforms Cosmos-DI-8x8 on all reconstruction metrics, while the scratch-trained text tokenizer has comparable GPIC-caption compression to Qwen3 despite a 9.3× smaller vocabulary.The text tokenizer also outperforms FLAN-T5 on GPIC test captions; higher bytes/token and lower tokens/caption indicate better compression.Source: §4.1 · p. 5, Table 1 · p. 5, Table 2 · p. 5

4.2 RESULTS: COMPARISON WITH LATE-FUSION T2I METHODS

• 35.3 vs. 41.0 gFD: Halcyon-1.6B outperforms JiT-T2I-XXL/8++ with 275× less pretraining compute; Halcyon-0.52B also beats LlamaGen-XL/8 (48.0 vs. 63.3 gFD).The comparisons are reported on GPIC.Source: Table 4 · p. 7, §4.2 · p. 8
• 147.0 →63.3 gFD for LlamaGen-XL after switching tokenizers, and 67.9 →48.9 →41.0 gFD for JiT-T2I-XXL after adding perceptual supervision and reducing patch size.The resulting LlamaGen-XL/8 and JiT-T2I-XXL/8++ configurations are used in subsequent Halcyon comparisons.Source: §4.2 · p. 7
• Reducing FLAN-T5 from 1.22B to 0.11B parameters changes LlamaGen gFD only from 63.3 to 66.7, despite 13.5× less language-model training compute.The image tokenizer and 0.90B image generator are held constant.Source: §4.2 · p. 7

4.3 RESULTS: EFFECT OF LANGUAGE PRETRAINING ON T2I GENERATION

• Under a fixed budget, text-only pretraining worsens gFD as its allocation grows, and neither a full T2I epoch after pretraining nor continual pretraining improves generation.The continual-pretraining comparison starts from Qwen3-0.6B-Base pretrained on 36T text tokens; the lower learning-rate variant also fails to improve T2I generation.Source: §4.3 · p. 9, Table 6 · p. 9, Table 7 · p. 9

4.4 RESULTS: EFFECT OF I2T SUPERVISION ON T2I GENERATION

• At 10% I2T allocation, staged and mixed training preserve comparable gFD; at 50%, neither beats baseline, though mixing substantially reduces captioning NLL.At 50%, staged training achieves 54.2 gFD versus 76.0 for mixing, while mixing yields conditional text NLL of 1.009 versus 20.437 for staging.Source: §4.4 · pp. 9–10, Table 8 · p. 10
• Replacing caption-dropped T2I examples with I2T sequences enables captioning under the same budget while keeping T2I metrics comparable across model sizes.Conditional text NLL falls by more than an order of magnitude across sizes, and caption alignment rises from 0.090 to 0.559 as model size increases, remaining below GPIC ground-truth captions.Source: §4.4 · p. 10, Table 9 · p. 10, Figure 6 · p. 11

Model scaling and classifier-free guidance

• As model size grows from 0.1B to 4B, image and text validation NLL decrease, while the optimal guidance scale drops from 7.5 to 4.5 and gFD improves from 142.7 to 28.4.Source: §5 · p. 11, Figure 7 · p. 11, Figure 8 · p. 12

Caption length and text conditioning

• Across model sizes, medium and long captions achieve lower gFD than short captions, but longer captions do not consistently outperform medium-length captions.Source: §5 · p. 12, Table 10 · p. 12

Implications of language pretraining for T2I generation

• Text-only pretraining does not guarantee useful image-generation initialization, and the study does not isolate joint training’s effect or compare autoregressive and diffusion generators.Source: §5 · p. 12

Limitations and future directions

• The findings use GPIC, and extending them to other datasets and higher resolutions remains important; controlled interventions focus on autoregressive models.Source: §5 · pp. 12–13

Conclusion

• Strong T2I systems can be trained from scratch without internet-scale language pretraining, and Halcyon outperforms strong late-fusion baselines with less total pretraining compute.Source: §5 · p. 13

• Generative AI tools assisted with implementing methods and creating or editing software code.Source: AI USE STATEMENT · p. 13
• The tools also helped find literature, draft and edit manuscript sections, and improve readability.Source: AI USE STATEMENT · p. 13
• The authors report that the tools did not propose hypotheses, design experiments, analyze data, or interpret results, and that AI-assisted work was reviewed.Source: AI USE STATEMENT · p. 13

ETHICS STATEMENT

• T2I models may be misused, and Halcyon and its baselines may inherit biases or limitations in GPIC images and captions.Source: ETHICS STATEMENT · p. 13

APPENDIX

• Table 11 provides architecture and training details for Halcyon models, alongside configuration tables for JiT-T2I and LlamaGen.Source: APPENDIX · p. 17, Table 11 · p. 17, Table 12 · p. 18

B SAMPLING DETAILS

• Halcyon generates 1,024 image tokens after the caption and [SOV], applies CFG only to image-token logits, and selects guidance scale on validation data.Source: B SAMPLING DETAILS · p. 18

C.1 PER-CATEGORY RESULTS ON GENEVAL AND DPG-BENCH

• On GenEval, I2T supervision matches the standard recipe within one standard deviation at every model size; on DPG-Bench, it scores slightly higher at four of five sizes.Source: §C.1 · p. 18
• Tables 18 and 19 report per-category GenEval and DPG-Bench results for models listed in Table 5.Source: §C.1 · p. 18

C.2 STANDARD DEVIATION (TRAINING SEEDS)

• The standard deviation of gFD across three training seeds is 5.54, a reference scale for interpreting differences between small models.The runs use Halcyon-0.10B with I2T supervision and keep other settings fixed.Source: §C.2 · p. 19
• The training-seed comparison uses three Halcyon-0.10B runs with I2T supervision and fixed settings.Each run is evaluated with CFG 7.5.Source: §C.2 · p. 19
• Precision, recall, density, and coverage remain stable across training seeds.Source: §C.2 · p. 19

C.3 STANDARD DEVIATION (SAMPLING SEEDS)

• Across Halcyon models, the standard deviation of gFD across sampling seeds is at most 0.14.Each model uses a fixed checkpoint and CFG scale for three sampling runs of 50K images each.Source: §C.3 · p. 19

C.4 INFERENCE COST

• Inference cost is measured on the GPIC 50k test set by generating all 50k images on a single 8-GPU H100 node with vLLM, reporting wall-clock time averaged over three runs.Sampling settings match the evaluation settings.Source: §C.4 · p. 19

D SAMPLES

• Rendering legible text remains challenging for all models at 256×256 resolution.Source: D SAMPLES · p. 20
• CFG sweeps select each model’s lowest-gFD scale on GPIC Validation-50K for subsequent evaluations.The sweeps cover LlamaGen and JiT-T2I variants.Source: Figure 9 · p. 21, Table 16 · p. 21
• Random initialization achieves lower image-token validation NLL, while language-pretrained initialization achieves lower text-token validation NLL.Source: Figure 10 · p. 22
• The appendix reports category-level GenEval and DPG-Bench results; GenEval overall scores are mean ± standard deviation.Source: Table 18 · p. 22, Table 19 · p. 23
• The appendix tabulates three-seed training and sampling variability; inference-cost timing uses three 50K-image runs and excludes decoding.The inference timing uses 8 GPUs with vLLM.Source: Table 17 · p. 21, Table 20 · p. 23, Table 21 · p. 23
• Halcyon sample quality improves as model size scales across tag, short, medium, and long captions.Larger models capture finer clothing, object attributes, and scene layout details.Source: D SAMPLES · p. 20, Figure 11 · p. 24, Figure 12 · p. 25