PaperlayerView exact revision on arXiv

Source-linked AI summary

Generating an Image From 1,000 Words: Enhancing Text-to-Image With Structured Captions

Eyal Gutflaish, Eliran Kachlon, Hezi Zisman, Tal Hacham, Nimrod Sarid, Alexander Visheratin, Saar Huberman, Gal Davidi, Guy Bukchin, Kfir Goldberg, Ron Mokady

arXiv:2511.06876v1cs.CV

TL;DR

Most text-to-image models learn to turn short prompts into detailed images, leaving a mismatch that limits fine-grained control. This paper trains FIBO on long structured captions, uses DimFusion to fuse LLM representations without increasing token length, and evaluates reconstruction with TaBR. FIBO achieves the highest prompt-alignment score among open-source models and the highest TaBR reconstruction fidelity across baselines.

  • Problem

    Most text-to-image models map short prompts to detailed images, leaving a gap between sparse input and rich output that limits professionals’ fine-grained control.

    Source: §1 · p. 2

  • Method

    The paper trains a text-to-image model on long structured captions, uses DimFusion to fuse LLM representations without increasing token length, and introduces TaBR to evaluate image-grounded reconstruction.

    Source: §1 · p. 2, §3.3 · p. 7

  • Results

    FIBO achieves the highest prompt-alignment score among open-source models and highest TaBR reconstruction fidelity across baselines, winning over 90% of comparisons against SD3.5-Large.

    Source: §4.3 · p. 9

  • Takeaways & Limitations

    FIBO supports single-attribute edits that change the intended visual factor while keeping other factors stable.

    Source: §5 · p. 9

    The authors say licensed-data training substantially reduces infringement risk, but do not claim it eliminates that risk.

    Source: §6 · p. 9

Abstract

from arXiv · show

Text-to-image models have rapidly evolved from casual creative tools to professional-grade systems, achieving unprecedented levels of image quality and realism. Yet, most models are trained to map short prompts into detailed images, creating a gap between sparse textual input and rich visual outputs. This mismatch reduces controllability, as models often fill in missing details arbitrarily, biasing toward average user preferences and limiting precision for professional use. We address this limitation by training the first open-source text-to-image model on long structured captions, where every training sample is annotated with the same set of fine-grained attributes. This design maximizes expressive coverage and enables disentangled control over visual factors. To process long captions efficiently, we propose DimFusion, a fusion mechanism that integrates intermediate tokens from a lightweight LLM without increasing token length. We also introduce the Text-as-a-Bottleneck Reconstruction (TaBR) evaluation protocol. By assessing how well real images can be reconstructed through a captioning-generation loop, TaBR directly measures controllability and expressiveness, even for very long captions where existing evaluation methods fail. Finally, we demonstrate our contributions by training the large-scale model FIBO, achieving state-of-the-art prompt alignment among open-source models. Model weights are publicly available at https://huggingface.co/briaai/FIBO

1. Introduction

The paper addresses the gap between short prompts and detailed images by training on long structured captions, then introduces DimFusion and TaBR to support efficient conditioning and evaluation. Its FIBO model demonstrates strong prompt adherence and fine-grained control.

Motivation

  • Training on short prompts leaves models to guess unspecified image details, limiting precision and creativity for professionals who need fine-grained visual control.The paper frames this as a mismatch between sparse textual input and the rich detail required for high-quality images.Source: §1 · p. 2

Structured captions

  • Long structured captions encode detailed visual attributes to improve prompt adherence and disentangled control, while a VLM converts natural user intent into practical structured prompts.The captions maximize expressive coverage rather than biasing generation toward average human preferences.Source: §1 · p. 2, Figure 2 · p. 2

DimFusion

  • DimFusion integrates intermediate LLM tokens with image tokens without increasing token count, and a relatively small LLM can encode structured prompts.It addresses the efficiency challenge introduced by training with long captions.Source: §1 · p. 2

TaBR

  • TaBR evaluates expressive power by captioning a real image, regenerating it from that caption, and comparing the reconstruction with the original.The protocol can use automatic similarity measures, but the authors find human judgments perform better; it targets controllability rather than user-preference popularity.Source: §1 · p. 2

FIBO

  • FIBO is the first open-source text-to-image model trained entirely on long structured captions and achieves state-of-the-art prompt adherence, including for captions exceeding 1,000 words.The model also enables fine-grained changes to color, expression, or composition without unintended changes elsewhere; its weights, code, and implementation details are released.Source: §1 · p. 2

2. Related Work

Related work traces progress in diffusion architectures, caption supervision, LLM text encoding, and cycle-consistency evaluation. The paper extends these directions with structured captions, DimFusion, and TaBR.

Text-to-image models

  • GLIDE [27], Imagen [36], and DALL·E 2 [33] showed diffusion conditioned on strong language encoders, while latent diffusion [35] made large-scale training practical.Source: §2 · p. 2

UNet-based scaling

  • Stable Diffusion 3 [11], FLUX, HiDream-I1 [7], and Qwen-Image advance transformer and flow-matching approaches amid growing demands to follow long prompts accurately.Source: §2 · p. 3

Synthetic captions

  • Building on synthetic-caption work [11] [22], this paper uses a consistent structured JSON schema to capture fine-grained visual details and convert human prompts into machine-readable descriptions.Source: §2 · p. 3

LLM Fusion

  • Prior LLM-fusion methods [22] [39] fuse intermediate layers but typically lack joint text–image attention, while HiDream-I1 [7] incurs high computational cost; DimFusion combines intermediate and final representations at constant token length.Source: §2 · p. 3

Cycle-consistency-based evaluation

  • Where prior cycle-consistency work uses regeneration as a reward [1] [25] or alignment metric, TaBR compares a caption-based reconstruction directly with the original real image.Source: §2 · p. 3

3. Method

Section 3 presents the paper’s main contributions through a workflow shown in Figure 2: structured-prompt training, DimFusion, a long-caption evaluation protocol, and the training setup for FIBO.

3.1

  • Section 3.1 motivates training with long structured prompts and analyzes their effects on controllability and disentanglement.Source: §3 · p. 3

3.2

  • Section 3.2 introduces DimFusion, an architecture for integrating an LLM with a text-to-image model under reduced computational requirements.Source: §3 · p. 3

3.3

  • Section 3.3 presents an evaluation protocol for long captions, addressing limitations of existing methods on inputs exceeding 1,000 words.Source: §3 · p. 3

3.4

  • Section 3.4 describes the large-scale training setup that results in the model FIBO.Source: §3 · p. 3

FLUX FIBO (Ours) Flux FIBO (Ours)

FIBO is trained on long structured captions and uses DimFusion to incorporate intermediate LLM representations without extending the text sequence. The section also presents TaBR, which evaluates caption-driven reconstruction against real images.

FLUX FIBO (Ours) Flux FIBO (Ours)

  • FIBO enables precise depth-of-field control from shallow to deep and simultaneous control of facial expressions for multiple subjects, unlike FLUX’s inconsistent deep-depth-of-field results.These are qualitative controllability examples in Figure 4.Source: Figure 4 · p. 4

3.1. Training with Long Structured Captions

  • Structured captions consistently specify details such as color, lighting, and composition, reducing ambiguity and encouraging outputs grounded in the prompt rather than model priors.The schema also describes objects and people with attributes including size, position, shape, pose, and expression.Source: §3.1 · pp. 4–5
  • FID was 19.01 with long structured captions versus 34.04 with short captions, and long-caption training converged faster with better image quality.The comparison trained a 1B-parameter model for 100K steps on the same images; Figure 5 reports FID on 30K COCO-2014 validation images.Source: §3.1 · p. 4, Figure 5 · p. 4
  • Changing one structured-caption attribute typically changes only its corresponding visual factor, enabling iterative refinement without an input image or additional editing technique.The model is not trained on paired images for this behavior; the described comparisons hold the random seed fixed.Source: §3.1 · p. 4, Figure 6 · p. 5
  • A dedicated VLM can expand short prompts, refine structured captions, or derive and edit captions from images, making the long-caption workflow more practical for users.The authors fine-tune Qwen-3 VL 4B as an open-source alternative to costly models such as Gemini 2.5.Source: §3.1 · p. 5

3.2. DimFusion Architecture

  • DimFusion achieved slightly better FID than TokenFusion while cutting average step time from 0.8 seconds to 0.5 seconds, a ∼1.6× reduction in wall-clock time.Both TokenFusion and DimFusion substantially outperformed the T5-XXL baseline.Source: §3.2 · p. 6

3.3. Evaluating Long Captions

  • TaBR captions a real image, generates reconstructions from that text, and asks annotators which candidate is more similar to the original image.Anchoring judgments to the original image is intended to assess expressive power while avoiding the need to read extremely long captions.Source: §3.3 · p. 7

3.4. Large Scale Training

  • FIBO is an 8B-parameter text-to-image model trained on 120M licensed image-caption pairs with long structured JSON captions, using DimFusion and SmolLM3-3B.Training uses progressive resolution increases, followed by aesthetic finetuning and DPO training to improve text rendering.Source: §3.4 · p. 7

4. Experiments

Experiments evaluate FIBO on benchmarks, reconstruction, and user preference, reporting strong prompt alignment and expressive fidelity. Qualitative examples show it preserving fine-grained details and specified atypical relations.

4.1. Evaluation metrics.

  • Evaluation uses PRISM-Bench-licensed, GenEval, TaBR, and overall user preference to assess prompt adherence, compositional reasoning, aesthetics, and expressiveness.TaBR compares reconstructions against original images on 60 held-out images; preference comparisons cover 150 captions, randomize image order and model identity, and aggregate over 5,000 votes.Source: §4.1 · pp. 7–8

4.2. Qualitative Results

  • TaBR examples show FIBO more faithfully preserving global structure, pose, camera angle, scene layout, and fine details, while its examples retain specified atypical attributes and relations.The paper attributes this fidelity to structured captions encoding concepts and context; examples include a bear performing a handstand and women writing with a dart.Source: §4.2 · pp. 8–9

4.3. Quantitative Results

  • FIBO attains the highest PRISM-Bench-licensed alignment score among open-source models, narrowing the gap to Gemini 2.5 Flash Image despite its moderate 8B scale.The authors attribute its prompt understanding and text–image grounding to structured-caption supervision; they also report competitive results using a smaller, more cost-effective open-source VLM for structured prompts.Source: Table 3 · p. 8, §4.3 · p. 9
  • On GenEval, FIBO performs strongly across tasks, with a pronounced gain in positional understanding; it is comparable to other open-source models despite using fewer parameters.GenEval probes object count, color attribution, and spatial relations using controlled prompts.Source: Table 4 · p. 8, §4.1 · p. 8, §4.3 · p. 9
  • FIBO wins over 90% of TaBR comparisons against SD3.5-Large and 88.9% against HiDream-II-Full, and is favored over SD3.5 and two FLUX variants in overall preference.Its overall preference is comparable to Qwen-Image and HiDream-II-Full; TaBR results are strong against FLUX and Qwen-Image as well.Source: §4.3 · p. 9, Table 5 · p. 9

5. Conclusions

The conclusions frame long structured captions as a way to improve controllability through faster convergence and native disentanglement, with DimFusion and TaBR supporting training and evaluation. FIBO’s overall user preference surpasses several open-source baselines and is comparable to Qwen-Image and HiDream.

Conclusions

  • FIBO clearly surpasses SD3.5, FLUX.1-dev, and Flux.1-Krea-dev in overall user preference, and performs comparably to Qwen-Image and HiDream.The conclusions also describe FIBO as outperforming larger baselines.Source: Table 6 · p. 9, §5 · p. 9
  • Long structured captions accelerate convergence and enable native disentanglement, so single-attribute edits change the intended factor while other image factors remain stable.The captions encode richer visual and compositional detail than prior open-source approaches.Source: §5 · p. 9
  • DimFusion integrates intermediate and final LLM representations without increasing token length, making long-caption processing feasible.Source: §5 · p. 9
  • Text-as-a-Bottleneck Reconstruction (TaBR) uses an image-grounded protocol to measure expressiveness and controllability when thousand-word prompts exceed human evaluators.Source: §5 · p. 9
  • The authors propose translating natural-language user input into a structured intermediate dialect to enable new creative controls and predictable graphics primitives atop image generation.Source: §5 · p. 9

6. Ethical Statement

FIBO is trained solely on licensed data to reduce infringement risk, while its improved controllability may help reduce bias and misalignment but also increases dual-use risks.

Ethical Statement

  • Training solely on licensed data substantially reduces infringement risk compared with models trained on unlicensed online data.The authors advocate licensed, consent-based training as essential to a sustainable AI ecosystem.Source: §6 · p. 9
  • Structured synthetic captions and LLM fusion may strengthen prompt adherence and semantic grounding, helping reduce unintended bias and misalignment.Source: §6 · p. 9
  • Greater controllability also creates dual-use risks by enabling more reliable generation of misleading or inappropriate content, motivating study of safeguards.Source: §6 · p. 9

Appendices · A. Structured Captions

Structured captions encode images as detailed JSON fields, providing broad semantic coverage and finer control. They are generated with Gemini 2.5 using a dedicated prompt and schema-constrained outputs.

A. Structured Captions

  • Each caption is a long, detailed JSON that decomposes an image into interpretable fields.The fields cover objects, background, lighting, aesthetics, photographic attributes, style, and text.Source: A. Structured Captions · p. 12
  • The predefined structure is intended to guarantee full semantic coverage and improve controllability during training and inference.Source: A. Structured Captions · p. 12
  • Captions are generated with Gemini 2.5 using a dedicated system prompt and structured outputs configured with a JSON schema.The prompt and schema are specified in Tables 8 and 9; the schema enforces completeness, consistency, and adherence to its predefined structure.Source: A. Structured Captions · p. 12

B. Additional Training Details

Training proceeds through increasing-resolution stages, using flow matching and a specified optimizer and noise schedule, followed by aesthetic finetuning and DPO training to improve text rendering.

B. Additional Training Details

  • The model uses AdamW and flow matching with a logit-normal noise schedule and resolution-dependent timestep shifting.AdamW uses weight decay 1 × 10−4, β1 = 0.9, β2 = 0.999, and ϵ = 1 × 10−15; the learning rate is 1 × 10−4 with a constant schedule and 10K warmup steps, plus 2.5K additional warmup steps when resolution increases.Source: B. Additional Training Details · p. 12
  • Training spans 652k steps at 256^2 resolution, then 100k at 512^2 and 50k at 1024^2, with batch size increasing from approximately 1K to 2K.The first 300k steps at 256^2 use REPA with DINOv2-L encodings and a coefficient of 0.1.Source: B. Additional Training Details · p. 12
  • Post-training, aesthetic finetuning on 3,000 hand-picked images is followed by DPO training with dynamic beta to improve text rendering.Source: B. Additional Training Details · p. 12

C. Additional Ablation Studies Details

The ablations isolate caption length and text-encoder architecture using 1B-parameter models under controlled training conditions. The section specifies their variants and shared evaluation setup, but reports no ablation results.

C. Additional Ablation Studies Details

  • All ablations used 1B-parameter models with the SDXL VAE and identical training and sampling conditions, varying caption type or architecture.Table 10 lists the architectural hyperparameters for these models.Source: C. Additional Ablation Studies Details · p. 12, Table 10 · p. 12
  • The caption-length ablation compared long and short structured captions with DimFusion and SmolLM3-3B; the architecture ablation compared Baseline, DimFusion, and TokenFusion on long captions.The Baseline follows [11] and uses T5 [32]; TokenFusion follows HiDream [7] and concatenates SmolLM3-3B hidden states with text embeddings, doubling text tokens in each attention layer.Source: C. Additional Ablation Studies Details · p. 12
  • Models trained on 70M licensed image-caption pairs shared the optimizer, learning rate, noise schedule, batch size 1024, and REPA setup.REPA [46] used DINOv2-L encodings [28] with coefficient 0.5.Source: C. Additional Ablation Studies Details · p. 12
  • Evaluation computed FID on 30K COCO-2014 validation samples using 50 inference steps and no classifier-free guidance.Source: C. Additional Ablation Studies Details · p. 12

D. Additional Samples from FIBO

The section presents additional FIBO samples across aspect ratios and examples of refinement and disentanglement.

D. Additional Samples from FIBO

Loading 2511.06876v1…