Source-linked AI summary
Giraffe: A Mapping Architecture from Hidden Text Representations to Visual Embeddings for Efficient Graphic Design
Nejla Ghaboosi
TL;DR
MLLMs remain limited in generating visual content, while graphic design requires long sequences that interleave text, images, and layout information. Giraffe maps language-model hidden representations to visual-model embeddings using one [IMG] token per image and a lightweight two-block MLP architecture. The method demonstrates strong text-to-design and image-to-design performance, capturing image semantics, style, and colour information.
Problem
MLLMs mainly generate textual outputs, and multiple visual tokens make graphic-design sequences long and difficult to model.
Method
Giraffe maps language-model hidden representations to visual-model embeddings with one [IMG] token per image using two shallow MLP blocks with compression and shared expansion modules.
Results
Strong performance is demonstrated in both text-to-design and image-to-design generation, with designs capturing image semantics, style, and colour information.
Takeaways & Limitations
A single token per image supports complex, visually harmonious graphic designs while keeping the inference solution lightweight.
Takeaways & Limitations
The paper focuses primarily on images, although the architecture is described as adaptable to other media.
Abstract
from arXiv · showhide
Multimodal large language models (MLLMs) have made significant progress in understanding and interpreting mul- timedia content. However, their ability to generate me- dia remains limited. Recent approaches have attempted to bridge this gap by translating the hidden representations of token sequences into the embedding space of visual models or directly into raw image data. However, these methods often represent each image using multiple specialised to- kens which significantly increases the input length. This be- comes a major limitation for tasks such as graphic design generation where the output typically involves a seamless blend of thousands of tokens across text, multiple images, and layout information. To address this challenge, a novel architecture is proposed that maps hidden token represen- tations to the embedding space of visual models, such as CLIP ViT-L/14, using a single [IMG] token per image. The architecture employs two shallow MLP blocks, each with a separate compression module followed by a shared expan- sion module, trained with six distinct loss functions. One block aids the other during training and is omitted during inference, resulting in a lightweight solution. Strong perfor- mance is demonstrated in both image-to-design and text-to- design generation tasks.
1. Introduction
MLLMs have advanced visual understanding but remain limited in generating visual content, especially graphic designs that combine text, images, and layout in long sequences. Giraffe addresses this by mapping language-model representations into visual embeddings with one [IMG] token per image, enabling coherent designs from text or image inputs.
- Motivation: MLLMs primarily process multimodal inputs while producing textual outputs, limiting their effectiveness for graphic design generation.Graphic designs combine text, images, SVG shapes, colours, and spatial layout.
- Limitations of prior work: Existing generation methods use multiple visual tokens or specialised representations for each image, increasing sequence length.Examples include GILL’s r [IMG] tokens, Emu’s N [IMG] tokens, and Chameleon’s 1024 discrete image tokens.
- Limitations of prior work: Long interleaved sequences make it difficult for transformer architectures to capture long-range dependencies, constraining coherent and consistent design generation.Graphic designs may contain multiple images alongside text, SVG shapes, layout, and visual attributes.
- Giraffe architecture: Giraffe maps language-model hidden representations to visual-model embeddings such as CLIP ViT-L/14 using a single [IMG] token per image.Its two shallow MLP blocks use separate compression modules and a shared expansion module; the assisting block is removed at inference.
- Results: Experiments report strong performance for both text-to-design and image-to-design generation, with outputs preserving design semantics, style, and colour information.The paper reports visually coherent and stylistically consistent designs from both input modalities.
2. Method
The method represents each image with a single [IMG] token, maps its hidden representation into a visual embedding space, and trains a two-block Giraffe architecture with complementary objectives. During inference, the assisting block is removed and predicted embeddings are decoded into images placed within design bounding boxes.
- 2.1. Graphic Design Structure: Graphic designs are represented as long sequential structures of interleaved text, image, and SVG-shape components with spatial and visual properties.Image components remain necessary because visual content cannot be fully captured through text alone.
- 2.2. Giraffe Architecture: Each image embedding is extracted with frozen ViT-L/14, projected into the language model’s embedding dimension, and substituted for one [IMG] token.The learnable projection W maps visual embeddings into the shared word-embedding space.
- 2.2. Giraffe Architecture: The language model is fine-tuned with next-token cross-entropy over sequences containing text tokens and [IMG] tokens.The objective predicts every next token, with sequence lengths and batch size accounted for in the loss.
- 2.2. Giraffe Architecture: A main MLP maps [IMG] hidden states to normalised visual embeddings, using MSE and cosine losses against clipped, min-max-normalised targets.The predicted embeddings are compared in the visual encoder’s projected representation space through a frozen projection head.
- 2.2. Giraffe Architecture: An assisting MLP acts as an autoencoder with its own compression module and a shared expansion module, adding reconstruction and InfoNCE objectives.The total training loss combines six weighted losses, including the two bottleneck InfoNCE terms.
- 2.2. Giraffe Architecture: At inference, the assisting block is removed; denormalised predicted embeddings are passed to FLUX with a CLIP ViT-L/14 IP Adapter and placed in bounding boxes.Using predicted embeddings directly avoids recomputing visual-encoder embeddings for each generated [IMG] token.
3. Experiments
Experiments evaluate Giraffe on text-to-design and image-to-design generation using qualitative comparisons, FID, and CLIP cosine similarity. The proposed architecture produces diverse, coherent designs and closely follows image inputs while using a single [IMG] token per image.
- Experimental setup: Experiments use text-to-design and image-to-design tasks, with training on 1,800,000 professionally created graphic designs.The dataset includes social media posts, banners, flyers, business cards, and logos, split 90% for training and 10% for validation.
- Text-to-design generation: The proposed architecture generates more visually appealing designs with greater layout diversity and stronger stylistic and thematic coherence than the baseline.Its multi-image designs often maintain harmonious color themes and styles, whereas baseline outputs commonly show limited layouts and inconsistent image relevance.
- Text-to-design generation: A single [IMG] token preserves image style and thematic information while allowing more complex, visually coherent, and better-aligned designs.The representation lets the model focus on finer-grained details within its attention span.
- Text-to-design generation: The proposed method achieves a considerably lower FID score than the baseline, indicating higher-quality and more diverse generated designs.FID compares generated-design distributions with those of real professionally created designs from the training set.
- Image-to-design generation: 0.85 ± 0.05 CLIP cosine similarity indicates strong adherence to original images in the image-to-design task.Qualitative outputs generally align closely with GPT-4o-generated inputs, despite minor detail inconsistencies.
4. Conclusion
Giraffe maps language-model hidden representations to visual embeddings using one [IMG] token per image, with a training-only assisting block. Experiments show that this compact representation captures image semantics, style, and color for both text-to-design and image-to-design generation.
- Conclusion: Giraffe maps language-model hidden representations to a visual model’s embedding space using a single [IMG] token per image.Its mapping architecture uses two shallow MLP blocks with separate compression modules and a shared expansion module.
- Conclusion: The assisting MLP block is removed during inference, resulting in a lightweight solution.The second block helps the main block learn the mapping more effectively during training.
- Conclusion: Text-to-design experiments produce more complex, diverse, and visually harmonious designs than textual image representations.The results emphasize the importance of using a single token per image for generated-design diversity and complexity.
- Conclusion: Image-to-design experiments show that one [IMG] token per image captures semantic, stylistic, and color information.The result supports compact image representation for image-to-design generation.