PaperlayerView exact revision on arXiv

Source-linked AI summary

ShareGPT-4o-Image: Aligning Multimodal Models with GPT-4o-Level Image Generation

Junying Chen, Zhenyang Cai, Pengcheng Chen, Shunian Chen, Ke Ji, Xidong Wang, Yunjin Yang, Benyou Wang

arXiv:2506.18095v1cs.CVcs.AIcs.LG

TL;DR

GPT-4o-Image offers advanced image generation but is proprietary and opaque. The paper distills its capabilities into ShareGPT-4o-Image and fine-tunes Janus-Pro into Janus-4o, which improves text-to-image generation and adds text-and-image-to-image generation with 91K synthetic samples and 6 hours of training.

  • Problem

    GPT-4o-Image provides advanced instruction-aligned image generation, but its proprietary and undisclosed implementation is inaccessible to open multimodal research.

  • Method

    The paper synthesizes 45K text-to-image and 46K instruction-guided editing examples with GPT-4o-Image, then fine-tunes Janus-Pro on the resulting dataset.

  • Results

    Janus-4o significantly improves text-to-image generation over Janus-Pro and achieves competitive text-and-image-to-image performance, including a 3.26 ImgEdit-Bench score.

  • Takeaways & Limitations

    ShareGPT-4o-Image and Janus-4o provide an open route toward multimodal image generation aligned with GPT-4o-level capabilities.

Abstract

from arXiv · show

Recent advances in multimodal generative models have unlocked photorealistic, instruction-aligned image generation, yet leading systems like GPT-4o-Image remain proprietary and inaccessible. To democratize these capabilities, we present ShareGPT-4o-Image, the first dataset comprising 45K text-to-image and 46K text-and-image-to-image data, all synthesized using GPT-4o's image generation capabilities for distilling its advanced image generation abilities. Leveraging this dataset, we develop Janus-4o, a multimodal large language model capable of both text-to-image and text-and-image-to-image generation. Janus-4o not only significantly improves text-to-image generation over its predecessor, Janus-Pro, but also newly supports text-and-image-to-image generation. Notably, it achieves impressive performance in text-and-image-to-image generation from scratch, using only 91K synthetic samples and 6 hours of training on an 8 A800-GPU machine. We hope the release of ShareGPT-4o-Image and Janus-4o will foster open research in photorealistic, instruction-aligned image generation.

1 Introduction

GPT-4o-Image demonstrates strong text-to-image and text-and-image-to-image generation, but its proprietary, undisclosed implementation limits accessibility. The paper addresses this gap with a synthesized dataset and Janus-4o, an open model supporting both tasks.

  • GPT-4o-Image offers strong instruction alignment, semantic coherence, and photorealism across text-to-image and text-and-image-to-image generation, while remaining proprietary and opaque.Existing open-source models such as Janus-Pro still fall significantly behind.
  • ShareGPT-4o-Image contains 45K text-to-image and 46K text-and-image-to-image prompts with 91K GPT-4o-generated image outputs.The dataset was curated for diversity and quality and covers varied styles, grounded visual reasoning, instruction-following, and visual aesthetics.
  • Janus-4o is developed by fine-tuning Janus-Pro on ShareGPT-4o-Image and supports both text-to-image and text-and-image-to-image generation.This adds image-to-image capability to a predecessor described as text-to-image-only.
  • Janus-4o improves over Janus-Pro by 4 points on EvalGen and 1.6 points on DPG-Bench, while achieving strong open-source image-editing performance.Human evaluations also report stronger preference for Janus-4o outputs.
  • The release of ShareGPT-4o-Image and Janus-4o is intended to foster open research in instruction-aligned multimodal image generation.

2 ShareGPT-4o-Image

ShareGPT-4o-Image is constructed through complementary pipelines for diverse text-to-image pairs and structured instruction-guided editing triplets. GPT-4o-Image generates the corresponding outputs from synthesized prompts and source images.

  • ShareGPT-4o-Image comprises 45k text-to-image pairs and 46k instruction-guided image-editing triplets.The data-generation process uses Gemini-Pro-2.5 for text synthesis tasks.
  • The text-to-image collection combines prompt-first generation with image-first grounding to cover controlled attributes and realistic visual content.
  • 2.1 Text-to-Image Data: The prompt-first pipeline samples attributes from a six-dimensional space and composes them into natural-language prompts before GPT-4o-Image generates paired images.The attribute vocabulary includes 1,000 ImageNet object categories.
  • 2.1 Text-to-Image Data: The image-first pipeline uses diverse real-world images from ALLaVA and generates detailed prompts describing their visual content.These pairs expose the dataset to language describing naturally occurring scenes.
  • 2.2 Instruction-Guided Image Editing Data: The editing pipeline defines 14 tasks across five high-level categories, selects source images and tasks, and synthesizes editing instructions.
  • 2.2 Instruction-Guided Image Editing Data: GPT-4o-Image executes each synthesized instruction on the source image, producing 46k triplets spanning common image-editing scenarios.

3 Janus-4o: Fine-Tuning with ShareGPT-4o-Image

Janus-4o fine-tunes Janus-Pro to jointly perform text-to-image and text-and-image-to-image generation. Its training represents prompts, target images, and—when editing—input-image semantics and tokens for autoregressive image-token prediction.

  • Janus-4o is built on Janus-Pro through fine-tuning on ShareGPT-4o-Image, with both generation tasks jointly trained.
  • 3.1 Text-to-Image Fine-Tuning: For text-to-image training, tokenized prompts and codebook-mapped target-image patches are embedded and passed to the LLM for autoregressive image-token prediction.
  • 3.1 Text-to-Image Fine-Tuning: 10% of text-prompt tokens are randomly masked during text-to-image fine-tuning to encourage pixel-level dependency modeling.
  • 3.2 Text-and-Image-to-Image Fine-Tuning: Unlocking New Capabilities: Text-and-image-to-image training combines an input image’s semantic embedding and tokenized representation with prompt tokens to generate target-image tokens.This design addresses Janus-Pro’s lack of native text-and-image-to-image support.
  • 3.2 Text-and-Image-to-Image Fine-Tuning: Unlocking New Capabilities: 50% of input-image tokens are randomly masked during editing fine-tuning to reduce overfitting to the input image.
  • Training jointly samples 45K text-to-image and 46K text-and-image-to-image examples for three epochs, completing in 6 hours on one 8×A800 GPU machine.

4 Experiments

The experiments evaluate Janus-4o on text-to-image, text-and-image-to-image, and human-preference tasks. Janus-4o improves text-to-image performance, achieves strong image-editing results, and aligns more closely with human preferences than prior models.

  • Evaluation Setup: The evaluation covers GenEval and DPG-Bench for text-to-image generation, ImgEdit-Bench for image-conditioned editing, and a human study of model preferences.The human study compares Janus-4o with Janus-Pro-7B and UltraEdit using real Twitter examples.
  • Text-to-Image Performance: 4 points on GenEval and 1.6 points on DPG-Bench represent Janus-4o’s improvements over Janus-Pro in text-to-image generation.Janus-4o also achieves 80% overall accuracy on GenEval and a score of 85.71 on DPG-Bench.
  • Text-and-Image-to-Image Performance: 3.26 on ImgEdit-Bench surpasses Step1X-Edit and ImgEdit-E1, with particularly strong results in Motion Change and Style Transfer.Janus-4o reaches this performance using only 91K training samples, although Janus-Pro lacked text-and-image-to-image support.
  • Human Evaluation: Janus-4o’s human-evaluation outputs align more closely with preferences than previous models across text-to-image and text-and-image-to-image tasks.It shows higher visual quality than Janus-Pro for text-to-image and improved instruction-following for image modifications.

5 conclusion

The paper concludes that ShareGPT-4o-Image captures GPT-4o’s image-generation capabilities and enables Janus-4o to generate images from text or combined image-text inputs. Janus-4o achieves improved or competitive performance with limited training compute.

  • 5 conclusion: ShareGPT-4o-Image is presented as a large-scale dataset covering GPT-4o capabilities in text-to-image and text-and-image-to-image generation.The dataset is described as the first large-scale dataset of this kind.
  • 5 conclusion: Janus-4o generates high-quality images from text alone or from combined image and text inputs.The model supports both text-to-image and text-and-image-to-image generation.
  • 5 conclusion: 6 hours on an 8×A800 GPU machine yields notable performance gains with minimal compute.The conclusion attributes this efficiency to MLLM-based autoregressive image generation.

A.2 Datasets for Image Generation

Image-generation datasets evolved from captioned image-text pairs toward larger-scale internet-collected pairs as researchers sought stronger performance and generalization. Multimodal LLMs then extended image understanding toward image generation by aligning visual features with text.

  • A.2 Datasets for Image Generation: MS-COCO and TextCaps provided early natural-language captions aligned with images for text-to-image generation.These datasets are described as early image-text pair resources.
  • A.2 Datasets for Image Generation: Later datasets collected larger-scale image-text pairs from the internet to enhance generative-model performance and generalization.The passage frames this as a subsequent development after early image-text datasets.
  • A.2 Datasets for Image Generation: Multimodal LLM research enabled image understanding by aligning visual features with text using paired image-text data and pretrained image encoders.Image generation is presented as a natural extension of this multimodal development.

B.1.1 Text-to-Image Dimensions

The paper organizes text-to-image evaluation around defined dimensions and sub-dimensions, with category distributions summarized visually. Object sub-dimensions are referenced separately through ImageNet.

  • B.1.1 Text-to-Image Dimensions: Table 4 defines the dimensions and sub-dimensions used to describe text-to-image data.The table is the stated source for the detailed text-to-image dimensions.
  • B.1.1 Text-to-Image Dimensions: Figure 6 presents the distribution of text-to-image examples across categories.The passage identifies Figure 6 as the visual summary of category distributions.
  • B.1.1 Text-to-Image Dimensions: Object-dimension sub-dimensions are available through ImageNet rather than being detailed in Table 4.This is the stated exception to the table’s coverage.

B.1.2 Text-and-Image-to-Image Categories

This section organizes text-and-image-to-image tasks into categories and subcategories, with their distribution presented separately.

  • Table 5 lists the categories and subcategories for text-and-image-to-image tasks.
  • Figure 7 presents the distribution of text-and-image-to-image categories.
  • The sampling appendix describes selecting the number of object categories k from 1 to 100 with weighted random sampling.

B.2.1 Exponential Decay Distribution

The section uses an exponential-decay distribution to sample category counts, making lower counts more likely while controlling the decay rate with λ.

  • An exponential-decay distribution assigns decreasing probability to larger category counts, favoring scenarios with fewer categories.
  • λ controls the decay rate, with larger λ producing faster decay.
  • The distribution is illustrated in a figure showing its exponential-decay characteristics.

E Ethical Considerations and Societal Impact

The dataset is intended to support open multimodal-AI research, but its synthetic proprietary-model outputs, possible biases, and misuse risks require downstream scrutiny and responsible use.

  • ShareGPT-4o-Image contains synthesized outputs from proprietary GPT-4o-Image, whose internal workings remain opaque.
  • GPT-4o-Image outputs may inherit or amplify biases related to gender, race, age, occupation, and other demographic attributes.
  • The dataset includes no post-hoc filtering for perceived bias because subjective filtering could introduce new biases.
  • LLM-generated document and descriptive text may be misconstrued or reflect unintentional training-data biases despite generic, fictional prompting.
  • The release is intended for academic research, while harmful content, stereotypes, misinformation, and rights infringement are explicitly discouraged.
  • The authors encourage systematic bias audits and shared responsibility for investigating and mitigating potential societal harms.
Loading 2506.18095v1…