Source-linked AI summary
StoryGAN: A Sequential Conditional GAN for Story Visualization
Yitong Li, Zhe Gan, Yelong Shen, Jingjing Liu, Yu Cheng, Yuexin Wu, Lawrence Carin, David Carlson, Jianfeng Gao
TL;DR
Story visualization asks models to turn a multi-sentence paragraph into an image sequence that remains globally consistent as scenes and characters evolve. StoryGAN addresses this with sequential context encoding and two-level adversarial discrimination, and the paper reports improved image quality and consistency over baselines.
Problem
Story visualization requires coherent image sequences that capture evolving scenes and characters, which single-sentence text-to-image processing does not ensure.
Method
StoryGAN uses a sequential conditional GAN with a Context Encoder that dynamically updates story context and image- and story-level discriminators.
Results
StoryGAN improves image quality and global consistency over previous state-of-the-art models in quantitative and human evaluations.
Takeaways & Limitations
The model jointly considers each sentence with contextual information to generate sequences that are coherent with the story.
Abstract
from arXiv · showhide
We propose a new task, called Story Visualization. Given a multi-sentence paragraph, the story is visualized by generating a sequence of images, one for each sentence. In contrast to video generation, story visualization focuses less on the continuity in generated images (frames), but more on the global consistency across dynamic scenes and characters -- a challenge that has not been addressed by any single-image or video generation methods. We therefore propose a new story-to-image-sequence generation model, StoryGAN, based on the sequential conditional GAN framework. Our model is unique in that it consists of a deep Context Encoder that dynamically tracks the story flow, and two discriminators at the story and image levels, to enhance the image quality and the consistency of the generated sequences. To evaluate the model, we modified existing datasets to create the CLEVR-SV and Pororo-SV datasets. Empirically, StoryGAN outperforms state-of-the-art models in image quality, contextual consistency metrics, and human evaluation.
1. Introduction
Story Visualization generates one image per sentence in a multi-sentence story, requiring coherent global depiction of evolving scenes and characters. StoryGAN addresses this with sequential context modeling and image- and story-level discrimination.
- Story Visualization generates a sequence of images from a multi-sentence paragraph, with one image describing each sentence.
- The task requires consistent and coherent depiction of the whole story, because individual sentences may omit previously established scene information.A later sentence describing a new object may not capture the entire scene when processed alone.
- Unlike video generation, story visualization emphasizes key static frames and coherent scene changes rather than smooth motion transitions.Its inputs are complex stories whose backgrounds and plots may change across images.
- StoryGAN is a sequential conditional GAN whose Context Encoder dynamically updates story information as sentences and generated images progress.The Context Encoder combines a GRU cell with Text2Gist to produce a changing Gist representation for image generation.
- Two discriminators assess sentence-image relevance and global coherence between the generated image sequence and the whole story.
- StoryGAN outperforms previous state-of-the-art models in image quality, global consistency metrics, and human evaluation.The paper evaluates the task using modified CLEVR and Pororo datasets.
2. Related Work
Story visualization relates to text-to-image, video generation, image retrieval, cartoon generation, and visual storytelling, but combines sequential image generation with story-level consistency and scene changes.
- Story visualization occupies a distinct position within broader generative modeling tasks despite its relation to established text-to-image methods.
- Text-to-image generation provides a close foundation, but story visualization uses longer and more complex multi-sentence input than single-image generation.Related tasks include dialogue-to-image generation, textual image editing, and placing specified objects from text.
- Video generation is another close relation, but existing approaches typically generate short clips without scene changes and emphasize smooth motion transitions.Motion features such as trajectories, skeletons, or landmarks help model transitions across video frames.
- Story image retrieval generates outputs from a pre-collected training set, while cartoon generation uses cut-and-paste techniques; both require large amounts of labeled training data.
- Visual storytelling reverses the direction of story visualization by generating a paragraph that describes a sequence of input images.
3. StoryGAN
StoryGAN generates one image per sentence while targeting both sentence-image matching and coherence across the full story. Its stochastic Story Encoder, recurrent Context Encoder, image generator, and two discriminators jointly model evolving contextual information and local-global consistency.
- StoryGAN: StoryGAN produces one generated image per sentence, with local consistency defined by sentence-image matching and global consistency by coherence with the full story.The story is represented as a variable-length sentence sequence, and the generated image sequence is evaluated at both levels.
- Story Encoder: The Story Encoder maps the whole story to a stochastic low-dimensional vector that initializes the Context Encoder.It samples from a Gaussian distribution with diagonal covariance, uses reparameterization, and adds KL regularization against a standard Gaussian.
- Context Encoder: The deep Context Encoder combines each sentence with recurrent contextual information, dynamically updating a Gist representation as objects and scenes change through the story.Its lower layer uses GRU cells, while the upper Text2Gist layer updates context and produces the image-generation representation.
- Context Encoder: Text2Gist uses gated updates and a learned convolutional filter to combine local sentence information with global story context.The filter transforms the GRU output into a multi-channel 1D filter, and the convolution with the contextual hidden vector produces the Gist output.
- Discriminators: The image discriminator checks sentence-image relevance, while the story discriminator evaluates coherence between the generated image sequence and the whole story.The story discriminator concatenates image and sentence features, combines them by element-wise product, and predicts whether the story pair is real or fake.
4. Experiments
Experiments evaluate StoryGAN against ablated and competing models on modified CLEVR and Pororo datasets, using structural, consistency, quality, and human-evaluation measures. Across these evaluations, StoryGAN produces more coherent and higher-quality story image sequences.
- Experimental setup: The experiments compare StoryGAN with ImageGAN, SVC, and SVFN, while noting that conditional video generation was too blurry for direct comparison.ImageGAN generates images independently; SVC uses concatenation, and SVFN uses a filter network.
- Datasets: CLEVR-SV modifies CLEVR with up to four objects having controlled materials, colors, sizes, shapes, and layouts.The input describes each object’s attributes and relative position, so generated appearance and positions can be compared with ground truth.
- CLEVR-SV results: StoryGAN generates more feasible CLEVR-SV sequences than competitors by tracking story progress, using image- and story-level discriminators, and initializing the recurrent cell with the Story Encoder.Baselines mix attributes, produce implausible frames, or improve consistency only partially.
- CLEVR-SV results: StoryGAN significantly outperforms other baselines on CLEVR-SV SSIM, which measures alignment between generated images and input descriptions.SSIM is used because CLEVR descriptions allow little visual variation.
- Datasets: 15,336 description-story pairs form Pororo-SV, with 13,000 training pairs and 2,336 testing pairs built from five continuous sampled images per story.The dataset adapts Pororo clip descriptions and samples one frame from each video clip during training.
- Pororo-SV results: On Pororo-SV, StoryGAN improves visual quality and sequence consistency, including stronger first images and more stable character appearance than competing methods.Human evaluations also favor StoryGAN: it outperforms ImageGAN in pairwise quality, consistency, and relevance, and achieves the highest average rank across four models.
5. Conclusion
StoryGAN treats story visualization as sequential conditional generation, using contextual information and adversarial structure to improve consistency. The paper reports improvements over baseline models in both quantitative and human evaluations.
- 5. Conclusion: StoryGAN jointly considers each input sentence with contextual information through the Text2Gist component in the Context Encoder.The Context Encoder provides the image generator with local and global conditional information.
- 5. Conclusion: The two-level discriminator and recurrent input structure help ensure consistency across generated images and the visualized story.
- 5. Conclusion: Both quantitative and human evaluations show that StoryGAN improves generation compared with baseline models.
- 5. Conclusion: Human evaluation criteria include visual appeal, consistency across images, natural story formation, and coverage of the input story’s main characters.
A. Network Configuration
The network configuration specifies the layer conventions used in StoryGAN and identifies the conditional fusion performed by the Filter Network and a designated table layer.
- A. Network Configuration: CONV layers are configured by output channels, kernel size, step size, and padding size.
- A. Network Configuration: LINEAR denotes a fully connected layer whose input and output dimensions are given in parentheses.
- A. Network Configuration: The Filter Network is contained in the Text2Gist cell and transforms its output into a filter.
- A. Network Configuration: The starred layer combines the conditional input with the encoded images.
B. More Examples of CLEVR-SV Dataset
Additional CLEVR-SV examples test whether models preserve the first object while later story descriptions change. The figures show StoryGAN generations under varied initial object attributes and report that only StoryGAN preserves story consistency in the comparison.
- B. More Examples of CLEVR-SV Dataset: The experiment keeps the first object’s attributes fixed to test consistency through subsequent generations.
- B. More Examples of CLEVR-SV Dataset: Figure 8 changes the first sentence to specify a large rubber cyan cylinder at (-0.46, -0.36).
- B. More Examples of CLEVR-SV Dataset: In Figure 8, each column represents one layout of three objects, and the first row is the original image to modify.
- B. More Examples of CLEVR-SV Dataset: Only StoryGAN maintains story consistency among the compared methods when the original story description is changed.
- B. More Examples of CLEVR-SV Dataset: Figure 9 provides additional StoryGAN samples using different first sentences and varied attributes for the first object.
C. Significance Test on Pororo-SV Dataset
The Pororo-SV ranking evaluation uses pairwise statistical tests to assess whether StoryGAN’s human-evaluated ranking advantage over baseline models is significant.
- C. Significance Test on Pororo-SV Dataset: Pairwise t-tests are performed on the human-evaluated ranking test.
- C. Significance Test on Pororo-SV Dataset: StoryGAN is statistically significant over the other baseline models in the human-evaluated ranking test.
- C. Significance Test on Pororo-SV Dataset: Table 6 reports p-values for the human-evaluated ranking test.
D. Characters Photo and More Examples of Pororo-SV Dataset
The Pororo-SV evaluation uses nine selected characters with corresponding profile photos. Additional test-set examples compare generated images with ground-truth story images, while allowing similar outputs from repeated input sentences.
- Character profiles: Nine characters—Pororo, Crong, Eddy, Poby, Loopy, Petty, Harry, Rody, and Tongtong—are selected for classification-accuracy comparisons.Their profile pictures are provided in Figure 10.
- Character profiles: Figure 10 presents the selected characters’ names alongside their corresponding dataset photos.
- More examples: Figure 11 shows Pororo-SV test-set samples, with generated images in the left five columns and ground-truth images in the right five.
- More examples: Generated samples need not exactly match ground truth, and similar images can result from repeated input sentences.