PaperlayerView exact revision on arXiv

Source-linked AI summary

Sidecar: Training-Free Semantic Reuse for Character-Consistent Free-form Visual Storytelling

Sibo Dong, Sarah Adel Bargal

arXiv:2608.27280v1cs.CV

TL;DR

Free-form storytelling uses abbreviated later character references, which can leave identity semantics incomplete and make consistency difficult. Sidecar extracts entity-level information from the initial description and injects it into later prompt encoding without training or architectural changes. On FreeStoryBench, it improves prompt-image alignment and character consistency across SDXL- and FLUX-based models with negligible overhead.

  • Problem

    Later type-level mentions or pronouns omit identity attributes from prompt representations, leaving free-form story conditions semantically under-specified.

  • Method

    Sidecar extracts identity-related representations from the initial character description and injects them into later prompt representations as a plug-and-play, training-free augmentation.

  • Results

    Sidecar improves prompt-image alignment and character consistency across SDXL- and FLUX-based models and existing storytelling methods, with negligible computational overhead.

  • Takeaways & Limitations

    Text-side semantic reuse complements visual consistency mechanisms by addressing identity information missing from later free-form prompts.

  • Takeaways & Limitations

    Sidecar relies on clear initial character descriptions, while the evaluation focuses on single-main-character stories and remains subject to CLIP's 77-token context limit.

Abstract

from arXiv · show

Visual storytelling requires generating images that follow a narrative while preserving consistent character identities across frames. In free-form story generation, a character is fully described only when first introduced and is later referred to by a type-level mention or pronoun. Although this setting better reflects natural storytelling, later prompts may omit important identity-related semantics, making character consistency more difficult to maintain. We propose \textbf{Sidecar}, a plug-and-play semantic augmentation module that preserves entity-level information from the initial description and injects the missing semantics into later prompt embeddings. Sidecar requires no additional training and does not modify the architecture of the base diffusion model. Experiments on FreeStoryBench show that Sidecar consistently improves prompt-image alignment and character consistency across multiple SDXL- and FLUX-based baselines, with negligible computational overhead.

1. Introduction

Free-form visual storytelling makes later character references more natural but can omit identity semantics needed for consistency. Sidecar restores those semantics through plug-and-play text augmentation, improving alignment and consistency without training or architectural changes.

  • Visual storytelling must satisfy both prompt-image alignment and cross-frame consistency, unlike independent single-image T2I generation.
  • Free-form stories describe a character fully once, then use shorter types or pronouns that better match natural narrative structure and reduce prompt length.
  • Later type-level mentions and pronouns can remove explicit identity attributes, leaving prompt embeddings semantically incomplete despite visual feature reuse.
  • Sidecar preserves entity-level semantics from the initial description and injects them into later text representations alongside the original prompt.The original prompt continues to specify the action and scene, while Sidecar supplements omitted identity information.
  • Sidecar requires no additional training, does not modify the base diffusion architecture, and can integrate with pretrained generation pipelines and existing storytelling methods.
  • Sidecar consistently improves prompt-image alignment and character consistency across SDXL- and FLUX-based models on FreeStoryBench, with negligible additional GPU memory and inference time.

2. Related Work

Prior storytelling methods use training, visual feature reuse, or expanded textual conditions to preserve consistency. Sidecar targets their complementary limitation by reusing initial entity semantics in later free-form prompts without rewriting them or modifying the diffusion model.

  • Story-level approaches improve visual consistency through previous prompts or images and additional modules that aggregate multimodal history.
  • These approaches require task-specific training, story-level supervision, trainable components, or changes to the original generation architecture.
  • Training-free methods preserve subject identity by reusing attention or intermediate visual features across generated images.
  • Other training-free methods modify textual conditions by concatenating prompts, replacing identity prompts, or reconstructing descriptions through prompt rewriting.
  • FreeStory uses a detailed first prompt and shorter later references, but later text remains under-specified because types and pronouns omit detailed identity information.
  • Sidecar complements visual feature reuse by preserving first-prompt entity semantics and injecting them into later representations without rewriting prompts or modifying the base model.

3. Method

Sidecar augments later prompt representations with entity-level semantics extracted from the initial character description, while preserving the original text-conditioning interface. It uses temporary, layer-wise semantic tokens and encoder-specific integration to support compatibility across generation pipelines.

  • Semantic extraction: Sidecar extracts layer-wise hidden representations from the initial character description and stores them separately for each encoder layer.The extracted representations preserve token-level semantics at different representation depths.
  • Semantic injection: During later-prompt encoding, Sidecar inserts the stored representations as temporary semantic tokens that current prompt tokens can access through self-attention.The current prompt retains its action and scene while retrieving omitted identity information.
  • Semantic injection: The temporary tokens are removed after each Transformer layer, so the resulting representation retains the original sequence length and hidden dimension.Layer-specific semantic representations are inserted only within their corresponding layer and do not accumulate across the encoder.
  • Pipeline integration: Sidecar operates in the text-conditioning stage and leaves the image-generation network and existing visual-consistency mechanisms unchanged.When combined with existing methods, Sidecar complements their visual-feature propagation across story frames.
  • Encoder-specific integration: For different backbones, Sidecar applies layer-wise augmentation to SDXL's CLIP encoders and combines FLUX's first and current prompts for T5 encoding.The FLUX pathway retains only hidden states corresponding to the current prompt after joint contextual encoding; the SDXL positive condition receives augmentation while the negative prompt does not.
  • Pipeline integration: Sidecar introduces no trainable parameters or additional optimization and preserves the original text-conditioning interface for integration with vanilla and training-free pipelines.This design supports use with both pretrained generation backbones and existing storytelling methods.

4. Experiments

Sidecar is evaluated on FreeStoryBench across SDXL- and FLUX-based models, reference types, and text-encoder integration strategies. It consistently improves alignment and character consistency with negligible overhead, while ablations clarify where its benefits are strongest and what limits its scope.

  • Experimental Setup: The evaluation uses 100 single-character stories with six scenes under mixed prompting, where only the first prompt contains the complete description.Subsequent prompts use type-level mentions and pronouns, creating a challenging free-form consistency setting.
  • Results on FreeStoryBench: Sidecar consistently improves prompt-image alignment and character consistency across vanilla, storytelling, SDXL-based, and FLUX-based baselines.The evaluated baselines include SDXL, StoryDiffusion, ConsiStory, FLUX, and FreeStory.
  • Results on FreeStoryBench: 0.776 to 0.859 in CLIP, 0.308 to 0.621 in DINO, and 0.540 to 0.333 in DreamSim for vanilla SDXL with Sidecar.Higher CLIP and DINO indicate better alignment and consistency, while lower DreamSim indicates better perceptual consistency.
  • Efficiency: Sidecar adds negligible computational overhead, with SDXL runtime increasing by 0.2% and peak memory remaining unchanged.On ConsiStory, runtime and peak memory increases are 0.9% and 0.5%, respectively.
  • Ablation Study: Pronoun prompts yield larger gains than type-level mentions, with FreeStory DINO improvement increasing from 0.077 to 0.309.FreeStory DreamSim improvement similarly increases from 0.061 to 0.206 for pronouns.
  • Ablation Study: CLIP Sidecar outperforms T5 concatenation alone and direct T5 Sidecar, while replacing CLIP concatenation with Sidecar improves CLIP/DINO/DreamSim from 0.860/0.594/0.324 to 0.869/0.649/0.291.The results indicate that integration strategies differ across text encoders.
  • Ablation Study: Layer-wise interaction between reused description semantics and the current action and scene outperforms post-encoder replacement.Both approaches restore omitted identity information, but Sidecar enables interaction within each CLIP layer.
  • Limitations: Sidecar relies on a clear initial character description, focuses evaluation on single-character stories, and remains subject to CLIP’s 77-token context limit.Multiple interacting characters require reliable entity-specific extraction and disambiguation.

5. Conclusion

Sidecar is a training-free semantic augmentation module for free-form visual storytelling that restores identity information omitted from later prompts. It improves character consistency and prompt-image alignment across SDXL- and FLUX-based models without additional training or substantial overhead.

  • Sidecar extracts identity-related representations from the initial character description and injects them into later prompt encodings.
  • Sidecar requires no additional training, adds negligible computational overhead, and integrates with vanilla diffusion models and existing storytelling methods.
  • Experiments on FreeStoryBench show improved character consistency and prompt-image alignment across SDXL- and FLUX-based models.

A. Multi-Character Evaluation

Sidecar remains effective for stories containing multiple characters by constructing entity-specific reference semantics independently. Across evaluated baselines, it improves consistency and prompt fidelity without evident interference between characters.

  • The multi-character evaluation uses 100 stories and 600 frames from FreeStoryBench.Each character receives an independently constructed Sidecar representation from its structured description span.
  • Independent entity-specific representations preserve character semantics without evident interference between characters.
  • Sidecar consistently improves performance across all evaluated baselines, with substantial DINO increases and corresponding DreamSim reductions.
  • CLIP alignment improves for every baseline, indicating that stronger multi-character consistency does not come at the expense of prompt fidelity.

B. Effect of Sidecar Placement Across Encoder Layers

Sidecar placement deeper in the CLIP text encoder is more effective than placement in early layers. Applying Sidecar throughout the encoder achieves the best performance.

  • The ablation compares Sidecar applied to the early, middle, or late one-third of SDXL’s CLIP encoder layers against the baseline and default all-layer configuration.
  • Late-layer application produces much larger improvements in character consistency and prompt alignment than early-layer application.Early layers provide little improvement, while middle layers yield moderate gains.
  • The results suggest that deeper CLIP layers form higher-level contextual representations important for recovering character semantics.
  • Applying Sidecar throughout the entire encoder consistently achieves the best performance.

C. Computational Efficiency

Sidecar adds minimal computational overhead while improving character consistency. Its text-conditioning-only design leaves the image-generation backbone unchanged.

  • Sidecar introduces only marginal computational overhead across the evaluated methods.The study measures peak allocated GPU memory and average inference time under identical generation settings.
  • Peak GPU memory remains nearly unchanged with Sidecar enabled for SDXL, StoryDiffusion, and FreeStory.
  • Sidecar is computationally lightweight because it augments only text conditioning while leaving the image-generation backbone unchanged.
  • Qualitative FreeStoryBench comparisons report improved identity preservation but no guarantee of complete visual consistency.
Loading 2608.27280v1…