PaperlayerView exact revision on arXiv

Source-linked AI summary

Not Just a Subject: Capturing Adaptive Scene-Specific Variance for Visual Storytelling

SeungJu Cha, Ye-Chan Kim, Kwanyoung Lee, Dong-Jin Kim

arXiv:2610.22320v1cs.CV

TL;DR

Visual-storytelling methods often prioritize identity consistency without adequately adapting to scene-specific semantics and structure. EOS-Story combines prompt decomposition and phrase-level appearance injection with scale-aware identity transfer, and is reported to achieve competitive or superior performance on both benchmarks. Its ⟨EOS⟩-based bridging is scoped to encoders where that token has a comparable contextual role.

  • Problem

    Existing visual-storytelling methods often underrepresent scene-specific semantic and visual variance, weakening narrative fidelity and structural adaptation.

    Source: §1 · p. 2

  • Method

    EOS-Story decouples identity and scene prompts, injects phrase-level appearance residuals into the scene’s ⟨EOS⟩ embedding, and adapts identity-feature transfer to scene scale.

    Source: §1 · p. 3, §5 · p. 14

  • Results

    EOS-Story achieves competitive or superior performance on the original and extended benchmarks, with consistent improvements over existing approaches reported on both.

    Source: §1 · p. 4, §5 · p. 14

  • Takeaways & Limitations

    The paper argues that coherent visual storytelling requires harmonizing identity consistency with each scene’s semantic and structural variance.

    Source: §1 · p. 2

    Cross-Contextual Bridging does not directly transfer to T5 or LLM-based text encoders, where ⟨EOS⟩ lacks a comparable role.

    Source: D Limitation & Future Work · p. 30

Abstract

from arXiv · show

Recent advancements in diffusion models have significantly improved text-to-image generation, enabling their application to visual storytelling, where coherent image sequences are generated from narrative prompts. However, existing methods primarily focus on identity preservation and often fail to adapt to diverse scene variations, resulting in semantic misalignment and structural inflexibility. We propose EOS-Story, a training-free framework that harmonizes identity consistency with scene-specific adaptation across text and vision spaces. In the text space, we decouple the narrative into identity and scene sub-prompts to prevent information dilution, but this leaves the scene without access to appearance cues. To bridge this gap, we introduce Cross-Contextual Bridging, which softly injects a phrase-level appearance residual into the scene's <EOS> embedding. In the vision space, we find that scale mismatch between identity and scene subjects causes a few identity tokens to be repeatedly transferred, producing structural artifacts. We address this with Adaptive Identity Preservation, which anchors identity features at the scene-specific scale (Identity Anchoring) and integrates them via Identity-Aware Self-Attention. To evaluate performance under varied spatial compositions, we extend the Consistory+ benchmark with diverse framing scenarios (Consistory+_+). Extensive experiments show that EOS-Story achieves competitive or superior performance on both benchmarks.

1 Introduction

Visual storytelling must preserve a coherent subject while adapting to each scene’s semantics and structure. EOS-Story addresses this through complementary text- and vision-space mechanisms and is reported to improve performance on the original and extended benchmarks.

1 Introduction

  • Existing methods preserve subject identity but often underrepresent scene-specific semantic and visual variance, weakening prompt alignment and structural flexibility.Monolithic prompts can dilute scene cues, while spatially blind feature transfer can produce rigid or mismatched subject structures.Source: §1 · pp. 1–2
  • Cross-Contextual Bridging separates identity and scene prompts, then injects phrase-level appearance residuals into the scene’s ⟨EOS⟩ embedding.The scene prompt remains a semantic anchor while receiving appearance cues without explicit word-level repetition.Source: §1 · p. 3
  • Adaptive Identity Preservation resizes identity features to the scene’s scale and integrates them through Identity-Aware Self-Attention.The approach addresses scale mismatch that can cause repeated transfer of only a few identity tokens and structural artifacts.Source: §1 · p. 3
  • Consistory+_+ extends Consistory+ with diverse framing prompts to evaluate scene-specific structural variation.The authors report validating EOS-Story on both Consistory+ and Consistory+_+ under framing variations.Source: §1 · pp. 3–4
  • EOS-Story demonstrates consistent improvements over existing approaches on both the original and extended benchmarks.The extended Consistory+_+ benchmark evaluates performance under diverse framing variations.Source: §1 · pp. 3–4

2 Related Work

Prior visual-storytelling methods commonly emphasize identity preservation, while the paper argues that faithful scene adaptation also requires attention to text-space semantics. EOS-Story distinguishes its phrase-level use of ⟨EOS⟩ embeddings from earlier embedding-manipulation approaches.

Visual Storytelling

  • Training-free storytelling methods such as Consistory and 1P1S [17] transfer attention features to preserve identity, but can produce rigid depictions and weak prompt alignment.Other approaches fine-tune diffusion models to capture identity explicitly.Source: §2 · p. 4
  • Personalization methods [7] [36] focus on subject appearance, often overlooking adaptation across visual contexts.Source: §2 · p. 4

⟨EOS⟩ Embedding in Diffusion Models

  • Prior EOS methods substitute embeddings for single-word semantic changes [20] or manipulate embeddings to suppress content, whereas EOS-Story transfers phrase-level appearance into scene prompts.Source: §2 · pp. 4–5

3 Method

EOS-Story models scene-specific variation in text and vision spaces: it transfers appearance cues through the scene’s ⟨EOS⟩ embedding and aligns identity features to each scene’s subject structure. Its visual-space approach combines scale-aware identity anchoring with attention-based refinement.

3.1 EOS Embedding as a Semantic Handle

  • Injecting an attribute into the ⟨EOS⟩ embedding shifts semantics more clearly than injecting it into the target word token or all non-EOS tokens.Addition and replacement at the ⟨EOS⟩ level induce more reliable coherent shifts than simple subtraction, which can leave original word-level semantics intact.Source: §3.1 · pp. 5–6, Figure 2 · p. 5

3.2 Cross-Contextual Bridging

  • Cross-Contextual Bridging constructs style-subject, identity, and scene prompts, then adds the identity-to-style-subject appearance residual to the scene’s ⟨EOS⟩ embedding.This restores appearance cues without lengthening the scene description or modifying its word-level tokens, while retaining the scene semantics.Source: §3.2 · p. 7

3.3 Adaptive Identity Preservation

  • Scale mismatch concentrates scene-to-identity correspondence on a few identity tokens, causing structural artifacts; resizing identity features to scene-subject scale supports more reliable matching.Identity Anchoring also filters matches using a threshold derived from the similarity distribution and blends matched identity features with scene features.Source: §3.3 · p. 8
  • Identity-Aware Self-Attention concatenates size-aligned identity keys and values with scene keys and values so scene features can attend to identity information during refinement.This follows feature blending, which may not integrate naturally with the surrounding scene context.Source: §3.3 · p. 9

4 Experiments

EOS-Story performs well across text alignment, image quality, subject similarity, and framing accuracy, while its ablations show complementary contributions from text- and vision-space adaptation. Experiments also find that EOS-level appearance injection improves alignment and size matching promotes structural coherence.

4 Experiments

  • Experiments use SDXL-base-1.0 with 50 denoising steps and evaluate alignment, subject similarity, aesthetics, detail fidelity, and framing.Consistory+_+ adds shot-size, camera-angle, and composition phrases; ShotVL-7B assesses framing fidelity.Source: §4 · pp. 9–10

4.1 Main Experiments

  • EOS-Story improves text alignment across Consistory+ and Consistory+_+ without compromising image quality, and leads on framing accuracy on Consistory+_+.CLIP-I remains competitive with the highest score, while DreamSim and image-quality metrics also favor EOS-Story.Source: Table 1 · p. 10, §4.1 · p. 10, Figure 5 · p. 10

4.2 Effectiveness of Cross-Contextual Bridging

  • Injecting information only into the ⟨EOS⟩ embedding achieves the highest text–image alignment and perceptual similarity among tested token positions.Subject-token injection fails to reliably encode identity cues, while distributing information across non-EOS tokens yields limited gains.Source: §4.2 · pp. 11–12, Table 2 · p. 13
  • The residual formulation eossc+∆app aligns best with both appearance and scene subspaces and with CLIP-T among the tested EOS variants.The comparison includes direct addition and an orthogonalized appearance component; eosinf also concentrates attention on relevant details such as a green knitted hat.Source: §4.2 · pp. 12–13, Table 3 · p. 13

4.3 Ablation Studies

  • Cross-Contextual Bridging primarily improves CLIP-T, while Adaptive Identity Preservation contributes more to CLIP-I; together, EOS-Story outperforms configuration (7) in CLIP-T and DreamSim.Size matching supports coherent subject integration; without it, scale discrepancies can misalign identity transfer and distort subject proportions.Source: §4.3 · pp. 13–14, Figure 10 · p. 14, Table 5 · p. 14

5 Conclusion

EOS-Story is a training-free framework for modeling scene-specific variance in visual storytelling across text and vision spaces. Its experiments show competitive or superior performance in prompt fidelity and structural adaptability on Consistory+ and Consistory+_+.

5 Conclusion

  • EOS-Story models scene-specific variance across text and vision spaces without training.Cross-Contextual Bridging injects phrase-level appearance residuals into the scene’s ⟨EOS⟩ embedding, while Adaptive Identity Preservation anchors subject features at the scene-specific scale and refines integration through Identity-Aware Self-Attention.Source: §5 · p. 14
  • Consistory+_+ extends Consistory+ with diverse framing variations to evaluate scene adaptability.The framework addresses scene-specific variation through complementary text- and vision-space modules.Source: §5 · p. 14
  • EOS-Story achieves competitive or superior performance in prompt fidelity and structural adaptability on both benchmarks.The conclusion characterizes the experiments as extensive.Source: §5 · p. 14

Supplementary Materials Not Just a Subject: Capturing Adaptive Scene-Specific Variance for Visual Storytelling

The EOS-linearity analysis tests whether appearance residuals behave as transferable semantic directions. The figure illustrates subject-independent alignment, transfer toward an explicit-prompt embedding, and controllable appearance strength.

Empirical Analysis of EOS Linearity

  • Appearance residuals for the same attribute stay aligned across subjects, while residuals for different attributes separate.This pattern indicates a subject-agnostic direction.Source: Figure 11 · p. 18
  • A transferred residual matches the explicit-prompt embedding more closely than the appearance-free base or a random-attribute embedding.The figure labels the explicit-prompt embedding as the transfer target.Source: Figure 11 · p. 18
  • Increasing λ progressively strengthens the target appearance, illustrated with round glasses.The figure describes the influence as controllable.Source: Figure 11 · p. 18

A EOS Analysis

The supplementary EOS analyses test whether appearance residuals transfer across subjects and prompts, whether semantic subspaces are compact, and whether EOS variants preserve meaning. They report subject-agnostic transfer behavior, high similarity to original embeddings, and stronger alignment from structured residual injection.

Empirical Analysis of EOS Linearity

  • Appearance residuals from the same attribute align across subjects, while different attributes separate; transfer also matches explicit prompts, even for long compositional phrases.Increasing the residual weight progressively strengthens the target appearance in generated images.Source: A EOS Analysis · p. 19, Figure 14 · p. 20, Figure 15 · p. 20

Subspace Construction

  • Capturing 90% of cumulative variance takes 81 dimensions for appearance and 221 for scene, from 768 CLIP text-embedding dimensions.The analysis constructs the subspaces from appearance- and scene-related phrases in Consistory+.Source: Figure 12 · p. 19, A EOS Analysis · p. 20

Additional Analysis on eosinf

  • eosinf remains close to eosorg, with pairwise cosine similarity µ = 0.9260; qualitative attention maps show stronger localization on semantically relevant regions.The t-SNE visualization places infused embeddings near their corresponding original embeddings.Source: Figure 13 · p. 19, A EOS Analysis · p. 20, Figure 14 · p. 20

Text Encoder Robustness

  • With the 1024-dimensional text encoder, eosorg and eosinf retain high pairwise cosine similarity (µ = 0.9407).The analysis presents this as evidence that Cross-Contextual Bridging works across different text-encoder configurations.Source: A EOS Analysis · p. 20, Figure 14 · p. 20

Additional Variants of the EOS-Based Control

  • Structured injection of ∆app into scene-grounded EOS anchors aligns best across appearance and scene subspaces and improves text alignment over replacement or naive EOS combinations.The analysis attributes lower performance from direct ∆app replacement to the absence of scene information in that embedding.Source: A EOS Analysis · p. 21, Table 6 · p. 21

B Consistory+_+ & Framing Evaluation Details

Consistory+_+ extends the benchmark with framing scenarios selected for structural and compositional relevance to static images. The section describes how phrases were filtered, added to prompts, and evaluated.

B Consistory+_+ & Framing Evaluation Details

  • The extension focuses on three framing categories, excluding descriptors redundant with subject or shot-size information, video-only camera movement, and lighting variations outside the study’s scope.The selection criteria target structural and compositional variance in static-image generation.Source: B Consistory+_+ & Framing Evaluation… · p. 22
  • About 90 scene prompts and 10 random seeds per phrase were used to measure top-1 and top-3 framing accuracy, and phrases with zero top-1 accuracy were filtered out.The example shot-size evaluation generated 6,300 images.Source: B Consistory+_+ & Framing Evaluation… · p. 23
  • GPT-4o assigns the most appropriate remaining framing phrase to animate-subject prompts, and generated images are evaluated by multiple-choice framing-category accuracy.For each prompt, GPT-4o first determines whether the subject is animate.Source: B Consistory+_+ & Framing Evaluation… · p. 23

C.1 Ablation Studies

The ablations examine layer placement, adaptive thresholds, blending strength, EOS substitution, and the switching timestep. Results indicate that decoder up-blocks and scene-specific parameter choices matter for identity and prompt alignment.

C.1 Ablation Studies

  • The adaptive threshold µ + σ adjusts to each scene and yields more stable identity integration without artifacts such as duplicated features.A fixed α can produce noisy or discarded correspondences when similarity distributions vary across scenes.Source: §C.1 · p. 23, Figure 17 · p. 24
  • Decreasing λ injects more subject-relevant features and improves identity capture; α = µ + σ and λ = 0.55 provide the selected CLIP-T/CLIP-I trade-off.The figure describes decreasing identity preservation as λ increases.Source: §C.1 · p. 23, Figure 18 · p. 24
  • Replacing all remaining EOS embeddings often degrades image quality and produces unnatural scene rendering, so EOS substitution is limited.The authors attribute this to redundant perturbations at padded EOS positions.Source: §C.1 · p. 23, Table 9 · p. 25, Figure 19 · p. 25
  • CLIP-I surpasses 0.880 when Identity Anchoring and Identity-Aware Self-Attention include decoder up-block layers.The experiments compare CLIP-T and CLIP-I across layer configurations.Source: §C.1 · p. 24, Figure 20 · p. 25
  • Switching from Identity Anchoring to Identity-Aware Self-Attention at t = 9 balances prompt alignment and identity consistency.The reported trade-off is between CLIP-T and CLIP-I.Source: Figure 21 · p. 25, §C.1 · p. 25

C.2 Extended Experiments

Extended experiments test EOS-Story across model variants, conditioning and prompt formats, diffusion architectures, and multi-character stories. They also report user preferences and generation costs.

C.2 Extended Experiments

  • Across Juggernaut-X-v10, Playground-v2.5, and RealVisXL, generated images consistently reflect intended semantics with and without framing phrases.The influence of framing can appear subtle when the original scene already has a similar layout.Source: §C.2 · p. 25, Figure 22 · p. 26
  • With ControlNet, EOS-Story generates subject-coherent image sequences with diverse poses.Source: Figure 23 · p. 26, §C.2 · p. 26
  • On SD3, EOS-Story consistently improves alignment after adapting attention to combine scene and subject-referenced paths without changing positional encodings.The method is evaluated using Stable Diffusion 3.5 Large.Source: Table 10 · p. 26, §C.2 · p. 27, Figure 25 · p. 27
  • GPT-4o parses free-form user instructions into EOS-Story’s intended prompt format, supporting more flexible inputs.Source: Figure 24 · p. 26, §C.2 · p. 26
  • On FLUX, EOS-Story is competitive or superior to CharaConsist across identity and prompt-alignment metrics on Consistory+, while FLUX-Kontext preserves identity but struggles with scene prompts.Because FLUX uses a T5 encoder whose EOS marker has negligible generation influence, EOS-Story applies only its model-agnostic AIP module.Source: §C.2 · pp. 27–28, Table 12 · p. 28, Figure 26 · p. 28
  • EOS-Story generates coherent, visually consistent sequences that capture multiple characters’ identities and the intended prompt.Source: Figure 27 · p. 29, §C.2 · p. 29
  • EOS-Story was selected as best in 52.8% of user-study cases; it uses slightly more memory than baselines while maintaining comparable latency.The study involved 20 participants and 40 storyboards.Source: §C.2 · p. 29

C.3 Additional Qualitative Results

Additional qualitative results show EOS-Story preserving subject identity while aligning images with prompts across complex framing scenarios. The method also supports storyboards of varying lengths and responds to different framing phrases.

C.3 Additional Qualitative Results

  • In complex scenes with diverse framing phrases, EOS-Story preserves subject identity while improving alignment with prompts.Source: §C.3 · p. 29
  • EOS-Story supports short, medium, and long storyboards.Source: §C.3 · p. 29
  • Generated framing structures adapt to the supplied framing phrases.Source: §C.3 · p. 29

D Limitation & Future Work

EOS-Story’s limitations include occasional cross-attention mislocalization, SDXL generation failures, and reliance on an ⟨EOS⟩ role not shared by T5 or LLM-based encoders; the paper identifies subject masks and broader encoder support as directions for addressing these boundaries.

D Limitation & Future Work

  • Cross-attention maps can miss the subject area and misplace generations; explicitly conditioning on the subject mask is a proposed remedy.The maps may also capture irrelevant regions.Source: D Limitation & Future Work · pp. 29–30
  • 52.8% of 20 participants preferred EOS-Story for balancing identity, prompt alignment, and scene adaptability.Participants chose among four methods’ outputs generated from the same storyboard.Source: Figure 28 · p. 30, Figure 29 · p. 30
  • SDXL sometimes fails to generate accurate images, such as the “faun” example, which the authors describe as an underlying-backbone limitation.Source: D Limitation & Future Work · p. 30
  • Cross-Contextual Bridging does not directly transfer to T5- or LLM-based encoders, motivating analogous residual-based semantic composition for those models.Source: D Limitation & Future Work · p. 30
  • In long-story examples, EOS-Story reflects detailed scene descriptions while preserving subject identity.Source: Figure 35 · p. 34

Related papers

Loading 2610.22320v1…