PaperlayerView exact revision on arXiv

Source-linked AI summary

Composing Concepts from Images and Videos via Concept-prompt Binding

Xianghao Kong, Zeyu Zhang, Yuwei Guo, Zhuoran Zhao, Songchun Zhang, Anyi Rao

arXiv:2512.09824v2cs.CVcs.AIcs.MM

TL;DR

Visual concept composition still struggles to extract complex concepts and flexibly combine elements from images and videos. BiCo binds visual concepts to prompt tokens and composes selected tokens from different sources; evaluations report gains in consistency, fidelity, and motion quality over prior approaches.

  • Problem

    Visual concept composition remains limited in accurately extracting complex visual concepts and flexibly combining concepts from both images and videos.

    Source: §1 · p. 2

  • Method

    BiCo is a one-shot method that binds visual concepts to prompt tokens and composes target prompts from bound tokens across sources.

    Source: §1 · p. 2

  • Results

    BiCo reports superior concept consistency, prompt fidelity, and motion quality; it outperforms prior methods across automatic metrics and human evaluations, with +54.67% subjective Overall Quality versus DualReal.

    Source: §1 · p. 2, §4.2.1 · p. 5

  • Takeaways & Limitations

    The authors present BiCo as a potential tool for creators to explore varied visual concept composition and video-creation tasks.

    Source: §1 · p. 2, §5 · p. 8

    BiCo treats prompt tokens equally, which can cause unintended concept drift when a concept is visually complex or differs substantially from the token's average appearance.

    Source: §G.1 · p. 14

Abstract

from arXiv · show

Visual concept composition, which aims to integrate different elements from images and videos into a single, coherent visual output, still falls short in accurately extracting complex concepts from visual inputs and flexibly combining concepts from both images and videos. We introduce Bind & Compose, a one-shot method that enables flexible visual concept composition by binding visual concepts with corresponding prompt tokens and composing the target prompt with bound tokens from various sources. It adopts a hierarchical binder structure for cross-attention conditioning in Diffusion Transformers to encode visual concepts into corresponding prompt tokens for accurate decomposition of complex visual concepts. To improve concept-token binding accuracy, we design a Diversify-and-Absorb Mechanism that uses an extra absorbent token to eliminate the impact of concept-irrelevant details when training with diversified prompts. To enhance the compatibility between image and video concepts, we present a Temporal Disentanglement Strategy that decouples the training process of video concepts into two stages with a dual-branch binder structure for temporal modeling. Evaluations demonstrate that our method achieves superior concept consistency, prompt fidelity, and motion quality over existing approaches, opening up new possibilities for visual creativity.

1. Introduction

Visual concept composition seeks coherent outputs assembled from image and video elements, but existing approaches struggle to extract complex concepts and flexibly combine both modalities. BiCo binds concepts to prompt tokens for composition, and experiments report gains in consistency, fidelity, motion quality, and video quality.

Motivation

  • Visual concept composition integrates elements from images and videos into one coherent visual output, a capability connected in the paper to visual creativity and filmmaking.The paper describes composition as combining inspirations into a new visual creation.Source: §1 · p. 2
  • Existing methods struggle to extract complex concepts, including occluded or non-object concepts, and prior image-video composition largely pairs image subjects with video motion.Flexible composition of attributes such as style and lighting across images and videos remains underexplored.Source: §1 · p. 2

Contribution

  • BiCo binds visual concepts to corresponding prompt tokens and composes selected tokens from different sources into a target prompt, enabling one-shot composition without explicit masks.The method uses a hierarchical binder, Diversify-and-Absorb Mechanism, and Temporal Disentanglement Strategy.Source: §1 · p. 2

Results

  • BiCo reports superior concept consistency, prompt fidelity, and motion quality, and outperforms prior baselines in concept-manipulation flexibility and composed-video quality.The paper positions the method as a potential tool for varied video-creation tasks.Source: §1 · p. 2

2. Related Work

Prior work spans text-to-video generation, video personalization, and visual concept composition, but leaves gaps in flexible decomposition and cross-modal composition. BiCo uses prompt tokens and binders to combine concepts from image and video sources.

T2V Diffusion Models

  • General text-to-video diffusion models provide satisfactory generation quality and consistency but have limited support for personalization and concept composition.The discussion describes DiT-based text conditioning and emerging text-to-video models.Source: §2 · p. 2

Visual Concept Composition

  • BiCo binds concepts to prompt tokens and composes them by routing tokens through different adapters to update the conditioning prompt.The overview depicts binding followed by prompt-token composition.Source: Figure 2 · p. 3
  • Prior video-composition methods support combining image subjects with video motion, while BiCo also supports composing broader attributes from image and video sources.The related-work discussion notes that previous approaches do not offer this broader flexibility.Source: §2 · p. 3

3. Methodology

BiCo binds visual concepts to prompt tokens and composes selected concepts through a hierarchical binder that conditions a text-to-video model. DAM diversifies prompts and absorbs distracting details, while TDS trains video concepts in two stages to address image-video temporal differences.

3.1. Overview

  • BiCo learns each source's visual appearance or motion in correspondence with prompt tokens, then routes designated prompt parts through their binders to condition composed output.The updated prompt tokens serve as cross-attention keys and values, injecting textual information into the DiT blocks.Source: §3.1 · p. 3

3.2. Hierarchical Binder Structure

  • The hierarchical structure uses global and per-block residual MLP binders to encode visual concepts into prompt tokens, with a dual-branch spatial-temporal structure for video inputs.A two-stage inverted training strategy is also used to optimize the hierarchical binders.Source: §3.2 · p. 4, Figure 3 · p. 4

3.3. Diversify-and-Absorb Mechanism (DAM)

  • DAM uses VLM-generated diverse prompts that retain key concepts, then adds a learnable absorbent token to capture distracting visual details and discards it during composition.Prompt construction separates spatial and temporal concepts; full-video second-stage training uses both, while image and first-stage video prompts use spatial concepts.Source: §3.3 · pp. 4–5

3.4. Temporal Disentanglement Strategy (TDS)

  • TDS trains video binders first on individual frames without temporal concepts and then on full videos with complete prompts, using spatial and temporal MLP branches.The second stage initializes from first-stage weights and fuses the branches with a learnable gate, a strategy reported to alleviate image-video temporal heterogeneity.Source: §3.4 · p. 5

4. Experiments

BiCo is evaluated against prior methods, through component ablations, and across creative applications. The reported comparisons favor BiCo, while the ablations and examples illustrate the roles of its components and the range of compositions it supports.

4.1. Implementation Details

  • BiCo uses Wan2.1-T2V-1.3B as its base model, trains binders for 2,400 iterations per stage, and generates 81-frame composed videos at inference.The stated binder learning rate is 1.0 × 10^-4, with noise threshold α set to 0.875.Source: §4.1 · p. 5

4.2. Comparisons to Prior Works

  • The comparisons cover Textual Inversion, DreamBooth-LoRA, DreamVideo, and DualReal, using 40 test cases and automatic metrics plus human ratings.Textual Inversion and DreamBooth-LoRA are adapted to the same text-to-video model; comparisons are limited to one image and one video per composition for fairness.Source: §4.2 · p. 5, §4.2.1 · p. 5

4.2.1. Quantitative Comparisons

  • BiCo outperforms all compared methods on automatic metrics and human evaluations, with +54.67% subjective Overall Quality versus DualReal.The comparison includes Table 1; the prose also reports flexibility beyond prior methods.Source: Table 1 · p. 5, §4.2.1 · p. 5

4.2.2. Qualitative Comparisons

  • BiCo best combines concepts according to the prompt while preserving input concepts; Textual Inversion and DreamVideo fail to combine them, DualReal is nearly static, and DB-LoRA drifts.Figure 5 presents the input concepts and composed prompts on the left.Source: §4.2.2 · pp. 5–6, Figure 5 · p. 6

4.3. Diagnostic Experiments

  • Ablations associate hierarchical binders with better concept preservation and motion, DAM’s absorbent token with fewer unwanted details, and TDS with higher Overall Quality and improved image-video detail preservation.The case study also reports that prompt diversification improves binding accuracy but can leave unwanted details, and that omitting two-stage inverted training makes optimization hard and unstable.Source: §4.3 · pp. 6–7, Table 2 · p. 6, Figure 6 · p. 7

4.4. Qualitative Results

  • BiCo examples include non-object concepts such as style and motion, and compositions involving multiple visual concepts, with satisfactory concept consistency, prompt fidelity, and motion quality.Figure 7 shows visual inputs in the upper row and composed videos in the lower row.Source: §4.4 · p. 7, Figure 7 · p. 8

4.5. Other Applications

  • BiCo supports visual-input decomposition by selecting desired concept tokens, and text-guided editing by passing unchanged elements through the binder while composing edited parts without updates.Figure 8 illustrates both applications.Source: §4.5 · pp. 7–8, Figure 8 · p. 8

5. Conclusion and Discussion

The conclusion presents BiCo as a one-shot approach for extracting and combining visual concepts across images and videos. It also identifies equal token treatment as a limitation and proposes adaptive weighting as future work.

Conclusion

  • BiCo binds visual concepts to prompt tokens, then composes a target prompt from tokens across sources using hierarchical binders, DAM, and TDS.The conclusion describes these components as supporting complex concept decomposition, more accurate binding, and image-video compatibility, respectively.Source: §5 · p. 8
  • The authors report that results across varied scenarios validate BiCo’s effectiveness and describe it as a tool for visual concept composition.Source: §5 · p. 8

Limitations

  • BiCo treats tokens equally despite uneven importance in text-to-video generation; adaptive designs for critical subject and motion tokens are future work.Source: §5 · p. 8

Supplementary Material

The supplementary material collects implementation details, additional results, analyses, and discussions. Its DAM appendix details VLM prompts for extracting concepts and generating diversified captions.

Supplementary Material

  • The supplement includes detailed DAM prompts, user-study settings, training-strategy details and ablations, absorbent-token justification, extra comparisons, case studies, and further limitations and societal-impact discussion.Source: Supplementary Material · p. 12

A. Detailed Prompts for DAM

  • For DAM, a VLM extracts spatial concepts from images and spatial plus temporal concepts from videos, then generates diversified prompts while retaining key conceptual words.Image and first-stage video training use image-caption prompts; second-stage video training uses prompts referring to spatial and temporal concepts and the video.Source: A. Detailed Prompts for DAM · pp. 12–13
  • The appendix’s prompt instructions call for concise, content-matching English captions, permitting rearrangement or paraphrase of extracted concepts.Source: A. Detailed Prompts for DAM · pp. 12–13

B. User Study Details

The supplementary material describes the user-study rating procedure and adds ablations of the two-stage inverted training strategy. It treats still frames without motion as low quality.

B. User Study Details

  • Study participants rate concept consistency, prompt fidelity, and motion quality on a five-point Likert scale across subsets of 10 test-case groups.Source: B. User Study Details · p. 13
  • Additional ablations assess two-stage global-binder-first training and inverted emphasis on high noise levels; both techniques are reported as crucial for satisfactory binder optimization.Source: Table 3 · p. 13
  • Still frames without motion are considered low quality in the user-study instructions.Source: B. User Study Details · p. 13

C. Extra Details and Ablations on Two-stage Inverted Training Strategy

The section describes inverted-training timestep sampling and reports that both the two-stage strategy and high-noise emphasis are crucial for satisfactory binder optimization.

C. Extra Details and Ablations on Two-stage Inverted Training Strategy

  • The timestep distribution assigns separate probability mass to positions below and above the threshold α.The two branches weight the intervals by (1 - β) and β, respectively, normalized by the number of timesteps in each interval.Source: C. Extra Details and Ablations on… · p. 13
  • Setting β = α exchanges probability mass between higher- and lower-noise levels, with α = 0.875 chosen to distinguish them.The section reports satisfactory performance with this setting because higher noise levels originally receive a smaller probability mass.Source: C. Extra Details and Ablations on… · p. 13
  • Both two-stage training and high-noise emphasis are crucial for satisfactory binder optimization.Two-stage training trains the global binder before the full hierarchical structure; inverted training emphasizes higher noise levels.Source: C. Extra Details and Ablations on… · p. 13

D. Analysis on Absorbent Token

The absorbent token captures irrelevant details while its removal strengthens target attention. Additional comparisons and a component case study report BiCo’s performance across composition tasks and attribute improvements to individual components.

D. Analysis on Absorbent Token

E. Additional Qualitative Comparisons

  • In motion transfer, BiCo best follows the prompt while preserving input concepts; the compared methods show composition failures, leakage, or visual drift.Textual Inversion and DreamVideo fail to combine concepts; DualReal shows inadequate preservation and leakage, while DB-LoRA shows concept drift.Source: E. Additional Qualitative Comparisons · p. 14
  • All compared prior methods fail to learn and compose the line-art style with the video subject in the style-transfer example.The passage presents this example as evidence of BiCo’s versatile controllability.Source: E. Additional Qualitative Comparisons · p. 14

F. Extra Case Study

  • The component case study attributes stronger preservation to hierarchical binders and DAM, better image-video compatibility to TDS, and steadier optimization to two-stage inverted training.The reported comparisons include improved results at the same optimization steps for the two-stage inverted strategy.Source: F. Extra Case Study · p. 14

G.1. Limitations

BiCo can drift on visually unusual concepts and can fail when composition requires common-sense reasoning.

G.1. Limitations

  • Equal treatment of prompt tokens can cause concept drift when a concept is visually complex or differs greatly from the text token’s average appearance.The colorful whimsical hat example is not accurately reproduced; the authors plan adaptive designs to highlight critical tokens.Source: §G.1 · p. 14
  • Compositions requiring common-sense reasoning can fail, as in the Doberman example that adds a leg rather than raising an existing one.The resulting dog has five legs; the authors suggest VLM reasoning may alleviate this issue.Source: §G.1 · p. 14

G.2. Societal Impacts

BiCo is presented as a way for individual creators and commercial teams to combine visual assets in their workflows. The authors also warn that realistic fabricated media can complicate authenticity verification and raise privacy concerns.

G.2. Societal Impacts

  • BiCo’s one-shot workflow lets individual creators compose visual concepts without extensive training and offers commercial teams a way to combine intermediate results and other assets.The authors state this may boost the novelty of produced visual content.Source: §G.2 · p. 14
  • Realistic fabricated images and videos can make authenticity verification more challenging and raise privacy concerns when generated without authorization featuring an individual.The authors note that such content can distort public perception.Source: §G.2 · p. 14

Related papers

Loading 2512.09824v2…