PaperlayerView exact revision on arXiv

Source-linked AI summary

A Neural Space-Time Representation for Text-to-Image Personalization

Yuval Alaluf, Elad Richardson, Gal Metzer, Daniel Cohen-Or

arXiv:2305.15391v1cs.CV

TL;DR

Text-to-image personalization must represent concepts while balancing visual fidelity, editability, and storage. NeTI addresses this by implicitly learning a time- and layer-dependent representation with a neural mapper, textual bypass, and ordered outputs. The method generates high-quality, controllable compositions across concepts and prompts without fine-tuning the generative model.

  • Problem

    Personalization requires a concept representation that balances visual fidelity, downstream editability, and storage, while a time-and-layer-dependent space is difficult to optimize directly.

  • Method

    NeTI uses a small neural mapper to output time- and layer-dependent embeddings, adds a textual-bypass residual, and imposes an ordered representation for controllable reconstruction-editability tradeoffs.

  • Results

    NeTI demonstrates effective, high-quality, and controllable compositions across concepts and prompts without fine-tuning generative-model parameters, outperforming other inversion methods and remaining competitive with DreamBooth.

  • Takeaways & Limitations

    A single compact learned representation supports control over reconstruction and editability while preserving composition with other prompt tokens.

  • Takeaways & Limitations

    The dropout-based reconstruction-editability traversal requires a separate inference pass for each dropout value, and concepts are still optimized independently over hundreds of steps.

Abstract

from arXiv · show

A key aspect of text-to-image personalization methods is the manner in which the target concept is represented within the generative process. This choice greatly affects the visual fidelity, downstream editability, and disk space needed to store the learned concept. In this paper, we explore a new text-conditioning space that is dependent on both the denoising process timestep (time) and the denoising U-Net layers (space) and showcase its compelling properties. A single concept in the space-time representation is composed of hundreds of vectors, one for each combination of time and space, making this space challenging to optimize directly. Instead, we propose to implicitly represent a concept in this space by optimizing a small neural mapper that receives the current time and space parameters and outputs the matching token embedding. In doing so, the entire personalized concept is represented by the parameters of the learned mapper, resulting in a compact, yet expressive, representation. Similarly to other personalization methods, the output of our neural mapper resides in the input space of the text encoder. We observe that one can significantly improve the convergence and visual fidelity of the concept by introducing a textual bypass, where our neural mapper additionally outputs a residual that is added to the output of the text encoder. Finally, we show how one can impose an importance-based ordering over our implicit representation, providing users control over the reconstruction and editability of the learned concept using a single trained model. We demonstrate the effectiveness of our approach over a range of concepts and prompts, showing our method's ability to generate high-quality and controllable compositions without fine-tuning any parameters of the generative model itself.

1. Introduction

The paper introduces NeTI, which represents concepts in a text-conditioning space varying across denoising time and U-Net layers. A neural mapper, textual bypass, and ordered representation improve expressiveness, fidelity, and control without tuning the generative model.

  • Contribution: NeTI represents concepts in P∗, a space dependent on both denoising timestep and U-Net layer, using a small neural mapper instead of hundreds of independently optimized vectors.The mapper receives time and layer parameters and outputs the corresponding token embedding, compactly encoding the concept in its learned parameters.
  • Contribution: A textual bypass adds a learned residual to the text encoder output, addressing the encoder’s limitations while retaining composition with other prompt tokens.The mapper outputs one vector for the text encoder and another residual that is added after encoding.
  • Contribution: Norm control improves editability, while an ordered representation lets users adjust reconstruction quality and editability at inference time.The method imposes an importance-based ordering so a single trained model supports different reconstruction-editability balances.
  • Evaluation: Across concepts and prompts, the authors report effective P∗ and NeTI representations and demonstrate controllable reconstruction-editability tradeoffs.The evaluation also analyzes attributes learned at different denoising timesteps.

2. Related Work

Prior work personalizes text-to-image models through inversion, fine-tuning, or encoder-based mappings. NeTI extends inversion by modeling text conditioning across both U-Net layers and denoising time, with an additional textual bypass.

  • Personalization: Personalization adapts text-to-image models to synthesize novel images of a specified subject or concept from free-form prompts.Applications include image editing and personalized 3D generation.
  • Inversion spaces: Textual Inversion uses one embedding in P, while P+ assigns embeddings to U-Net layers and improves reconstruction and editability over P.Both spaces treat the representation as independent of denoising timestep.
  • Inversion spaces: NeTI introduces P∗, which varies across attention layers and denoising time, and extends prior inversion spaces with a textual bypass outside the text encoder input space.The paper distinguishes this joint space-time representation from prior layer-dependent or timestep-conditioned approaches.

3. Preliminaries

Stable Diffusion encodes images into latent representations and generates them by denoising while conditioned on text embeddings. Textual Inversion adapts this process by optimizing a concept token embedding while leaving the latent diffusion model fixed.

  • Latent Diffusion Models: Stable Diffusion encodes an image into a spatial latent code, trains diffusion in that latent space, and conditions denoising on an input vector derived from a prompt.The encoder and decoder form the latent representation, while the diffusion model operates within it.
  • Latent Diffusion Models: At each timestep, the denoising network removes noise from a noised latent using the latent, timestep, and text conditioning vector.Prompt tokens are mapped to embeddings before conditioning the denoising network.
  • Text conditioning: The text encoder maps N token embeddings to a conditioning tensor c(y) ∈ R^N×D, with N = 77 tokens and D = 768 dimensions.The conditioning is produced from the token embeddings after they pass through a pretrained CLIP text encoder.
  • Textual Inversion: Textual Inversion introduces a concept token embedding v∗ in P and optimizes it on concept images while keeping the entire latent diffusion model fixed.The optimized embedding is used to construct the conditioning c(y, v) for reconstruction.

4. Method

The method represents concepts in a timestep- and U-Net-layer-dependent space using a neural mapper, then adds rescaling, ordered representations, and a textual bypass to balance fidelity, editability, and compactness.

  • Space-Time Representation: P* represents each concept with vectors conditioned on denoising timestep and U-Net layer, while a neural mapper produces these vectors without directly optimizing hundreds of embeddings.The mapper receives timestep-layer inputs and outputs the matching concept embedding for the corresponding U-Net layer.
  • Mapper Optimization: The mapper is trained by sampling timesteps and 16 cross-attention layers, encoding the entire target concept in its learned parameters.At inference, the mapper is queried for each timestep and U-Net layer before the embeddings are passed through the text encoder.
  • Positional Encoding: The positional encoding uses smooth variation across time and separated encodings across U-Net layers because biasing toward nearby layers performs less favorably.The encoding maps timestep-layer pairs into a 160-dimensional representation using random Fourier features and anchor pairs.
  • Output Rescaling: Output rescaling matches mapper-output norms to a super-category token and substantially improves editability without harming reconstruction.The super-category embedding supplies the target norm; Figure 4 reports the resulting improvement in editability while preserving visual fidelity.
  • Ordered Representation: Nested Dropout orders the mapper’s hidden representation so truncation at inference changes effective dimensionality and lets users traverse the reconstruction-editability tradeoff with one model.Training with randomly sampled truncation values concentrates more information in earlier output vectors and supports post-training compression.
  • Textual Bypass: The textual bypass outputs vbase for text-encoder mixing and vpass as a residual added after encoding, supplying information the encoder fails to capture while retaining prompt compositionality.The bypass residual is norm-scaled relative to the encoded base vector, and the final representation is passed to U-Net cross-attention.

5. Results

NeTI generates high-fidelity, editable personalized compositions across concepts and prompts, while its time-space conditioning and ordered representation support controlled reconstruction–editability trade-offs.

  • Evaluations and Comparisons: NeTI attains high-fidelity reconstructions while remaining faithful to prompts, matching DreamBooth and XTI training conditions after 500 steps.The qualitative comparison reports stronger concept-specific detail and novel-scene composition than Textual Inversion, XTI, and DreamBooth in the shown examples.
  • Controlling Editability: More hidden units improve visual fidelity, whereas stronger Nested Dropout produces coarser representations that are more amenable to edits and new compositions.NeTI achieves improved reconstruction and editability across the dropout curve relative to XTI trained for the same number of steps.
  • Evaluations and Comparisons: NeTI outperforms other inversion methods and remains competitive with DreamBooth in a 35-respondent user study without tuning the generative model.Ratings jointly assessed similarity to the concept images and text prompt, with 560 ratings per method.
  • Time and Space: Conditioning the mapper on both denoising time and U-Net space is essential for attaining high visual fidelity.The ablation compares space-conditioned, time-conditioned, and unconditioned mapper variants.
  • Per-Timestep Decomposition: At early denoising timesteps, NeTI captures coarse concept details, revealing a timestep-dependent decomposition of personalized representations.The analysis queries one timestep across all 16 U-Net layers and applies the resulting embeddings across timesteps.

6. Conclusions

The paper concludes that P∗ and NeTI provide a compact, time- and space-aware representation for personalization. It also identifies inference-time cost and per-concept optimization as remaining limitations.

  • Conclusions: P∗ models both denoising time and U-Net attention layers, while NeTI represents concepts in this space with a neural mapper.The representation uses time and layer conditioning rather than directly optimizing hundreds of vectors.
  • Limitations: The method still requires a dedicated inference pass for each dropout value used to traverse the reconstruction–editability trade-off.This limitation concerns the dropout-based control mechanism at inference time.
  • Limitations: NeTI optimizes each concept independently and still requires hundreds of optimization steps per concept.The authors suggest pairing the approach with faster encoder-based methods as a possible direction.

A.2. Evaluation Setup

The evaluation compares NeTI with inversion and fine-tuning baselines across 16 concepts and 15 prompts, using shared training data and initialization tokens where applicable.

  • Evaluation Setup: NeTI is evaluated against Textual Inversion, Extended Textual Inversion, DreamBooth, and CustomDiffusion.The comparison covers both inversion methods and approaches that fine-tune the generative model.
  • Evaluation Setup: The protocol contains 15 prompts spanning background modifications and artistic styles.Examples include placing the subject on a beach and rendering it as a manga drawing.
  • Evaluation Setup: All models use the same training set and initialization token when applicable.This controls the shared input conditions across the evaluation protocol.
  • Evaluation Setup: The time-space ablation trains NeTI with and without joint conditioning for the same number of optimization steps.Figure 13 validates the importance of combining time and space for visual fidelity.

B. Storage Requirements

NeTI stores each learned concept in roughly 2 MB, substantially below fine-tuning baselines, and Nested Dropout can reduce this footprint further.

  • Storage Requirements: 560,000 parameters with textual bypass require 2.2MB per concept, while 460,000 parameters without bypass require 1.86MB.These are the reported storage requirements for the two NeTI variants.
  • Storage Requirements: Using 32 final-layer units reduces the no-bypass model to 390,000 parameters and 1.56MB per concept, a 15% decrease.Nested Dropout enables this compression by dropping a subset of final-layer parameters.
  • Storage Requirements: DreamBooth requires several gigabytes per concept and CustomDiffusion approximately 73MB, compared with NeTI’s roughly 2MB footprint.The paper reports comparable or better performance for NeTI alongside its lower storage requirement.

C. Ablation Study

The ablations show that positional encoding, Nested Dropout, and textual bypass each contribute distinct benefits to NeTI’s fidelity or editability. The bypass mainly restores fine details, while Nested Dropout improves editability without harming reconstruction quality.

  • Network Architecture and Training Scheme: Removing positional encoding prevents NeTI from adequately capturing important visual details such as object shape and distinctive patterns.The ablation reports failures to capture a mug’s shape and a cat statue’s colorful stripes.
  • Network Architecture and Training Scheme: Nested Dropout preserves comparable reconstructions while improving editability by encouraging focus on concept-specific rather than spurious training-image details.Without dropout, the cat statue cannot receive a plausible edit, whereas dropout supports more accurate novel compositions.
  • Network Architecture and Training Scheme: Textual bypass improves reconstructions by adding details that the text encoder misses, without harming editability on complex concepts.Examples include mug skulls, a cat’s tail, and a child’s string-like legs.
  • The Role of the Textual Bypass: The base vector captures coarse concept structure, while the bypass vector refines shape, texture, and other fine-level details.Adding vpass makes the metal bird and elephant more faithful and transforms a realistic bird into the target stone-like concept.

D. Additional Comparisons

Additional comparisons show that NeTI can personalize from one image, converge rapidly, outperform CustomDiffusion at matched training budgets, and support controllable style mixing. Its main limitation is weaker layer disentanglement for concepts that converge quickly.

  • Single Image Personalization: NeTI remains effective in the single-image setting, reconstructing the target concept while retaining prompt consistency.The evaluation trains without textual bypass for 500 optimization steps.
  • Training Convergence: NeTI with textual bypass matches XTI with four times fewer optimization steps and improves image similarity over XTI after 1,000 steps with only a small text-similarity decrease.Table 3 reports the comparison at 250 NeTI steps versus 1,000 XTI steps.
  • Comparison to CustomDiffusion: NeTI with textual bypass outperforms CustomDiffusion in image and text similarity after 500 steps, reaches DreamBooth-comparable performance after 1,000 steps, and uses approximately 75× less disk space per concept.These results require no tuning of the generative model.
  • Style Mixing: Style mixing may be difficult for concepts that converge quickly because shared mapper weights can weaken disentanglement across U-Net layers.The paper contrasts this with independently optimized embeddings in XTI.

F. Additional Analysis

The additional analysis examines the embedding distribution and positional encoding used by NeTI. Real CLIP token norms occupy a narrow range, while the encoding separates layer identity more strongly than small timestep differences.

  • Distribution of Token Embedding Norms: Real CLIP token-embedding norms cluster narrowly around 0.25−0.45, motivating normalization of NeTI outputs toward real-token norms.The distribution supports the rescaling technique used by the method.
  • Positional Encoding: NeTI’s positional encoding separates inputs with different U-Net layers, while inputs sharing a layer receive similar encodings despite differing timesteps.Inputs differing in both timestep and layer receive distinct encodings.

G. Additional Qualitative Results

Additional qualitative results illustrate how NeTI captures timestep-dependent concept details, controls reconstruction-editability through dropout, and generalizes across varied prompts and comparison settings.

  • Additional Analyses: The additional visualizations track which concept-specific details are captured at different denoising timesteps using NeTI.These examples complement the timestep-dependent analyses of the representation.
  • Additional Analyses: Inference-time Nested Dropout provides control over the reconstruction-editability tradeoff across concepts and prompts.The visualizations show that dropping layers changes reconstruction behavior while exposing the ordered representation’s control.
  • Text-Guided Synthesis: NeTI produces personalized generations across a diverse set of text prompts, including artistic styles and object depictions.The examples include sketches, cartoons, comic-book renderings, Banksy-style art, and statues.
  • Additional Qualitative Comparisons: Additional qualitative comparisons evaluate NeTI alongside alternative personalization methods using matched concepts, prompts, and random seeds.Textual Inversion uses 5,000 optimization steps, while the remaining methods use 500 steps in the cited comparison.
Loading 2305.15391v1…