PaperlayerView exact revision on arXiv

Source-linked AI summary

Kingfisher: Fast Personalized Visual Autoregressive Generation via Scale-Aware Prompt-Conditioned Scale Decomposition

Dinh-Khoi Vo, Akihiro Sugimoto, Minh-Triet Tran, Trung-Nghia Le

arXiv:2610.23238v1cs.CV

TL;DR

Personalization in visual autoregressive models has been less explored than diffusion personalization, particularly how content and style are organized across scale hierarchies. Kingfisher decomposes and recomposes early scale codes from two images in a compact prefix, then lets a frozen generator complete the image. On CSD-100, it reports strong content preservation and leading style and text alignment, alongside substantially faster pair-specific preparation.

  • Problem

    How content and style are organized across visual autoregressive scale hierarchies remains underexplored, despite the potential to personalize directly through a frozen generator’s native scale-code trajectory.

    Source: §1 · p. 2

  • Method

    Kingfisher uses prompt-conditioned subspaces to decompose early residual codes into content and appearance components, recomposes components from two images in a compact prefix, and predicts remaining scales with a frozen generator.

    Source: §1 · p. 2

  • Results

    On CSD-100, Kingfisher achieves the highest style and text alignment among evaluated methods while maintaining strong content preservation, with pair-specific preparation about 54× faster than diffusion methods and 7.5× faster than VAR alternatives.

    Source: §1 · p. 3

  • Takeaways & Limitations

    Native scale trajectories provide an efficient interface for content–style personalization without model fine-tuning.

    Source: §7 · p. 10

    Transfer is weaker for visually indistinctive styles, and effectiveness depends on how accurately prompts describe target content and how well the frozen generator responds.

    Source: H LIMITATIONS · p. 19

Abstract

from arXiv · show

Personalized image generation has been extensively studied in diffusion models, often relying on subject-specific optimization of model weights, textual embeddings, LoRA modules, or adapters. In this work, we explore a different direction for fast personalization in visual autoregressive models. Our key observation is that a short prefix of early autoregressive scale codes is sufficient to strongly determine both object structure and global appearance, while later scales primarily refine the established generation trajectory. Based on this insight, we introduce Kingfisher, a model-tuning-free framework that decomposes early residual scale codes into content- and style-related components through prompt-conditioned scale-code subspaces. Given two real images, Kingfisher first decomposes their early scale codes, then recomposes the content components of one image with the style components of the other into a compact semantic prefix. The frozen generator subsequently predicts the remaining scales through bitwise self-correction for coherent refinement. Experiments on CSD-100 show that Kingfisher achieves strong content preservation and the highest style and text alignment among the evaluated methods, while reducing pair-specific preparation time by approximately 54x compared with diffusion-based methods and 7.5x compared with VAR-based alternatives. These results demonstrate that effective personalization can be achieved by directly manipulating the native scale-space representation of a frozen visual autoregressive generator.

1 INTRODUCTION

Personalization is well studied in diffusion models, but how content and style are organized across visual autoregressive scale hierarchies remains underexplored. Kingfisher addresses this gap by composing prompt-conditioned content and appearance components in a compact prefix of a frozen generator.

1 INTRODUCTION

  • Personalization research spans diffusion, GAN, autoregressive, and multimodal generators, with diffusion methods operating in textual, feature, noise, or parameter spaces.Methods include inversion, optimization, learned encoders or adapters, and hybrid pipelines.Source: §1 · p. 2
  • Early autoregressive scales act as semantic anchors, making a compact prefix sufficient to steer content and appearance while the frozen generator completes later scales.Earlier work often intervenes on individual scales, whereas the authors examine consecutive scales as a generation trajectory.Source: §1 · p. 2
  • Kingfisher decomposes mixed early scale codes into content, appearance, and residual factors using prompt-conditioned subspaces, then recomposes content from one image with appearance from another.The frozen autoregressive generator predicts remaining scales through bitwise self-correction.Source: §1 · p. 2
  • On CSD-100, Kingfisher achieves the highest style and text alignment among evaluated methods while maintaining strong content preservation, with pair-specific preparation approximately 54× and 7.5× faster than diffusion- and VAR-based alternatives, respectively.Source: §1 · p. 3

2 RELATED WORK

Prior work exposes visual controls through generator layers, latent spaces, and autoregressive scales. Kingfisher studies content and appearance in visual autoregressive scale trajectories, where early scales form entangled semantic bottlenecks.

2 RELATED WORK

  • Generator layers and stages can expose controllable visual factors, as shown in StyleGAN, StyleSpace, W+, and per-layer diffusion prompt methods.These approaches include localized channel-wise controls and object–style mixing.Source: §2 · p. 3
  • Unlike clean layer-wise controls, Kingfisher finds early residual scales form entangled semantic bottlenecks coupling object structure, appearance, composition, and style priors.The related work frames this contrast against prior generator-layer and latent-space controls.Source: §2 · p. 3
  • Token-based autoregressive generators typically flatten discrete visual codes into sequences, whereas VAR predicts next scales coarse-to-fine and Infinity adds bitwise prediction and self-correction.These scale-wise models motivate decomposing and recomposing early scale codes for personalization.Source: §2 · p. 3

3 SCALE-WISE INSIGHTS AND PRELIMINARIES

Early residual scales strongly constrain the visual trajectory: short prefixes nearly recover image semantics, while later scales primarily refine details. This motivates composing content and appearance within a compact early prefix and letting the frozen generator predict the remainder.

3 SCALE-WISE INSIGHTS AND PRELIMINARIES

  • The first four ground-truth scales already nearly reconstruct the input, and routing prompts to early scales shifts outputs toward those prompts whereas routing them to large suffixes does not.The authors conclude that early scales act as semantic anchors and later scales primarily refine local details.Source: §3 · p. 4, Figure 2 · p. 4
  • The autoregressive process is expressed as a factorization over scale codes, with each scale conditioned on earlier scales and the text prompt.The model represents images with N residual scale codes.Source: §3 · p. 4
  • Infinity uses scale-wise autoregressive prediction with bitwise visual-token prediction and self-correction, conditioning each next scale on the accumulated visual prefix and text prompt.The next-scale prediction attends to prior prefix states; for example, predicting R5 depends on information accumulated from R1:4.Source: §3 · p. 4
  • Kingfisher encodes content and style images into residual scale codes, decomposes selected early codes into factors, recomposes a compact prefix, and uses the frozen generator to predict remaining scales.The pipeline aims to retain the content image’s object structure while adopting the style image’s appearance.Source: Figure 3 · p. 5
  • Early consecutive scales form an entangled semantic bottleneck coupling identity, shape, color, background, lighting, and style priors, rather than a clean content–style split.This motivates decomposing and recomposing factors in the early prefix while leaving later scales to the frozen generator.Source: §3 · p. 5

4 PROPOSED METHOD

Kingfisher encodes two real images into residual scale codes, decomposes their early scales into content and appearance components, and recomposes them into a prefix. The frozen generator predicts the remaining scales through bitwise self-correction.

4 PROPOSED METHOD

  • Kingfisher combines content from image A with appearance from image B in an early semantic prefix, then uses the frozen generator to predict remaining scales.The generator completes the suffix through bitwise self-correction.Source: §4 · p. 5, §4.3 · p. 7

4.1 PROMPT-CONDITIONED SCALE SUBSPACES

  • Neutral-style and object-conditioned prompt predictions define subspaces that separate appearance variation from target-object directions without adding trainable parameters.Neutral prompts provide a natural-appearance reference; object prompts are centered relative to it before constructing the object-content subspace.Source: §4.1 · pp. 5–6

4.2 PROMPT-CONDITIONED SCALE DECOMPOSITION

  • Each selected scale code is decomposed as R_i = C_i + S_i + e_i, representing object content, visual appearance, and unexplained residual information.The components are jointly optimized with constraints that preserve the original code, reduce content–appearance overlap, and align components with their respective subspaces.Source: §4.2 · p. 6

4.3 SCALE-AWARE PREFIX COMPOSITION

  • The composed prefix transfers appearance at scales R1 and R2, preserves R3 as a semantic anchor, and applies content-aware structural correction at R4.The R4 correction uses an appearance-aware base and adds the content difference between images A and B.Source: §4.3 · p. 7
  • Kingfisher predicts rather than copies later scales, avoiding source-specific details from either image while completing the composed prefix with the frozen generator.The same formulation can support other prompt-definable factors by constructing corresponding scale-code subspaces.Source: §4.3 · p. 7

5 EXPERIMENTS

Kingfisher is evaluated on 500 ordered cross-image pairs from CSD-100 against diffusion and visual autoregressive baselines. It leads on four reported metrics, balances content preservation with style transfer, and receives the highest human preference while requiring less pair-preparation time than CSD-VAR.

5.1 EXPERIMENTAL SETUP

  • On 100 CSD-100 images, the evaluation forms 500 ordered pairs, excluding shared object categories, and generates one image per pair using a fixed seed.Each content image is paired with five style references from different categories; the target prompt names the content subject and style.Source: §5.1 · p. 7
  • The baselines span diffusion and visual autoregressive methods, and evaluation measures content preservation, style alignment, and text alignment.The diffusion baselines use SDXL; the metrics include CSD-C, CSD-S, CLIP-I, DINO, and CLIP-T.Source: §5.1 · pp. 7–8

5.2 COMPARISON WITH PRIOR METHODS

  • Kingfisher achieves the best performance on CLIP-I, CSD-S, DINO, and CLIP-T in the CSD-100 quantitative comparison.The comparison covers all 500 ordered content–style pairs.Source: §5.2 · p. 8, Table 1 · p. 8
  • Figure 4 compares cross-image compositions; Kingfisher preserves the content image’s object and spatial structure while transferring the style image’s appearance.The reported comparison describes coherent outputs across representative content–style pairs.Source: §5.2 · pp. 8–9, Figure 4 · p. 8
  • Kingfisher requires 23.54 seconds of pair preparation versus CSD-VAR’s 171.65 seconds, and receives the highest human preference among the evaluated methods.Human preference comes from 600 forced-choice selections; efficiency was measured on one A100 GPU.Source: §5.2 · p. 9, Table 2 · p. 9

6 DISCUSSION

Kingfisher treats early residual scales as an entangled semantic bottleneck and explicitly models content and appearance within that trajectory. This provides a way to analyze and recompose those factors in a frozen generator while reducing personalization cost relative to optimization-heavy alternatives.

6 DISCUSSION

  • Early scales jointly determine object structure and global appearance rather than separating content and style scale by scale.The discussion characterizes the early scales as an entangled semantic bottleneck.Source: §6 · p. 9
  • Kingfisher constrains content- and appearance-related roles through prompt-conditioned scale subspaces in the frozen generator’s native residual trajectory.This explicitly models and recomposes the two factors within the scale trajectory.Source: §6 · p. 9
  • Operating on a compact early prefix substantially reduces personalization cost compared with optimization-heavy alternatives.Source: §6 · p. 9

7 CONCLUSION

Kingfisher personalizes a frozen pretrained visual autoregressive generator by recomposing content and appearance in an early semantic prefix. On CSD-100, it achieves strong content–style composition while reducing personalization cost without model fine-tuning.

7 CONCLUSION

  • Kingfisher operates directly on the residual scale trajectory of a frozen pretrained visual autoregressive generator.Source: §7 · p. 9
  • It achieves content–style personalization without model fine-tuning by recomposing content- and appearance-related components within the early semantic prefix.Source: §7 · p. 10
  • CSD-100 experiments show strong content–style composition and substantially reduced personalization cost.The conclusion presents native scale trajectories as an efficient interface for personalized visual autoregressive generation.Source: §7 · p. 10

AI USE STATEMENT

The authors report that ChatGPT was used only for grammar correction and language polishing; technical work was developed, conducted, and verified by the authors.

AI USE STATEMENT

ETHICS STATEMENT

The human preference study used voluntary, anonymous judgments and collected no personal identifiers or sensitive information.

ETHICS STATEMENT

REPRODUCIBILITY STATEMENT

The paper identifies materials intended to support reproduction, describes scale-wise evaluation and prompt construction, and details its human preference protocol.

REPRODUCIBILITY STATEMENT

  • Reproduction materials include the evaluation protocol, baselines, metrics, implementation settings, hyperparameters, qualitative results, ablations, and prompt templates.Source: REPRODUCIBILITY STATEMENT · p. 10

A ADDITIONAL SCALE-WISE ANALYSIS

B CONDITION-PROMPT CONSTRUCTION

C DETAILS OF HUMAN EVALUATION

D ABLATION STUDIES

The ablations identify a balanced early-prefix configuration and show trade-offs in correction strength and decomposition constraints; qualitative results illustrate transfer behavior, with weaker performance on indistinctive styles.

D ABLATION STUDIES

E ADDITIONAL QUALITATIVE EVALUATION

F QUALITATIVE RESULTS

G IMPLEMENTATION DETAILS

  • Experiments use frozen Infinity-2B across 13 scales, editing R1–R4 and generating R5–R13; personalization updates no model weights, adapters, or subject embeddings.Source: G IMPLEMENTATION DETAILS · p. 18

H LIMITATIONS

  • Weak or indistinctive appearance cues can make the estimated style component less discriminative, and effectiveness also depends on prompt accuracy and the frozen generator’s response.Source: H LIMITATIONS · p. 19, Figure 9 · p. 20

Related papers

Loading 2610.23238v1…