PaperlayerView exact revision on arXiv

Source-linked AI summary

FRPSS: Feature Rearrangement in Pre-Shape Space for Single-Image Generation

Yuexing Han, Haoxuan Zhang, Bing Wang

arXiv:2609.16594v1cs.CV

TL;DR

Single-image generation methods often lack explicit global structural constraints, limiting structural coherence when generating variations. FRPSS rearranges single-image features in Pre-Shape Space to guide multiscale generation and reports strong fidelity across three datasets plus controllable stylization.

  • Problem

    Existing single-image generation methods commonly lack explicit global structural constraints, leaving structural misalignment and incoherent results as research gaps.

  • Method

    FRPSS rearranges and augments latent features in Pre-Shape Space to provide structurally plausible guidance for subsequent multiscale generation and uses directional CLIP constraints for stylization.

  • Results

    FRPSS achieves competitive single-image generation performance across three datasets and supports controllable semantic guidance and multiple downstream tasks.

  • Takeaways & Limitations

    The framework offers structurally guided single-image generation with interactive control and semantic guidance for stylization.

  • Takeaways & Limitations

    In complex scenes with large backgrounds and sparse semantic objects, FRPSS may omit local content.

Abstract

from arXiv · show

Generative models trained on a single image often struggle to balance global structural integrity and local diversity. Existing single-image generation methods commonly rely on random noise to drive the generation process and lack explicit global structural constraints, making the generated results prone to spatial structural misalignment when structural variations occur. To address the issue, Feature Rearrangement in Pre-Shape Space for Single-Image Generation (FRPSS) is proposed in this paper. The core of FRPSS is the Manifold Structural Rearrangement with Feature Augmentation on Geodesic Surface (MSR-FAGS) module. MSR-FAGS replaces the randomly initialized features of the low-scale generator with rearranged Pre-Shape features and uses the features to guide image generation at subsequent scales, thereby reducing the risk of structural misalignment. To support downstream tasks such as stylization, a Scale-adaptive Sliding-window Patch Extraction (SSPE) strategy is further designed, and a directional Contrastive Language-Image Pre-training supervision module with SSPE (CLIP-SSPE) is constructed. Qualitative and quantitative experiments demonstrate that FRPSS achieves the best Single Image Fréchet Inception Distance (SIFID) scores on all three datasets while maintaining competitive Learned Perceptual Image Patch Similarity (LPIPS). Further qualitative experiments verify the effectiveness of FRPSS across multiple downstream tasks with the CLIP-SSPE module.

1 Introduction

FRPSS addresses data scarcity and structural instability in single-image generation by rearranging features in Pre-Shape Space and guiding multi-scale synthesis. Its MSR-FAGS and CLIP-SSPE modules support diverse, structurally plausible generation and text-guided stylization.

  • Motivation: FRPSS targets domains where high-quality training data are costly to acquire and annotate, including material analysis, historical restoration, and private artistic creation [6].The broader generative-model landscape includes GANs [2] and DDPMs [3] [4], but their capabilities rely on massive training datasets.
  • Motivation: Existing single-image generation methods depend on random-noise initialization, allowing low-scale structural deviations to accumulate and causing spatial misalignment during generation.Direct CLIP-based semantic constraints can further aggravate error accumulation and introduce hallucinations in unstable noise-initialized models.
  • Proposed method: FRPSS uses MSR-FAGS to project local features into Pre-Shape Space and apply Geodesic surface interpolation, generating diverse, structurally plausible features.The framework is built on the Hierarchical Patch VAE-GAN framework [10] and incorporates FAGS and Pre-Shape Space theory.
  • Downstream stylization: CLIP-SSPE applies scale-adaptive sliding-window patch extraction and directional CLIP supervision during multi-scale generation to guide stylized images from user-provided text prompts.This module extends FRPSS to downstream tasks such as stylization.
  • Proposed method: MSR-FAGS reduces structural misalignment and improves generated-image fidelity, while PRE-MSR preserves representation-form consistency between rearranged and output features.These components are part of the overall FRPSS framework shown in Fig. 1.

2 Related work

Single-image generation methods primarily use multi-scale GANs or diffusion models, but noise-based initialization can cause structural misalignment that worsens across scales. FRPSS addresses this limitation with Pre-Shape Space modeling and geodesic feature augmentation, while using directional CLIP supervision to support stylization without disturbing spatial layout.

  • Single-image generation: Single-image generation methods mainly comprise multi-scale GANs and diffusion models, which learn diverse samples from local patch statistics of one input image.SinGAN [8] introduced cascaded coarse-to-fine generators; later variants targeted efficiency, diversity, and global structure, while diffusion methods such as SinDDM [13] and SinDiffusion extended the alternative paradigm.
  • Limitations: Noise-based initialization in existing GAN and diffusion methods lacks explicit structural priors, allowing initial misalignment to accumulate as outputs are upsampled.The issue is identified across both major single-image generation paradigms, with particularly progressive error accumulation in multi-scale GAN architectures.
  • Stylization supervision: Global CLIP supervision can destabilize optimization or distort spatial layout, so FRPSS applies directional CLIP loss [16] at each scale after stabilizing image structure.SinDDM [13] updates predicted clean images during iterative denoising, but simultaneously changing structure and texture can cause hallucinated content and structural distortion.
  • Pre-Shape Space: FRPSS adopts Pre-Shape Space because it removes translation and scaling while retaining rotational information and supports corresponding feature augmentation.Pre-Shape samples lie on a hypersphere, providing the geometric representation used to model features.
  • Feature augmentation: FRPSS uses geodesic interpolation to augment Pre-Shape samples on the manifold, with FAGS iteratively synthesizing a Pre-Shape that approximates their weighted Fréchet mean.The interpolation follows shortest geodesics, with endpoints recovered at t = 0 and t = 1; FAGS uses Dirichlet-sampled weights.

3 Methodology

FRPSS uses a two-stage Patch-VAE/Patch-GAN framework that reconstructs low-scale structure before generating high-frequency diversity. Its MSR-FAGS module deterministically rearranges Pre-Shape features for structurally plausible variation, while SSPE supports scale-adaptive directional supervision for downstream stylization.

  • Framework: FRPSS combines Patch-VAE reconstruction with Patch-GAN refinement: low scales reconstruct the image, while later scales generate diverse high-frequency texture information.The Patch-VAE stage uses E, PRE-MSR, and G0,…,GM; the Patch-GAN stage uses MSR-FAGS and subsequent generators.
  • Patch-VAE: PRE-MSR converts sampled encoder features into the same Pre-Shape representation as MSR-FAGS by grouping them along width, projecting each group, and concatenating the projections.For groups of width α, each feature group is projected into a 2C-dimensional Pre-Shape representation before assembly.
  • Patch-VAE: During Patch-VAE training, E, G0, and the current-scale generator are jointly optimized while earlier intermediate generators remain frozen, preserving the evolving latent-to-image mapping.The stage uses reconstruction and KL-divergence losses, with β weighting the KL term.
  • MSR-FAGS: MSR-FAGS uses the encoder’s mean feature as a deterministic structural prior, rearranges it in Pre-Shape Space, and produces structurally plausible features with variations.Reparameterization is removed in Patch-GAN because the mean feature encodes the original image’s structure; shortest Geodesic distance matching preserves spatial coherence during recombination.
  • CLIP-SSPE: SSPE adaptively extracts sliding-window patches with window sizes restricted to [64, 128], but falls back to whole-image directional supervision when local extraction is infeasible.This fallback makes the patch-wise constraint degenerate into a global constraint for sufficiently small images or unsatisfied extraction conditions.
  • CLIP-SSPE: The directional supervision mechanism extends from text-guided changes to image-guided style transfer by using a target style image to provide the change direction.Thus, CLIP-SSPE supports downstream style-transfer settings beyond text-guided supervision.

4 Experiments · 4.1 Experimental datasets

FRPSS is evaluated on three single-image generation datasets spanning varied textures, semantic structures, and topological variation. The datasets collectively test generation across complex scenes, diverse visual content, and structurally varying images.

  • 4.1 Experimental datasets: FRPSS is evaluated on Places50 [8], MSID16 [8], and SIGD16.These are general single-image generation datasets.
  • 4.1 Experimental datasets: Places50 [8] contains 50 scene images with complex semantic structures.
  • 4.1 Experimental datasets: MSID16 [8] contains 16 images spanning plants, animals, and natural landscapes.
  • 4.1 Experimental datasets: SIGD16 contains 16 images with significant topological variations.It is used to evaluate the method under topological variation.

4.2 Evaluation metrics

The evaluation uses SIFID to assess similarity between generated and original images, and LPIPS to measure perceptual diversity among generated samples. Both metrics aggregate results across 20 samples per original image.

  • Evaluation metrics: SIFID [8] and LPIPS quantitatively evaluate generated results, respectively measuring similarity to the original image and diversity among samples.SIFID uses spatial feature distributions extracted by a shallow pre-trained Inception-V3 block, while LPIPS uses deep features from pre-trained AlexNet [8].
  • SIFID: For each original image, 20 generated samples are evaluated with SIFID, and their mean score is reported; lower SIFID indicates closer feature distributions.SIFID compares the spatial feature distribution of each generated image with that of the original image.
  • LPIPS: For each original image, the 190 non-repeated pairs among 20 generated samples are evaluated with LPIPS, whose mean measures diversity; higher values indicate greater perceptual differences.LPIPS is computed from normalized, layer-weighted AlexNet feature representations.

4.3 Qualitative and quantitative analysis of image generation

FRPSS improves both structural fidelity and local visual quality in single-image generation. It achieves the best SIFID results across all three datasets while maintaining competitive LPIPS and reducing spatial misalignment artifacts.

  • Qualitative analysis: SinGAN [8] and ConSinGAN [9] exhibit structural misalignment that can accumulate into path distortion and mountain discontinuity during progressive generation.HP-VAE-GAN [10] can achieve relatively high LPIPS while still showing obvious global structural misalignment.
  • Qualitative analysis: MSR-FAGS rearranges and matches low-scale features to enable global layout variation while preserving spatial coherence.This mechanism improves the balance between global structural integrity and local visual fidelity.
  • Qualitative analysis: FRPSS produces fewer spatial misalignment artifacts and better preserves the original images’ color distribution and contrast.These qualitative comparisons are shown on the Places50 and SIGD16 datasets.
  • Quantitative analysis: FRPSS achieves the best SIFID results on Places50, MSID16, and SIGD16, indicating generated images more closely match original images’ deep feature distributions.Table 1 reports the quantitative comparison across all three datasets.
  • Quantitative analysis: FRPSS maintains competitive LPIPS scores across all three datasets, preserving perceptual diversity while retaining the original images’ visual characteristics.

4.4 Image stylization transformation

FRPSS extends beyond stylization to text-guided content generation, image-guided style transfer, and paint-to-image synthesis. Across these tasks, CLIP-SSPE and directional guidance help preserve spatial structure while producing target styles, semantics, textures, and details.

  • Text-guided style transfer: FRPSS preserves original scene content and spatial structure more effectively than SinDDM [13] during text-guided style transfer while introducing target visual styles.CLIP-SSPE constrains visual and semantic changes from “a photo” toward anime, Monet, Picasso, and Van Gogh styles.
  • Text-guided content generation: FRPSS produces more obvious target semantic changes than SinDDM [13] for prompts such as “Oasis,” “A fire in the forest,” and “Sunset.”The task changes both visual appearance and image content according to the target text, whereas SinDDM may mainly alter color or local texture.
  • Image-guided style transfer: FRPSS transfers reference-image textures while preserving clear main structures, avoiding the over-smoothing and blurred boundaries produced by GPDM.The evaluation uses content images and five style references, including Van Gogh, ink landscape, expressionist, watercolor city, and golden hall styles.
  • Paint-to-image: FRPSS progressively adds natural textures and visual details while preserving a paint’s main spatial layout, producing more natural results than SinGAN and SAMDSinGAN.Its directional guidance mechanism uses the paint as the source and a natural image as the reference, unlike rigid implicit feature fusion at selected low-scale stages.

4.5 Outpainting

FRPSS performs outpainting by generating extension features on both sides of the encoded original image and feeding their concatenation into its fully convolutional generator for continued multiscale synthesis.

  • Outpainting: FRPSS encodes the original low-scale image, generates left and right extension features with MSR-FAGS, and concatenates them with the original feature along the width.This extends the feature representation before subsequent generation while preserving the original image feature.
  • Outpainting: Outpainting generates visual content beyond the boundaries of the original image, illustrated by Figure 14 with yellow boxes marking the originals.The figure presents qualitative image-outpainting results.
  • Outpainting: The concatenated extended feature can be directly processed because the generator is fully convolutional and does not depend on a fixed input spatial size.Bilinear interpolation remains part of the subsequent multiscale outpainting process.

4.6 Parameter influence analysis

On SIGD16, MSR-FAGS parameter analysis shows that feature slice width α and interpolated-feature count Q trade off generation diversity, fidelity, and structural coherence. Moderate feature rearrangement and interpolation improve novelty while avoiding artifacts and excessive reproduction.

  • Experimental scope: The study quantitatively and qualitatively evaluates α and Q on SIGD16 using Table 2 and the corresponding visualizations in Fig. 15 and Fig. 16.The analysis focuses on their effects on generation quality and diversity.
  • Feature slice width α: α = 2 yields LPIPS 0.125 and SIFID 0.043, but its limited rearrangement range makes outputs reproduce the original image and reduces diversity.As α increases to 16, LPIPS reaches 0.353 while SIFID remains at a relatively stable level.
  • Interpolated features Q: With α = 16, Q = 0 gives LPIPS 0.356 but introduces local stitching artifacts, whereas Q = 8 lowers SIFID to 0.031 and LPIPS to 0.347.Introducing an appropriate number of interpolated features helps synthesize more natural, coherent semantic features with structural plausibility and novelty.

4.7 Stylization ablation experiment

The stylization ablation analyzes how global and local directional losses affect text-guided stylization. Using a sunset scene and multiple target styles, it shows that global-loss-only stylization mainly changes overall color tone.

  • Stylization ablation experiment: The ablation evaluates the global directional loss Lglobal and local directional loss Lpatch for text-guided stylization.The original image is a sunset scene, with src set to “a photo” and target prompts spanning Rococo, Turner, Cubism, Ink Painting, and Pop Art styles.
  • Stylization ablation experiment: With only Lglobal, the generated results mainly change the image’s overall color tone.
  • Stylization ablation experiment: The experiment compares stylization under five target prompts: Rococo Style, Turner Style, Cubism Style, Ink Painting Style, and Pop Art Style.

4.8 Limitations

FRPSS remains limited in complex scenes with large backgrounds and sparse semantic objects, where local content may be missing. Its stylization mechanism is also coupled to target styles, requiring retraining when styles or guidance conditions change.

  • Complex-scene limitations: MSR-FAGS feature rearrangement can weaken sparse semantic structures because they are overwhelmed by proportionally larger background features in low-scale representations.This mechanism explains the observed local content missing.
  • Stylization limitations: FRPSS stylization couples the multi-scale generative model to the target semantic direction, so changing styles or guidance conditions generally requires retraining and adds training overhead.Consequently, switching among multiple styles remains costly.
  • Complex-scene limitations: In complex scenes with large background regions and sparse semantic objects, FRPSS may fail to preserve local content, including thatched cottages and human faces.These failure cases are shown in Fig. 18.

5 Conclusion

The conclusion presents FRPSS as a response to structural misalignment in single-image generation through Pre-Shape-space feature rearrangement, while acknowledging scene-dependent limitations and future conditional style-control directions.

  • Conclusion: FRPSS addresses structural misalignment by using MSR-FAGS to rearrange, augment, and match single-image latent features in Pre-Shape Space for multiscale generation.The resulting latent features provide structural variations for subsequent generation.
  • Limitations: In complex scenes with large backgrounds and sparse semantic objects, low-scale feature rearrangement may weaken local semantic-region structure.
  • Future Work: Future conditional injection could decouple semantic control from generative-model training, enabling text- or reference-image-based style control during inference with less repeated retraining.
Loading 2609.16594v1…