PaperlayerView exact revision on arXiv

Source-linked AI summary

Multimodal Garment Designer: Human-Centric Latent Diffusion Models for Fashion Image Editing

Alberto Baldrati, Davide Morelli, Giuseppe Cartella, Marcella Cornia, Marco Bertini, Rita Cucchiara

arXiv:2304.02051v2cs.CVcs.AIcs.MM

TL;DR

Fashion image editing has largely been distinguished from prior work focused on virtual try-on, and this paper proposes editing guided by text, human pose, and garment sketches. It introduces a human-centric latent diffusion model and reports improved realism and coherence with multimodal inputs on two extended fashion datasets.

  • Problem

    Prior fashion work mainly focused on virtual try-on, and suitable datasets for multimodal-conditioned fashion image editing were lacking.

    Source: §1 · pp. 1–2

  • Method

    The authors propose MGD, a human-centric latent diffusion model for fashion image editing conditioned on text, body pose, and garment sketches.

    Source: §1 · p. 2, §3 · p. 3

  • Results

    On two extended fashion benchmarks, MGD outperforms considered competitors in realism and coherence with multimodal inputs, including in human evaluations.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The authors describe MGD as a promising early attempt to mimic designers’ creative work in fashion, with human oversight.

    Source: §6 · p. 9

Abstract

from arXiv · show

Fashion illustration is used by designers to communicate their vision and to bring the design idea from conceptualization to realization, showing how clothes interact with the human body. In this context, computer vision can thus be used to improve the fashion design process. Differently from previous works that mainly focused on the virtual try-on of garments, we propose the task of multimodal-conditioned fashion image editing, guiding the generation of human-centric fashion images by following multimodal prompts, such as text, human body poses, and garment sketches. We tackle this problem by proposing a new architecture based on latent diffusion models, an approach that has not been used before in the fashion domain. Given the lack of existing datasets suitable for the task, we also extend two existing fashion datasets, namely Dress Code and VITON-HD, with multimodal annotations collected in a semi-automatic manner. Experimental results on these new datasets demonstrate the effectiveness of our proposal, both in terms of realism and coherence with the given multimodal inputs. Source code and collected multimodal annotations are publicly available at: https://github.com/aimagelab/multimodal-garment-designer.

1. Introduction

Fashion image editing in this work extends beyond garment try-on to multimodal guidance, addressing limitations in fashion-domain diffusion. The authors introduce a human-centric latent diffusion model and multimodally annotated benchmarks, reporting gains in realism and coherence.

1. Introduction

  • Only a few fashion text-conditioned editing works were proposed, while latent diffusion remained unexplored in fashion and existing approaches struggled with human-centric generation and multiple conditioning signals.Earlier computer-vision work also focused on clothing recognition, retrieval, recommendation, and virtual try-on.Source: §1 · pp. 1–2
  • MGD defines multimodal-conditioned fashion image editing and uses latent diffusion to follow multiple modalities while preserving a person’s identity and body shape.Its denoising network also accounts for pose consistency between input and generated images.Source: §1 · p. 2
  • The authors extend Dress Code and VITON-HD with textual descriptions and garment sketches through a semi-automatic annotation framework.Source: §1 · p. 2
  • On the two multimodal fashion benchmarks, MGD outperforms considered competitors in realism and coherence with multimodal inputs, including in human evaluations.The experiments report quantitative and qualitative results as well as human judgments.Source: §1 · p. 2

2. Related Work

Related work spans text-guided fashion generation and methods for conditioning diffusion models on different modalities. MGD applies multimodal conditioning to human-centric fashion editing.

Text-Guided Image Generation

  • Fashion text-to-image work was sparse: Zhu et al. use text and semantic layouts, [35] uses CLIP-guided GAN inversion, and Jiang et al. use one-hot clothing vectors.The paper notes that the one-hot conditioning limits expressiveness.Source: §2 · p. 2

Multimodal Image Generation with Diffusion Models

  • Diffusion conditioning includes reference-latent matching, noisy-stroke inversion, semantic-latent finetuning [51], and sketch concatenation or edge prediction [49].Source: §2 · p. 2
  • ControlNet [57] uses locked and trainable model copies, while T2I-Adapter [31] learns modality-specific adapters; MGD targets fashion with text, pose, and sketches.Source: §2 · pp. 2–3

3. Proposed Method

MGD frames fashion image editing as multimodal garment replacement that preserves a person’s identity and body characteristics. It builds on latent diffusion and conditions generation with text, pose, and garment sketches.

3. Proposed Method

  • MGD generates a new human-centric fashion image from an input model image, pose map, garment description, and sketch while retaining the model’s identity and physical characteristics.The task is treated as garment replacement in the input image.Source: §3 · p. 3

3.1. Preliminaries

  • Stable Diffusion supplies the latent-diffusion backbone, operating in a pre-trained autoencoder’s latent space with an encoder, decoder, conditional U-Net, and CLIP text encoder.The paper describes latent-space operation as more computationally efficient while preserving generation quality.Source: §3.1 · p. 3

3.2. Human-Centric Image Editing

  • The method treats garment replacement as human-body inpainting, concatenating the noisy latent, resized mask, and masked-image encoding as spatial input.Pose-map conditioning adds 18 keypoint channels; extending the pre-trained input layer with randomly initialized kernels reduces training steps and enables training with less data.Source: §3.2 · pp. 3–4
  • Garment sketches add fine-grained spatial details such as shape and size that text may not convey, and condition only the early denoising steps.The spatial input combines the resized pose map and sketch with the latent, mask, and masked-image encoding.Source: §3.2 · p. 4
  • A final face-mask composition copies the input model’s face into the decoded image to preserve identity.The output combines the original face pixels with the decoded result outside the face mask.Source: §3.2 · p. 4

3.3. Training and Inference

  • Training teaches the denoiser to predict added noise under the multimodal inputs, while independently masking conditions during training enables optional control signals at inference.The paper reports that tuning the condition-masking probability can affect final image quality.Source: §3.3 · p. 4
  • At inference, fast multicondition classifier-free guidance follows the joint direction of the text, pose, and sketch conditions, reducing feed-forward executions from N + 1 to 2.The guidance scale controls extrapolation toward the condition.Source: §3.3 · p. 4

4. Collecting Multimodal Fashion Datasets

The paper extends Dress Code and VITON-HD with garment text and sketches through a semi-automatic annotation framework. It combines noun-chunk selection and manual review with sketch extraction, and compares the resulting textual annotations with other fashion datasets.

4. Collecting Multimodal Fashion Datasets

  • Dress Code and VITON-HD Multimodal extend the original 1024 × 768 garment-model image pairs with textual sentences and garment sketches.Figure 3 shows sample images and multimodal data from the collected datasets.Source: §4 · p. 5, Figure 3 · p. 5

4.1. Dataset Collection and Annotation

  • For Dress Code, the annotation framework extracts more than 60k unique noun chunks from fashion captions and uses CLIP and OpenCLIP similarity to associate 25 candidates with each garment.The chunks are grouped into upper-body clothes, lower-body clothes, and dresses; the 25 candidates come from the top five per model with repetitions removed.Source: §4.1 · p. 5
  • Annotators select three representative chunks for 26,400 Dress Code garments, while a fine-tuned OpenCLIP model assigns three similar chunks to the remaining items and to VITON-HD garments.The manually annotated set includes all items in the original Dress Code test set; VITON-HD annotations are restricted to upper-body garments.Source: §4.1 · pp. 5–6
  • PiDiNet extracts garment sketches; for unpaired test images, a warped in-shop garment is first aligned to the target model using a thin-plate spline and U-Net refinement.Paired-set sketches are extracted from the garment worn by the model using its human parsing mask.Source: §4.1 · p. 6

4.2. Comparison with Other Datasets

  • The new datasets contain greater variety in textual items and words than the compared datasets, and the other datasets lack in-shop garment images.Table 1 compares textual-annotation statistics across the fashion datasets.Source: Table 1 · p. 5, §4.2 · p. 6

5. Experimental Evaluation

The evaluation compares MGD with competitors across paired and unpaired settings, using image realism and modality-coherence metrics alongside human judgments. It also examines the effects of masking inputs and varying training conditions.

5.1. Implementation Details and Competitors

  • Models are trained for 150k steps on one NVIDIA A100 GPU, and inference uses 50-step DDIM with classifier-free guidance parameter 7.5.Training uses batch size 16, learning rate 10^-5, and 0.2 fractions for sketch conditioning and masked conditions; competitors include Stable Diffusion inpainting, FICE, and SDEdit.Source: §5.1 · p. 6

5.2. Evaluation Metrics

  • Evaluation uses FID and KID for realism, CLIP-S for text adherence, and newly proposed pose and sketch distances for modality coherence.Pose distance compares confidence-weighted keypoint distances within the generation mask; sketch distance compares extracted garment sketches using frequency-weighted mean squared error.Source: §5.2 · pp. 6–7

5.3. Experimental Results

  • MGD consistently outperforms the considered competitors on realism and input coherence across the paired and unpaired dataset settings.Table 2 reports results for Dress Code Multimodal and VITON-HD Multimodal.Source: §5.3 · p. 7, Table 2 · p. 7
  • In the high-resolution setting, MGD outperforms SDEdit on all metrics, while Stable Diffusion has worse pose distance than both methods.The authors attribute SDEdit’s weaker performance to sketch conditioning only at initialization, whereas MGD conditions denoising at multiple steps.Source: §5.3 · pp. 7–8, Figure 4 · p. 8
  • The user study favors MGD over competitors for realism and adherence to multimodal inputs.About 7,000 evaluations from more than 150 users were collected; Table 4 reports preference percentages on the unpaired setting.Source: Table 4 · p. 7, §5.3 · p. 8
  • Masking sketch or pose inputs changes the corresponding coherence measures, supporting the use of these modalities as optional, disentangled conditions.Sketch masking affects sketch distance and slightly affects pose distance; pose masking differs from text-only conditioning across metrics except CLIP-S.Source: Table 3 · p. 7, §5.3 · p. 8
  • The best results in the tested ablations use 0.2 for both the unconditional-training portion and the sketch-conditioning fraction.Sketch distance consistently decreases as the number of sketch-conditioning steps increases.Source: §5.3 · pp. 8–9, Table 5 · p. 8

6. Conclusion

The conclusion presents MGD as a promising first latent-diffusion approach to human-centric fashion image editing with multimodal conditioning. It frames the work as an early step toward diffusion models in creative industries, with human oversight.

6. Conclusion

  • MGD is presented as the first latent diffusion model for human-centric fashion image editing conditioned on text, body pose, and sketches.The architecture is trained on two newly semi-automatically annotated datasets and evaluated with standard and newly proposed metrics and user studies.Source: §6 · p. 9
  • The authors describe the results as a promising attempt to mimic designers’ work in fashion’s creative process.Source: §6 · p. 9
  • The conclusion suggests the work could be a starting point for broader adoption of diffusion models in creative industries under human oversight.Source: §6 · p. 9

A. Dress Code Multimodal and VITON-HD Multimodal Datasets

This appendix section provides further information about the multimodal datasets, covering their collection and annotation, statistics, and examples.

A. Dress Code Multimodal and VITON-HD Multimodal Datasets

A.1. Data Preparation

The preparation pipeline extracts and filters noun chunks, then uses CLIP-based similarity to associate garment images with candidate text prompts. A custom tool supports fine-grained human annotation of those candidates.

A.1. Data Preparation

  • FashionIQ captions are lemmatized before noun-chunk extraction because they describe properties to change rather than a single garment.Source: §A.1 · p. 11
  • Noun chunks are extracted with spaCy, stripped of leading articles, filtered for special characters, and deduplicated.Source: §A.1 · p. 11
  • Using cosine similarity between CLIP-based image and text embeddings, the method associates each garment with the 25 most representative noun chunks, applying prompt ensembling for zero-shot association.Source: §A.1 · p. 11
  • Figure 5 provides examples of FashionIQ data types, and Table 6 reports unique-caption and noun-chunk counts by category.Source: Figure 5 · p. 11, Table 6 · p. 11

A.2. Annotation Tool for Fine-Grained Annotation

  • The annotation tool presents the model image and corresponding in-shop garment for users to select three representative noun chunks, with manual entry available when suggestions fail.The tool was developed using Django and Angular.Source: §A.2 · p. 11

A.3. Coarse-Grained Annotation

The coarse-grained annotations extend Dress Code and VITON-HD using noun-chunk annotations supported by fine-tuned OpenCLIP. A separate warping-and-refinement pipeline generates garment sketches fitted to target models.

A.3. Coarse-Grained Annotation

  • Fine-tuned OpenCLIP ViT-B/32 helps annotate the remaining 27,392 Dress Code and 13,679 VITON-HD items with noun chunks.The manually annotated set contains 26,400 model-garment pairs, with 8,800 items per category and three noun chunks per pair.Source: §A.3 · p. 11
  • The annotation interface lets users select proposed noun chunks or manually annotate garments.The manual annotation process on Dress Code yields three noun chunks for each model-garment pair.Source: §A.3 · p. 11, Figure 6 · p. 12
  • Top-3 accuracy rises from 12.95% to 16.60% after ViT-B/32 fine-tuning, versus 16.21% for the computationally heavier ViT-g/14.The authors note that the ViT-g/14 predictions help define the noun-chunk ground truth, so the actual performance difference may be higher.Source: §A.3 · p. 12

A.4. Extracting Sketches

  • For unpaired inputs, a warping module fits an in-shop garment to a target model using pose and masked-image representations to predict a thin-plate spline transformation.The transformation produces a coarse warped garment, which a U-Net then refines using the garment, pose map, and masked model image.Source: §A.4 · p. 12
  • The dataset appendix reports category-level image counts and illustrates the number of samples per category.Figure 7 highlights the different annotation strategies on Dress Code Multimodal.Source: Table 7 · p. 12, Figure 7 · p. 13
  • The refinement U-Net is trained on both datasets with L1 and perceptual losses; training converges after 30 VITON-HD epochs and 80 Dress Code epochs.Training uses 256 × 192 resolution, Adam, and a learning rate of 10^-4.Source: §A.4 · p. 12

A.5. Additional Statistics and Annotated Samples

The appendix summarizes dataset composition and annotation vocabulary across Dress Code Multimodal and VITON-HD Multimodal, and provides examples from the annotated subsets.

A.5. Additional Statistics and Annotated Samples

  • The datasets share noun chunks between training and test sets, which the authors attribute as likely reflecting limited textual capacity to represent the full image semantics.Table 7 summarizes image and unique noun-chunk counts by category.Source: §A.5 · p. 13
  • Word-cloud visualizations show that frequent annotation terms vary by garment category, while semantic richness is consistent across garment types.The visualizations cover Dress Code categories and VITON-HD clothing items.Source: §A.5 · p. 13
  • The appendix shows samples from Dress Code’s fine- and coarse-grained subsets and additional examples from VITON-HD Multimodal.Figures 11 and 12 show Dress Code examples; Figure 13 shows VITON-HD examples.Source: §A.5 · p. 13

B. Evaluation Metrics

The appendix specifies image-text, pose, and sketch evaluation measures and describes a user study comparing generated images for realism and multimodal coherence. The study collected about 7,000 evaluations from more than 150 users.

B. Evaluation Metrics

  • CLIP-S measures coherence between generated images and conditioning text using the image-caption embedding cosine similarity, scaled by 100 and floored at zero.The generated garment crop is pasted on a white background before embedding.Source: B. Evaluation Metrics · p. 13
  • Pose distance measures masked-region pose coherence through distances between corresponding keypoints from real and generated images, weighted by detector confidence.The keypoints are extracted with OpenPifPaf, and the metric ignores pairs unless both detector confidences reach 0.5.Source: B. Evaluation Metrics · p. 14
  • The proposed sketch distance compares edge maps of ground-truth and generated garments using mean squared error weighted by inverse pixel frequency.The authors avoid thresholding so the metric can compare hand-drawn grayscale ground-truth sketches.Source: B. Evaluation Metrics · p. 14

C. User Study

  • The user study separately asks participants to choose the more realistic image and the output more coherent with the multimodal inputs and model pose.It collected around 7,000 evaluations—about 3,500 per survey—from more than 150 users.Source: C. User Study · p. 14, Figure 9 · p. 14

D. Additional Results

Additional results report that MGD generally outperforms competitors, with a noted exception for unpaired pose metrics on Dress Code. Ablations show modality-specific metric responses and a trade-off between sketch adherence and realism; failure cases include small hands and ill-fitting warped sketches.

D. Additional Results

  • MGD outperforms competitors on Dress Code category-wise metrics except unpaired pose metrics, where imperfect warped-sketch fit to body shape is implicated.KID is described as more accurate than FID for categories with only 1,800 images; Table 8 provides the category-wise results.Source: D. Additional Results · pp. 14–15, Table 8 · p. 15
  • With input modalities masked, FID and KID remain comparable, while pose and sketch distances generally track whether pose maps and sketches are provided.On Dress Code, warped-garment mismatch also affects pose distance.Source: D. Additional Results · p. 15, Table 11 · p. 15
  • MGD outperforms ControlNet by a significant margin across configurations on both datasets and paired and unpaired settings.ControlNet is adapted to Stable Diffusion inpainting and conditioned on text plus either pose or sketch; Table 12 presents the comparison.Source: D. Additional Results · p. 15, Table 12 · p. 16
  • Qualitative comparisons show MGD alongside Stable Diffusion and SDedit at 512 × 384 resolution, and alongside Stable Diffusion and FICE at low resolution.The comparisons are shown for Dress Code and VITON-HD Multimodal.Source: D. Additional Results · p. 15, Figure 14 · p. 20, Figure 15 · p. 21
  • Increasing sketch-conditioning steps improves sketch adherence but slightly reduces garment realism; starting conditioning in central or final denoising steps yields sketch mismatch and artifacts.The time-window examples fix sketch conditioning to around one third of diffusion steps and vary the starting timestep.Source: D. Additional Results · p. 16, Figure 10 · p. 16, Figure 19 · p. 25
  • MGD sometimes renders hands inaccurately when they occupy little source-image area, a failure the authors attribute to latent diffusion models’ 8× spatial compression.Figure 20 shows the failure cases.Source: D. Additional Results · p. 16, Figure 20 · p. 25
  • When geometric warping produces a sketch that does not fit the model’s shape, generation also fails and can create artifacts.The examples include a sketch smaller than the body shape, producing an artifact near the left hand.Source: D. Additional Results · p. 16, Figure 20 · p. 25

Related papers

Loading 2304.02051v2…