Source-linked AI summary
Grounding Free-Form Instructions for Fashion Complementary Image Generation
Matteo Attimonelli, Claudio Pomo, Alessandro De Bellis, Danilo Danese, Dietmar Jannach, Tommaso Di Noia
TL;DR
Fashion CIG benchmarks have relied on rigid templates that do not reflect free-form user queries or reveal behavior across linguistic specificity. The paper introduces instruction-enriched benchmarks and StyleFlow, a jointly conditioned Rectified Flow Matching model. Across multiple evaluations, StyleFlow produces instruction-aligned, stylistically coherent garments while reducing architectural complexity and inference cost relative to auxiliary-module approaches.
Problem
Fixed-template CIG benchmarks do not test grounding of natural-language user intent across specificity levels.
Method
The paper enriches three CIG benchmarks with human-validated low-, medium-, and high-specificity instructions and proposes StyleFlow for this setting.
Results
StyleFlow consistently produces instruction-aligned and stylistically coherent garments while reducing architectural complexity and inference cost relative to auxiliary-module approaches.
Takeaways & Limitations
Free-form instructions provide a controlled basis for evaluating language-grounded complementary garment generation.
Takeaways & Limitations
Auxiliary conditioning designs used for comparison increase architectural complexity and memory overhead.
Abstract
from arXiv · showhide
Fashion complementary image generation (CIG) aims to create garments that stylistically match a seed item based on user intent, making it a natural multimodal grounding problem where models must interpret language in visual context. Existing CIG benchmarks rely on rigid template prompts (e.g., "a photo of a skirt"), failing to reflect natural user queries and obscuring model behavior across levels of linguistic specificity. We introduce fashion complementary image generation with free-form instructions, a multimodal language-grounding setting where a model generates a compatible garment from a seed image and a natural-language instruction. To this end, we enrich three CIG benchmarks with low-, medium-, and high-specificity instructions generated by a vision-language model and validated by human annotators. We instantiate the task with StyleFlow, a Rectified Flow Matching model that jointly conditions on the seed image and instruction within a single multimodal transformer. Across image quality metrics, catalog-alignment analysis, ablations, and human evaluation, StyleFlow consistently produces instruction-aligned and stylistically coherent garments while reducing architectural complexity and inference cost relative to auxiliary-module approaches.
1 Introduction
The paper reframes fashion complementary image generation as multimodal grounding with free-form instructions, addressing the mismatch between natural user queries and fixed category-template benchmarks. It introduces controlled instruction-enriched benchmarks and StyleFlow, which jointly conditions on seed images and instructions while producing aligned, coherent garments.
- Motivation: Existing CIG methods use fixed category templates, so benchmarks do not test grounding of free-form user intent across specificity levels.User queries can range from underspecified categories to detailed descriptions of material, fit, and visual attributes.
- Model contribution: StyleFlow uses Rectified Flow Matching and a single multimodal transformer to jointly condition on the seed image and free-form instruction without auxiliary modules.The unified design uses 20 inference steps and supports direct ablations of language and visual inputs.
- Evaluation: Across image-quality metrics, catalog alignment, ablations, and human studies, StyleFlow produces instruction-aligned and stylistically coherent garments while reducing complexity and inference cost.The reported evaluation combines automated measures with human judgments of realism, compatibility, and authenticity.
- Problem formulation: The paper formalizes CIG with free-form instructions as multimodal grounding under varying linguistic specificity.The task generates a complementary garment from a seed item while interpreting textual intent in visual context.
- Benchmark contribution: Three CIG benchmarks are enriched with low-, medium-, and high-specificity instructions generated by a VLM and validated by human annotators.Validation covers fluency, informativeness, and visual grounding, enabling controlled evaluation of instruction-driven generation.
2 Proposed Approach
The approach defines complementary generation as producing a target garment compatible with a seed image and aligned with a free-form instruction. It constructs controlled multi-specificity instructions and implements StyleFlow with Rectified Flow Matching, joint multimodal conditioning, and fixed-step latent generation.
- Task definition: The task generates a target garment that is stylistically compatible with a seed image and semantically aligned with a free-form instruction.The seed supplies compatibility cues, while the instruction constrains target attributes.
- Instruction setting: Instructions range from minimal category descriptions to detailed style-rich descriptions while remaining paired with seed and target garments.This design exposes varying levels of specificity for controlled study.
- Instruction synthesis: A VLM first creates structured captions of target attributes and then rewrites them into low-, medium-, and high-detail free-form instructions.The described attributes include category, color, fit, material, and style.
- Rectified Flow Matching: StyleFlow models deterministic flows between source noise and data latents with a learned time-dependent velocity field.Rectified Flow Matching uses a linear path between noise and target samples to simplify the flow dynamics and support conditional generation.
- StyleFlow: StyleFlow jointly injects seed-image and instruction conditioning into a multimodal transformer for latent garment generation.The seed latent and instruction are processed alongside the model's flow-based generation, with the result decoded through a pretrained VAE.
- Training: The model adapts a pretrained FLUX backbone using LoRA while keeping the pretrained backbone frozen.LoRA is applied to the MM-DiT attention blocks and jointly optimized with newly introduced parameters.
3 Experiments
The experiments evaluate free-form instruction-driven CIG across three benchmarks using automated metrics, catalog alignment, ablations, and human studies. Results show that richer instructions improve generation and catalog alignment, while StyleFlow remains competitive and benefits strongly from both language and seed-image conditioning.
- Experimental setup: The evaluation spans three fashion compatibility benchmarks, instruction-specificity levels, image-quality and alignment metrics, ablations, and human studies.The benchmarks are FashionVC, ExpFashion/ExpReduced, and FashionTaobao-TB; instructions are evaluated at low, medium, and high specificity.
- Experimental setup: StyleFlow is compared with DiFashion, GeCo, MGCM, and Pix2PixCM using matched experimental protocols.DiFashion is adapted to use the top image and free-form instructions, while the other baselines retain their established generative formulations.
- Generation quality: FID decreases from 46.05 to 33.88/33.21 on FashionVC as StyleFlow prompts move from low to medium/high specificity.The same richer-prompt trend appears on ExpReduced and FashionTaobao-TB, alongside improved perceptual similarity and CLIP-Score.
- Catalog alignment: Recall@50 rises from 0.5348 to 0.8806 to 0.9339 on FashionVC as instruction specificity increases from low to medium to high.The corresponding increases are 0.5578 → 0.9047 → 0.9359 on ExpReduced and 0.3858 → 0.6042 → 0.6904 on FashionTaobao-TB.
- Human evaluation: StyleFlow leads compatibility at 3.24 ± 1.29 and has a 27.9% fake rate, outperforming the listed baselines and approaching ground-truth judgments.Human evaluation also describes the generated garments as instruction-aligned, perceptually convincing, and compatible with the seed.
- Ablation study: Removing instructions raises StyleFlow FID by 189.9% and lowers Recall by 81.7%, showing substantial dependence on textual guidance.The corresponding changes are much smaller for GeCo and MGCM, indicating weaker sensitivity to textual cues in those baselines.
- Ablation study: Blanking the seed image increases StyleFlow FID by 318.2%, 32.0%, and 30.4% under low, medium, and high detail prompts on ExpReduced.The effect is largest for low-detail prompts, whereas richer language reduces the relative impact of removing visual context.
4 Related Work
Fashion CIG has progressed from GANs to diffusion-based models, but complementary generation remains distinct from image editing because it must create a different garment category while preserving compatibility.
- GAN-based CIG methods translated seed features into compatible targets but were limited in resolution, realism, and controllability.
- Latent Diffusion Models improved fidelity and controllability and became widely adopted for fashion image generation tasks.
- Flow Matching learns deterministic mappings through ODE-based probability paths, enabling stable training and faster sampling.
- Prior image-editing methods inject visual conditions through control branches, adapters, or instruction-specific fine-tuning of text-to-image backbones.
- Complementary generation differs from editing because its target belongs to a different garment category rather than preserving the input’s spatial structure.
5 Conclusion
The paper reformulates CIG around seed garments and natural-language queries, enriches benchmarks for controlled specificity evaluation, and introduces StyleFlow as a unified multimodal model. StyleFlow generates realistic, compatible, instruction-aligned garments while avoiding auxiliary conditioning branches, with future work targeting broader pairings and real user interaction data.
- The work conditions complementary garment generation on a seed garment and a natural-language query.
- Three CIG benchmarks were enriched with multiple instruction-specificity levels to evaluate language detail, catalog alignment, and visual grounding.
- StyleFlow jointly conditions on seed images and free-form instructions within a multimodal transformer without auxiliary conditioning branches.
- StyleFlow produces realistic, compatible, and instruction-aligned garments.
- Future work will extend the setting beyond top–bottom pairs and incorporate real user queries, preferences, and interaction histories.
A More Details On Flow Matching
Flow Matching models deterministic probability paths from a simple base distribution to data using an ODE and a learned velocity field. Rectified Flow Matching uses a rectified path to simplify this field, while StyleFlow applies the approach with a 20-step sampling schedule for instruction-driven CIG.
- Flow Matching interpolates between a simple base distribution and the data distribution along deterministic probability paths.
- Sampling integrates an ordinary differential equation from noise to data using a time-dependent velocity field parameterized by a neural network.
- Conditional Flow Matching trains the velocity field using paired samples from the base and target distributions and a tractable surrogate objective.
- Rectified Flow Matching enforces a rectified path, such as a straight-line trajectory, between a Gaussian prior and the data distribution.
- Reducing probability-path curvature simplifies velocity-field approximation and can support stable generation with fewer integration steps.
- 20 steps: StyleFlow uses this sampling schedule while maintaining strong fidelity and instruction alignment.
B More Examples on Free-Form Instruction Synthesis
The instruction-synthesis pipeline first captions garment attributes with a vision-language model, then rewrites those captions into low-, medium-, and high-detail free-form instructions. Manual verification and leakage-controlled test splits support consistent evaluation of how instruction detail affects CIG.
- The first synthesis stage prompts a model to describe garment color, fit, material, type, and style in an image caption.
- The second stage reformulates captions into high-, medium-, and low-detail instructions by progressively reducing attribute coverage.
- Generated instructions were manually verified for correctness, fluency, and relevance, with incomplete or inconsistent outputs revised or regenerated.
- The enrichment pipeline was applied uniformly across FashionVC, ExpReduced, and FashionTaobao-TB.
- Figure 4 presents examples of prompts used to generate free-form instructions at different detail levels.
- Test splits excluded bottom garments appearing in training under different pairings to prevent leakage and enable controlled evaluation of instruction detail.
C Human Evaluation Protocol Details
The human evaluations used web-based trials to assess instruction quality, image quality, garment compatibility, and perceived authenticity of generated outputs.
- Study Procedure: Both studies recruited researchers through institution-hosted interfaces and required informed consent before participation.Participants were informed that the studies had no commercial purpose and that collected information would comply with privacy regulations.
- Study Procedure: Each participant completed 8 trials, while the backend stored ratings and session or device metadata but no direct identifiers.Stored fields included a random session identifier, browser user agent, screen resolution, sampled items, and submitted ratings.
- Instruction Evaluation Study: Participants rated three automatically generated descriptions for each reference garment on quality and alignment with the image.Quality was summarized in terms of clarity, fluency, and informativeness.
- Generation Evaluation Study: Participants evaluated three candidate bottom garments paired with each top garment for visual quality, compatibility, and whether the image appeared generated.Candidate order was randomized, and candidates were sampled uniformly when more than three were available.
D Additional Implementation Details
The implementation uses three established top–bottom compatibility benchmarks, standard image-generation and catalog-alignment metrics, adapted instruction-capable baselines, and controlled training settings.
- Datasets and Splits: Experiments use FashionVC, ExpReduced, and FashionTaobao-TB, with disjoint target bottom garments across train, validation, and test splits.The datasets contain 18,640, 18,640, and 88,326 pairs respectively, at resolutions of 128×128, 224×224, and 512×512.
- Metric Computation: Generation quality is measured with FID, KID, LPIPS, and CLIP-Score, while catalog alignment uses MRR, Recall, and nDCG from SigLIP embeddings.FID and KID use InceptionV3 features; catalog items are ranked by cosine similarity.
- Baselines: DiFashion is adapted for joint top-image and instruction conditioning, while GeCo and MGCM receive pretrained CLIP text encoders.Baseline hyperparameters were explored within ranges specified in their respective papers.
- Implementation-Related Findings: StyleFlow maintains strong performance and robustness across all prompt types despite challenges in detailed settings.The supplied passage states this conclusion without reporting a numerical value.
- Training Setup: StyleFlow and DiFashion use fixed seed 0, other baselines use seed 42, and StyleFlow runs on one NVIDIA H100 GPU with classifier-free guidance scale 3.5.StyleFlow training uses batch size 1, eight-step gradient accumulation, bfloat16 precision, and prompt dropout probability 0.1.
E More Examples of Generations
Qualitative examples show that StyleFlow generates coherent complementary bottom garments, with richer instructions improving fine-grained attribute alignment and sparse prompts reducing specificity.
- Qualitative Results: StyleFlow consistently generates realistic, visually coherent bottom garments aligned with both the reference top and the instruction across prompt granularities.The examples cover low, medium, high, and template-based settings.
- Qualitative Results: Detailed and medium instructions particularly improve capture of fine-grained attributes such as color, fit, and style.These settings provide richer semantic cues than low-detail prompts.
- Prompt Specificity: Template prompts behave more like low-specificity prompts: outputs remain compatible but show reduced specificity compared with richer instructions.The template setting supplies minimal detail and guidance.
- Baseline Comparison: DiFashion changes little across prompt granularities, while GAN-based baselines often produce blurrier or less plausible outputs.DiFashion also occasionally introduces irrelevant elements.