Source-linked AI summary
PRISM: Learning Design Knowledge from Data for Stylistic Design Improvement
Huaxiaoyue Wang, Sunav Choudhary, Franck Dernoncourt, Yu Shen, Stefano Petrangeli
TL;DR
PRISM addresses stylistic design improvement when VLMs’ pretrained style knowledge is too general and misaligned with domain-specific design data. It clusters tagged designs, extracts actionable contrastive knowledge, and retrieves it during inference. On Crello, PRISM achieved the best average rank of 1.49 for style alignment, with user studies showing designer preference across multiple design axes.
Problem
VLMs’ pretrained style knowledge is often too general and lacks alignment with specific domain design data, complicating stylistic improvement from language instructions.
Method
PRISM clusters stylistically tagged designs, extracts distinctive actionable knowledge through contrastive summarization, and proportionally retrieves that knowledge during inference.
Results
1.49/5 average rank: PRISM achieves the best style-alignment performance, with user studies showing designers prefer it across multiple design axes.
Takeaways & Limitations
Learned design knowledge enables improvements that reflect the original design-data distribution while supporting diverse, style-aware outputs.
Takeaways & Limitations
PRISM treats design data as static and its optimal knowledge representation may depend on the downstream module.
Abstract
from arXiv · showhide
Graphic design often involves exploring different stylistic directions, which can be time-consuming for non-experts. We address this problem of stylistically improving designs based on natural language instructions. While VLMs have shown initial success in graphic design, their pretrained knowledge on styles is often too general and misaligned with specific domain data. For example, VLMs may associate minimalism with abstract designs, whereas designers emphasize shape and color choices. Our key insight is to leverage design data -- a collection of real-world designs that implicitly capture designer's principles -- to learn design knowledge and guide stylistic improvement. We propose PRISM (PRior-Informed Stylistic Modification) that constructs and applies a design knowledge base through three stages: (1) clustering high-variance designs to capture diversity within a style, (2) summarizing each cluster into actionable design knowledge, and (3) retrieving relevant knowledge during inference to enable style-aware improvement. Experiments on the Crello dataset show that PRISM achieves the highest average rank of 1.49 (closer to 1 is better) over baselines in style alignment. User studies further validate these results, showing that PRISM is consistently preferred by designers.
1. Introduction
Stylistic design improvement is difficult because designers explore multiple directions, while VLM style knowledge can be too general and poorly aligned with real-world design data. PRISM addresses this by learning actionable style knowledge from clustered design data and using it to guide improvements.
- Graphic design improvement requires exploring multiple stylistic directions, which is time-consuming and challenging for non-experts.
- Existing design-improvement methods mainly rearrange elements for visual appeal without conditioning edits on specific design styles.
- Designs sharing a stylistic tag can vary substantially, making direct extraction of fine-grained, actionable knowledge difficult.
- PRISM constructs a design knowledge base by partitioning style space, extracting contrastive knowledge, and retrieving proportional priors for inference-time edits.
- 0.847 → 0.999: PRISM achieves the highest average style alignment compared to the best baseline on Crello.
2. Related Works
Prior graphic-design systems improve aesthetics through evaluation or layout adjustments, but style-conditioned editing remains underexplored. PRISM instead uses learned design knowledge to guide edits across multiple design elements.
- Prior design-improvement methods primarily adjust asset positions and sizes rather than conditioning edits on specific design styles.
- PRISM targets stylistic improvement by using learned design knowledge to edit colors, fonts, assets, and layouts.
- The framework relates to LLM and VLM approaches that use pretrained knowledge or knowledge bases to support graphic-design generation and retrieval.
3. Approach
PRISM assumes stylistically tagged design data and converts its visual variation into retrievable knowledge for style-aware improvement. Its pipeline clusters designs, distills contrastive actionable guidelines, and retrieves them during inference.
- PRISM assumes a collection of designs annotated with stylistic tags such as “abstract” or “elegant.”
- Style Space Partitioning: Style Space Partitioning groups visually varied designs sharing a tag into meaningful clusters.
- Style Space Partitioning: GRAD represents designs as graphs of image patches and uses optimal transport to capture semantic and spatial similarities.
- Style Space Partitioning: K-medoids clusters same-tag designs using pairwise GRAD distances, with K selected by the highest silhouette score.
- Style Knowledge Extraction: Each cluster is distilled into actionable and discriminative design knowledge describing concrete visual attributes.
- Style Knowledge Extraction: Contrastive summarization uses nearby medoid examples as positives and other same-tag clusters as negatives to learn distinctive knowledge.
- Style Knowledge Extraction: The knowledge base indexes each entry by style tag, a single-sentence summary, and detailed guidelines for retrieval.
- Prior-Informed Editing: During inference, PRISM retrieves the closest knowledge for one variation or samples knowledge proportionally to cluster size for multiple variations.
4. Quantitative Experiments
PRISM is evaluated on Crello across 15 design styles using fidelity, diversity, expected rank, qualitative comparisons, and ablations. It achieves the strongest overall balance of fidelity and diversity, while data curation and contrastive knowledge extraction explain its advantage over data-driven baselines.
- Setup: Crello provides 15 major design styles, with training, validation, and test splits used to construct reference and evaluation data.The dataset contains 19,479 training designs, 1,852 validation designs, and 1,971 test designs.
- Metrics: Fidelity measures how closely generated improvements resemble real designs, while diversity measures whether they cover the real designs’ variability.Both metrics use the same GRAD visual distance; nearest-neighbor calculations use k = αN with α = 0.05, and results use 10,000 bootstrap resamples.
- Overall Results: 0.999 fidelity and 1.49/5 average rank make PRISM the strongest fidelity method, where rank 1 is always best.Prompt2Diverse records 0.744 average fidelity and a 4.07/5 rank.
- Overall Results: 0.683 diversity and a 2.64/5 rank make PRISM the strongest diversity method, outperforming Data2Diverse’s 2.75/5 rank.Methods injecting multiple design-knowledge variants form the higher-diversity ranking group, while OpenCole and Data2One rank 3.27 and 3.49.
- Data Curation: PRISM’s data curation yields lower input variance and more meaningful clusters than Data2One and Data2Diverse, with average distance 0.146 versus 0.536 and silhouette score 0.666.The baselines sample designs randomly, whereas PRISM clusters visually coherent designs before knowledge extraction.
- Ablations: Ablations show that negative examples improve fidelity, while combining individual designs with collages slightly improves diversity.The positive-only PRISM (No Neg) baseline has fidelity 0.965.
5. User Study
A user study with 30 designers compared PRISM against Data2Diverse on stylistic alignment and diversity. Designers evaluated outputs blindly across multiple design aspects.
- Study Setup: 30 designers compared PRISM and Data2Diverse using original designs, language instructions, blinded outputs, and reference examples.Each designer evaluated 15 randomly ordered style-specific test cases.
- Evaluation Criteria: Designers judged stylistic alignment across color schemes, decorations, and text layouts.They also assessed which approach produced more visually diverse improvements.
- Results: Designers consistently judged PRISM more effective for aligning improvements with requested design styles.The comparison used Data2Diverse as the most competitive baseline.
- Results: Designers selected PRISM as producing more diverse improvements 66.2% of the time.This result concerns the study’s visual-diversity judgment.
6. Conclusion
PRISM learns design knowledge from existing designs to address VLM style knowledge that is too general and poorly aligned with domain data. The authors identify static design data and downstream-dependent knowledge representations as limitations.
- Approach: PRISM clusters designs, uses contrastive learning to capture distinctive visual varieties, and proportionally retrieves knowledge during inference.The retrieval strategy aims for improvements reflecting the original design-data distribution.
- Limitations: PRISM treats design data as a static set even though the data may evolve over time.Future work is proposed for incorporating new information and updating design knowledge efficiently.
- Limitations: The optimal design-knowledge representation may depend on the downstream module.The authors propose learning an adapter using downstream execution signals.
7. Iterative Knowledge Refinement
Iterative knowledge refinement improves cluster-specific design knowledge by detecting misaligned examples, generating feedback, and editing the knowledge across iterations. A classifier supplies contrastive signals for this process.
- Overall Approach: The refinement process contrasts positive designs from a cluster with negative designs from other clusters to extract discriminative knowledge.The goal is knowledge that aligns with its own cluster while distinguishing other clusters.
- Overall Approach: A design classifier identifies false negatives and false positives by testing whether designs align with cluster knowledge.Each cluster is treated as a class with associated learned knowledge.
- Knowledge Refinement: Feedback is generated for misclassified positive and negative examples, then accumulated for the next refinement step.The feedback generator analyzes misclassifications and proposes actionable improvements.
- Knowledge Refinement: The knowledge refiner analyzes accumulated feedback and edits the current knowledge for the next iteration.This produces progressively refined knowledge through repeated classifier-feedback-refinement cycles.
- Implementation Details: The classifier compares design knowledge pairwise and assigns the label with the highest number of wins.It returns c(k, X)=1 when knowledge k best matches a design and 0 otherwise.
8. User Study
The per-style study shows that PRISM was preferred across most styles, although Data2Diverse led on some questions for three styles and on all questions for one style.
- Per-Style Results: 11/15 styles had users preferring PRISM over Data2Diverse across all questions.The questions concerned stylistic alignment and visual diversity.
- Per-Style Results: 3/15 styles—colorful, floral, and simple—favored Data2Diverse on one or two questions.These were exceptions rather than across-the-board baseline preferences.
- Per-Style Results: 1/15 styles—classic—favored Data2Diverse on all questions.The remaining styles did not show this complete baseline preference.
9. Quantitative Experiments
PRISM is evaluated across design styles using user preferences, fidelity and diversity, and iterative knowledge refinement. It leads in fidelity across most styles, while refinement improves both fidelity and diversity on a qualitative failure case.
- Per-Style Results: PRISM has the highest fidelity on 11/15 styles and the highest diversity on 5/15 styles.These per-style results match the expected-rank results across all styles.
- Iterative Knowledge Refinement: The refinement experiment targets styles where PRISM lacks the highest fidelity and diversity, including colorful, dynamic, graphic, and modern.PRISM’s value is close to the baselines for bright, classic, and floral.
- Iterative Knowledge Refinement: Iteration 1 raises fidelity from 0.898 to 0.978 and diversity from 0.610 to 0.658 on a refined qualitative example.The refinement adds details, emphasis, and guidelines to make the knowledge more concrete and discriminative.
- Per-Style Results: Figure 12 reports bootstrapped average fidelity and diversity for each of the 15 styles and categorizes styles by which methods achieve the highest values.The categories distinguish cases where PRISM leads on both metrics, fidelity alone, or neither metric.
- Iterative Knowledge Refinement: Figure 13 contrasts iteration 0 and iteration 1 knowledge, showing that refined knowledge distinguishes a negative example that iteration 0 misclassifies.The figure also includes qualitative examples of improved designs guided by the learned knowledge.