PaperlayerView exact revision on arXiv

Source-linked AI summary

Beyond Atomic Layouts: Compositional Design Understanding with Vision-Language Models

Yiyang Huang, Zhaowen Wang, Simon Jenni, Jing Shi, Yitian Zhang, Yizhou Wang, Yun Fu

arXiv:2608.26716v1cs.CV

TL;DR

Existing VLMs perform well on atomic layouts but struggle with visually entangled elements and hierarchical, layer-aware relationships in compositional layouts. The paper introduces CoDeLayout and MASON, which combines multimodal alignment with structural perception. Qwen2.5-VL 7B with MASON reaches 91.66% accuracy versus 79.68% for GPT-o3 and outperforms full-data Direct Finetune with 30% of the training data.

  • Problem

    Existing layout methods focus on independent atomic elements, leaving compositional layouts with visual entanglement and hierarchical relationships underexplored.

  • Method

    The paper introduces CoDeLayout and MASON, whose multimodal alignment grounds metadata-defined elements while structural perception extracts layer-aware spatial relationships.

  • Results

    91.66% accuracy is achieved by Qwen2.5-VL 7B with MASON, compared with 79.68% for GPT-o3; MASON also surpasses full-data Direct Finetune using 30% of training data.

  • Takeaways & Limitations

    CoDeLayout establishes a benchmark for compositional layout understanding, while MASON improves performance across compositional categories and uses training data more efficiently.

  • Takeaways & Limitations

    Future work is needed on layer-aware visual encoders and stronger cross-modal alignment objectives for multilayer compositional layouts.

Abstract

from arXiv · show

Layout understanding, or the interpretation of element organization, is essential for document analysis, user interface (UI) creation, and graphic design. While recent vision-language models (VLMs) excel at interpreting atomic layouts composed of independent elements, they struggle with compositional layouts that require reasoning over visually entangled elements within hierarchical multi-layer structures. In this paper, we introduce a new task, compositional layout understanding, and present CoDeLayout, a VQA dataset of ~20K real-world multi-layer layouts annotated with compositional element pairs and design intent. Through empirical analysis on CoDeLayout, we identify two key challenges for existing VLMs: semantic drift between textual metadata and visual content, and structural ambiguity in hierarchical inter-element relationships. To address these challenges, we propose MASON, a post-training paradigm that integrates multimodal alignment (MA) and structural perception (SP). MA enhances element interpretation by grounding metadata-defined elements to their visual counterparts, mitigating semantic drift, while SP models layer-aware inter-element spatial relationships to improve hierarchical understanding and reduce structural ambiguity. Experiments reveal substantial gaps in existing VLMs: even the strongest baseline, GPT-o3, achieves only 79.68% accuracy, whereas Qwen2.5-VL 7B with MASON reaches 91.66%. Notably, MASON surpasses full-data Direct Finetune using only 30% of the training data and scales better with additional data.

1 Introduction

Existing VLM research handles atomic layouts well but leaves compositional layouts underexplored because visually entangled, hierarchical elements require layer-aware reasoning. This paper introduces CoDeLayout and MASON to address semantic drift and structural ambiguity, with MASON substantially improving accuracy and data efficiency.

  • Motivation: Compositional layouts entangle multiple elements within hierarchical structures, requiring layer-aware spatial reasoning beyond atomic-layout methods.Representative operations include clipping, blending, overlaying, and morphing.
  • Challenges: Empirical analysis identifies semantic drift and structural ambiguity as two key reasons existing VLMs struggle with compositional layouts.Semantic drift concerns metadata-to-visual grounding, while structural ambiguity concerns hierarchical, layer-aware relationships.
  • Method: MASON integrates multimodal alignment and structural perception to ground metadata-defined elements and model layer-aware spatial dependencies.Multimodal alignment targets semantic drift; structural perception targets structural ambiguity.
  • Results: 91.66% accuracy is achieved by Qwen2.5-VL 7B with MASON, compared with 79.68% for the strongest baseline, GPT-o3.MASON also surpasses full-data Direct Finetune using only 30% of the training data and scales better as data increases.
  • Contributions: CoDeLayout introduces compositional layout understanding as a new VLM task and provides approximately 20K real multi-layer layouts with element-pair and design-intent annotations.The dataset pairs rendered layouts with element-level metadata and QA-style annotations.

2 Related Work

Prior layout research spans VLMs, layout generation, and layout understanding, but most datasets and methods assume independent atomic elements. CoDeLayout instead targets hierarchical, layer-aware interactions among visually entangled elements.

  • Vision-Language Models: VLM research combines language models with visual encoders and includes interleaved reasoning, bridged vision-language backbones, and instruction-tuned models.Examples include Flamingo, BLIP-2, LLaVA, and Qwen-VL.
  • Layout Generation: Layout generation methods synthesize document, interface, and graphic-design layouts from learned structural patterns or textual and visual inputs.Representative systems include LayoutVAE, LayoutGAN++, LayoutDM, Text2Poster, Pix2Code, and Design2Code.
  • Compositional Layouts: CoDeLayout represents compositional designs through rendered images, element-level metadata, and QA annotations for compositional element pairs and design intents.The dataset focuses on four compositional types: Overlaying, Clipping, Blending, and Morphing.
  • Layout Understanding: Layout understanding datasets and models support document analysis and UI navigation by encoding semantic and spatial organization with 2D positional representations.Examples include PubLayNet, DocLayNet, LayoutLM, DiT, RICO, and ScreenQA.
  • Research Gap: Existing layout understanding methods assume independent elements, limiting their ability to model hierarchical and layer-aware interactions among visually entangled elements.This limitation motivates a dedicated compositional-layout dataset and modeling paradigm.

3 Compositional Layout Understanding: Task & Dataset

Compositional layout understanding identifies interacting elements in hierarchical layouts and explains their design intent. CoDeLayout provides multi-layer designs with explicit compositional relationships, while analysis highlights semantic drift and structural ambiguity as key challenges.

  • Task Formulation: Compositional layout understanding identifies interacting elements within hierarchical layouts and generates answers explaining their design intent.
  • Comparison with Existing Layout Datasets: Existing layout datasets primarily target atomic layouts or generation, whereas CoDeLayout explicitly annotates inter-element compositional relationships and design intent.
  • Dataset: CoDeLayout is a high-quality, multi-layer dataset centered on element-level compositional relationships.
  • Dataset: CoDeLayout includes rendered designs, element-level metadata, and QA annotations specifying compositional element pairs and design intent.
  • Challenges: Semantic drift reflects failures to align textual metadata with visual content, causing incorrect interpretation of element semantics and design intent.
  • Challenges: Higher grounding and spatial-relation perception performance consistently correspond to better compositional element identification accuracy.

4 MASON: A Baseline Post-training Paradigm

MASON integrates multimodal alignment and structural perception into VLM post-training to address semantic drift and structural ambiguity in compositional layouts.

  • 4 MASON: A Baseline Post-training Paradigm: MASON combines multimodal alignment (MA) and structural perception (SP) as a post-training paradigm for compositional layout understanding.MA grounds metadata-defined elements to visual counterparts, while SP models layer-aware spatial relationships.
  • 4.1 Multimodal Alignment (MA): MA introduces an element-grounding objective that aligns textual metadata with corresponding visual content under visual entanglement.The objective uses grounding QA supervision generated for sampled layout elements.
  • 4.1 Multimodal Alignment (MA): The grounding data samples 20% of elements from 1K training layouts and generates QA pairs describing each element’s semantic identity and spatial placement.Each instance includes the rendered layout, an element crop, element metadata, and a grounding question.
  • 4.2 Structural Perception (SP): SP augments element metadata with layer-aware relationships to the query element, including overlap, containment, distance, direction, and stacking order.These attributes provide explicit geometric context for interpreting layout hierarchy and inter-element structure.
  • 4.3 Integration into VLM Post-training: Post-training jointly supervises compositional and grounding QA tasks using metadata augmented with spatial relationships, while inference retains the augmentation for compositional evaluation.The training objective uses either a compositional or grounding question and optimizes cross-entropy against the corresponding answer.

5 Experiments

The experiments evaluate heuristic, open-source, proprietary, and post-trained VLMs on CoDeLayout under standardized compositional layout understanding settings.

  • 5 Experiments: The evaluation compares heuristic methods, 7B open-source VLMs, proprietary large-scale models, and two Qwen2.5-VL 7B post-training variants.The post-training comparison contrasts Direct Finetune with MASON.
  • 5 Experiments: Heuristic baselines select elements using maximum overlap or nearest-centroid proximity, while VLM baselines use zero-shot or default inference settings.Reasoning-capable proprietary models are evaluated with their default medium reasoning configuration.
  • 5 Experiments: The evaluation measures compositional element identification with Accuracy and explanation quality with GPT-Score, BLEU, and ROUGE.GPT-Score assesses semantic alignment with annotated design intent, whereas BLEU and ROUGE measure lexical overlap.

Metrics.

On CoDeLayout, existing VLMs show substantial performance gaps, while MASON achieves higher accuracy, remains data-efficient, and performs better as training data increases.

  • 5.3 Comparison on CoDeLayout: 79.68% weighted accuracy and 76.28% average accuracy are achieved by GPT-o3, the strongest baseline on CoDeLayout.These results underscore the difficulty of modeling visually entangled and layer-aware structures.
  • 5.3 Comparison on CoDeLayout: Table 4 reports category-level Accuracy alongside GPT-Score, BLEU, and ROUGE, with weighted and average accuracy summarizing performance across categories.Weighted accuracy uses test sample proportions, while average accuracy is the unweighted mean across categories.
  • 5.3 Comparison on CoDeLayout: 91.66% weighted accuracy and 88.07% average accuracy are achieved by full-data MASON, compared with GPT-o3’s 79.68% and 76.28%.MASON especially improves the Clipping category, reaching 90.91% versus GPT-o3’s 63.64%.
  • 5.3 Comparison on CoDeLayout: 89.32% weighted accuracy and 86.12% average accuracy allow MASON trained on 30% of the data to surpass full-data Direct Finetune at 88.80% and 84.30%.MASON uses approximately 8K QA pairs, including 6K compositional and 2K grounding pairs, versus 20K for Direct Finetune.
  • 5.3 Comparison on CoDeLayout: MASON consistently outperforms Direct Finetune across training-data scales, with a clear advantage under low-resource settings.The comparison plots weighted test accuracy and MASON’s accuracy gain over Direct Finetune for 10%, 15%, 30%, and 100% of training data.

5.4 Ablation Study

The ablation study finds that multimodal alignment and structural perception each improve accuracy, with their combination providing complementary gains across compositional layouts.

  • 5.4 Ablation Study: 84.11% weighted accuracy and 83.40% average accuracy are achieved when multimodal alignment and structural perception are combined in MASON.Direct Finetune reaches 80.21% weighted and 78.36% average accuracy under the type-balanced setting.
  • 5.4 Ablation Study: 82.03% weighted and 80.32% average accuracy with MA, followed by 83.85% and 82.51% with SP, show separate gains over Direct Finetune.The results indicate improvements in element-level alignment and inter-element spatial-structure modeling, respectively.
  • 5.4 Ablation Study: MA and SP provide complementary improvements, but their effects vary across composition types such as Overlaying, Clipping, Blending, and Morphing.The passage reports that noisy grounding can limit MA for heavily occluded compositions, while SP helps recover text-replacement relations in Morphing.

Module Ablation

The ablations examine MASON’s data efficiency and whether its gains depend on the alignment paradigm or a particular grounding model. Across increasing training-data scales, MASON remains advantageous over Direct Finetune.

  • Grounding Model: Comparable performance across grounding models suggests that MA gains arise from the alignment paradigm rather than grounding-model strength.The ablation compares GPT-4o with Qwen3-VL under direct and script-based grounding settings.
  • Data Scale: MASON consistently outperforms Direct Finetune across training-data scales from 10% to 100%.At 10%, Direct Finetune suffers a notable accuracy drop, while MASON maintains a clear advantage.
  • Data Scale: MASON surpasses full-data Direct Finetune using only 30% of the training data.The smaller subsets use type-balanced subsampling.
  • Data Scale: MASON continues benefiting from additional supervision beyond 30%, whereas Direct Finetune shows diminishing returns.This indicates better scalability as training data increases.

Grounding Model Ablation

Visual input is critical for compositional layout understanding, because removing visual features substantially reduces performance for both Direct Finetune and MASON.

  • Visual Dependency: Removing visual features causes a substantial performance drop for both Direct Finetune and MASON.The result indicates that metadata and layer-aware spatial relationships alone do not resolve the task.

6 Case Study

Case studies compare Direct Finetune and MASON across four compositional categories, showing MASON’s ability to identify compositional pairs when spatial or semantic cues are subtle.

  • Case Studies: Fig. 6 compares Direct Finetune and MASON predictions across overlaying, clipping, blending, and morphing cases.Each example shows the full design, query element, and both model predictions.
  • Case Studies: MASON identifies underlying compositional pairs that Direct Finetune often fails to localize or associate.The contrast is most apparent when spatial or semantic cues are subtle.
  • Clipping: In the clipping example, MASON selects the cutout element while Direct Finetune selects a decorative element matching the base image width.The case tests recognition of an implicit relationship between the cutout and base image.

7 Conclusion

The paper introduces compositional layout understanding and CoDeLayout, then proposes MASON to address semantic drift and structural ambiguity through multimodal alignment and structural perception. Experiments report consistent improvements and data efficiency, while future work targets specialized visual encoders and stronger alignment objectives.

  • Contributions: The paper defines compositional layout understanding as reasoning over visually entangled elements and layer-aware relationships in hierarchical layouts.CoDeLayout is introduced as a dataset dedicated to this setting.
  • Method: MASON integrates multimodal alignment and structural perception to address semantic drift and structural ambiguity.The approach combines grounding-based alignment supervision with layer-aware spatial relationships.
  • Results: MASON consistently outperforms open- and closed-source VLMs, including reasoning-capable models, while remaining data-efficient under limited supervision.These results are reported across extensive experiments.
  • Future Work: Future work includes layer-aware visual encoders for multilayer layouts and stronger cross-modal alignment objectives.These directions target element-level grounding and relational reasoning.
Loading 2608.26716v1…