PaperlayerView exact revision on arXiv

Source-linked AI summary

Multimodal Cultural Heritage Architectural Style Classification for Residential Buildings in the UAE Based on CLIP Embeddings and SVM

Ahmed Ammar Kubba, Manar Abu Talib, Iman Ibrahim, Qassim Nasir

arXiv:2609.17181v1cs.CVcs.AI

TL;DR

Architectural style classification has often relied on visual data, while multimodal analysis of detailed styles in region-specific settings remains underexplored. This paper combines CLIP image and text features for UAE residential buildings, then applies visualization, clustering, and SVM classification; it reports 98% accuracy across eight style clusters.

  • Problem

    Prior architectural classification commonly focuses on visual data, while detailed multimodal style analysis for residential buildings in distinct regional contexts such as the UAE remains underexplored.

    Source: §II · p. 2

  • Method

    The paper combines CLIP visual and textual features for UAE residential buildings, uses UMAP and K-Means with manual cluster labeling, and trains an SVM classifier.

    Source: §II · p. 2, §III · p. 3, E. Clustering and Classification · p. 4

  • Results

    98% accuracy across eight identified style clusters is reported for the SVM classifier.

    Source: §II · p. 2

  • Takeaways & Limitations

    The authors present a scalable, interpretable pipeline for cultural heritage analysis tailored to the UAE’s architectural identity.

    Source: §II · p. 2

    The 634-building dataset’s coverage could be expanded across more UAE regions and historical periods to enhance generalizability and robustness.

    Source: §V · p. 7

Abstract

from arXiv · show

The analysis and classification of cultural heritage architectural styles remain challenging due to the complexity of visual images of buildings, which are highly relied on in traditional CNN-based classification approaches in comparison to textual descriptions, and the relative lack of non-western region-specific datasets. This paper addresses this gap by proposing a multimodal machine learning framework to analyze and classify Emirati residential architecture using OpenAI's CLIP model. We integrate visual features from images and textual features from expert descriptions into a unified 512-dimensional embedding, followed by dimensionality reduction with UMAP for visualization and unsupervised clustering using K-Means. Cluster labels, which are derived from manual analysis of the K-Means clusters, are used to train an SVM classifier for automated architectural style classification. Our approach achieves a classification accuracy of 98% across eight identified style clusters, higher than every other study in the literature, demonstrating the effectiveness of combining visual and textual modalities. Overall, this paper highlights the potential of using multimodal AI to support architectural heritage analysis, offering scalable and interpretable tools for exploring regional architectural identities.

II. RELATED WORK

Prior architectural classification work has largely relied on visual CNN approaches, while multimodal analysis of detailed architectural styles in region-specific contexts remains underexplored. This work applies a multimodal CLIP pipeline to UAE residential buildings and reports 98% accuracy across eight style clusters.

II. RELATED WORK

  • Architectural style classification commonly uses visual CNNs, which can overlook textual information such as historical descriptions, design specifications, and local context.Prior examples report accuracies of 84.66%, 77%, and 96.17%, alongside other results of 94.7%, 90%, and 95%.Source: §II · p. 2
  • Although multimodal learning has architectural applications, detailed style classification for residential buildings in a distinct regional context such as the UAE remains an open research area.A cited multimodal precedent integrates Computer-Aided Design data and reports 97% accuracy.Source: §II · p. 2
  • The proposed contribution uses CLIP with 634 UAE buildings and expert-written descriptions, combining visual and textual features before UMAP, K-Means labeling, and SVM classification.The authors characterize the pipeline as scalable and interpretable for cultural heritage analysis.Source: §II · p. 2
  • Table 1 supports the reported 98% accuracy across eight style clusters and the claim that this work outperforms all previous research.The authors also describe it as one of only two multimodal approaches in the comparison.Source: Table 1 · p. 2, §II · p. 2

III. METHODOLOGY

The methodology combines visual and textual information, reduces dimensionality for visualization, groups datapoints through clustering, and uses the labels to enable architectural style classification training.

III. METHODOLOGY

  • The workflow integrates visual and textual information, reduces dimensionality for visualization, labels datapoints through clustering, and supports architectural style classification training.Figure 1 illustrates the study’s major stages.Source: §III · p. 3, Figure 1 · p. 3
  • The methodology is presented as a staged process for integrating multimodal information and preparing datapoints for classification.The overview identifies integration, dimensionality reduction, visualization, clustering, and classification training as its aims.Source: §III · p. 3
  • Figure 1 summarizes the major stages of the study’s methodology.The text identifies the figure as an overview of the process.Source: Figure 1 · p. 3, §III · p. 3

A. Data Collection and Preprocessing

The study pairs UAE residential building images with expert-written descriptions, then uses CLIP to derive image and text embeddings for multimodal analysis.

A. Data Collection and Preprocessing

  • The dataset contains 634 residential building images captured in Sharjah, Dubai, and Abu Dhabi, each paired with domain experts’ descriptions of architectural features, styles, and context.The text files contain multiple keywords and sentences.Source: A. Data Collection and Preprocessing · p. 3
  • The feature extraction process uses the pretrained CLIP ViT-B/32 model to obtain visual and textual representations.The paper describes CLIP’s joint-embedding capability as the reason for its use.Source: A. Data Collection and Preprocessing · p. 3
  • Each image is converted to RGB, resized, cropped, and normalized before CLIP generates a 512-dimensional visual embedding that is L2-normalized.Source: A. Data Collection and Preprocessing · p. 3
  • Text descriptions are split at periods, embedded in segments, normalized, and averaged into one textual embedding per description.The segmentation handles varied description lengths under CLIP’s fixed-size embedding requirement.Source: A. Data Collection and Preprocessing · p. 3

C. Multimodal Feature Fusion

The study fuses CLIP image and text features using a weighted average, selecting equal weights based on an ablation of classification metrics.

C. Multimodal Feature Fusion

  • Image and text features are combined through a weighted average, with the ablation used to select the weighting ratio for the SVM classifier.Source: C. Multimodal Feature Fusion · pp. 3–4
  • At 50% image and 50% text weights, Accuracy and F1 Score are highest and Log Loss is lowest in the reported ablation.The comparison varies modality weights and evaluates F1 Score, Accuracy, and Log Loss.Source: Figure 2 · p. 4, C. Multimodal Feature Fusion · p. 4
  • The selected equal weighting yields one 512-dimensional embedding per building in a shared semantic space.The resulting representation combines visual and textual characteristics.Source: C. Multimodal Feature Fusion · p. 4

D. Dimensionality Reduction and Visualization

UMAP reduces the fused building features to two dimensions for visualization, where proximity represents similarity in combined visual and textual attributes. The projection is used to explore stylistic patterns and relationships in the dataset.

D. Dimensionality Reduction and Visualization

E. Clustering and Classification

The authors use K-Means to group fused building features, choose eight clusters using elbow and silhouette analyses, then manually label clusters and train an SVM classifier to predict styles.

E. Clustering and Classification

  • K-Means assigns each building to the nearest centroid, and elbow and silhouette analyses are used to select the cluster count.The elbow method considers diminishing reductions in squared error; silhouette analysis assesses cluster fit and separation.Source: E. Clustering and Classification · p. 4
  • k=8 yields the highest tested average Silhouette Score, while the elbow plot places the apparent optimum between k=6 and k=8.The analysis selects eight clusters overall.Source: Figure 4 · p. 5, E. Clustering and Classification · p. 5, Figure 3 · p. 5
  • Experts manually inspect recurring, filtered keywords within clusters to identify architectural elements, periods, or regional influences and assign style labels.Source: E. Clustering and Classification · p. 5
  • An SVM is trained on the combined features and manual cluster labels, with a train-test split and accuracy, loss, precision, recall, and F1-score used for evaluation.Source: E. Clustering and Classification · p. 5

IV. RESULTS AND DISCUSSION

The authors report that the UMAP, K-Means, and SVM stages collectively demonstrate the effectiveness of integrating visual and textual features for architectural analysis.

IV. RESULTS AND DISCUSSION

  • The UMAP projection, K-Means clustering, and SVM classification collectively demonstrate the effectiveness of integrating visual and textual features for architectural analysis.Source: §IV · p. 5

A. UMAP Projection and Visual Analysis:

The UMAP projection shows groupings and a continuum of architectural styles among UAE residential buildings, with outliers often corresponding to less common designs.

A. UMAP Projection and Visual Analysis:

B. K-Means Clustering and Manual Labeling

K-Means groups buildings into eight clusters, which are manually labeled using recurring keywords to create a dataset of architectural styles. The labels are mapped onto the UMAP projection.

B. K-Means Clustering and Manual Labeling

C. SVM Classification Performance

The SVM classifier performed well overall and across most architectural-style classes. Evaluation used a held-out test set, with per-cluster metrics and misclassifications summarized in the reported visuals.

C. SVM Classification Performance

V. CONCLUSION AND FUTURE WORK

The conclusion presents multimodal CLIP embeddings, followed by UMAP, K-Means clustering, manual labeling, and SVM training as a feasible approach to classifying UAE residential architecture. It also identifies dataset breadth and annotation and fusion choices as areas for future work.

V. CONCLUSION AND FUTURE WORK

  • The paper reports meaningful clusters of buildings sharing visual and textual attributes and demonstrates the feasibility of style classification using K-Means, manual labels, and an SVM.UMAP projects the high-dimensional data into an intuitive two-dimensional space.Source: §V · p. 7
  • A 50% visual and 50% textual feature fusion provided a richer representation than either modality alone.Visual features distinguish observable physical attributes, while textual keywords add semantic context for stylistic nuances.Source: §V · p. 7
  • The unified representation and automated categorization offer possibilities for architectural documentation, urban planning, and heritage preservation.The paper presents the work as a foundational step toward systems for understanding and preserving regional architectural identities.Source: §V · p. 7
  • The 634-pair dataset limits the study’s breadth; the authors identify wider UAE regional and historical coverage as a way to enhance generalizability and robustness.Future directions also include reducing reliance on extensive manual annotation and exploring more sophisticated fusion approaches.Source: §V · p. 7
Loading 2609.17181v1…