Source-linked AI summary
Beyond Memorization: A Multi-Modal Ordinal Regression Benchmark to Expose Popularity Bias in Vision-Language Models
Li-Zhong Szu-Tu, Ting-Lin Wu, Chia-Jui Chang, He Syu, Yu-Lun Liu
TL;DR
Building-age estimation lacks a global, large-scale open benchmark, and it remains unclear whether VLMs learn architecture or memorize landmarks. YearGuessr frames the task as ordinal regression and pairs its dataset with YearCLIP and popularity-aware evaluation; the benchmark reports a pronounced popularity gap in VLM accuracy.
Problem
Building-age estimation lacks a global, large-scale open benchmark, and whether VLMs learn architecture or memorize landmarks remains an open question.
Source: §1 · pp. 1–2Method
The authors introduce YearGuessr, a 55,546-image, 157-country benchmark spanning 1001–2024 CE, and evaluate ordinal prediction with popularity-aware metrics alongside YearCLIP.
Source: §1 · p. 2Results
Gemini-2.0 gains +34.18% accuracy on high-popularity buildings, while other general models degrade.
Source: §1 · p. 2Takeaways & Limitations
The benchmark study finds that VLM performance varies with building fame rather than architectural features, indicating landmark memorization.
Source: §1 · p. 2The dataset is geographically and temporally skewed toward modern examples, and original construction-year labels can be noisy for renovated or rebuilt buildings.
Source: §6 · p. 8
Abstract
from arXiv · showhide
We expose a significant popularity bias in state-of-the-art vision-language models (VLMs), which achieve up to 34% higher accuracy on famous buildings compared to ordinary ones, indicating a reliance on memorization over generalizable understanding. To systematically investigate this, we introduce the largest open benchmark for this task: the YearGuessr dataset, a collection of 55,546 building images with multi-modal attributes from 157 countries, annotated with continuous ordinal labels of their construction year (1001-2024), GPS data, and page-view counts as a proxy for popularity. Using this dataset, we frame the construction year prediction task as ordinal regression and introduce popularity-aware interval accuracy metrics to quantify this bias. Our resulting benchmark of 30+ models, including our YearCLIP model, confirms that VLMs excel on popular, memorized items but struggle significantly with unrecognized subjects, exposing a critical flaw in their reasoning capabilities. Project page: https://sytwu.github.io/BeyondMemo/
1. Introduction
Building-age estimation matters for sustainability audits, heritage preservation, and post-disaster assessment, but lacks a global, large-scale open benchmark. The paper introduces YearGuessr and evaluates whether VLMs learn architecture or rely on landmark popularity.
1. Introduction
- Gemini-2.0 gains +34.18% accuracy on high-popularity buildings, while other general models degrade.Source: §1 · p. 2
- YearGuessr is presented as the first open building-age estimation benchmark, with 55,546 images across 157 countries spanning 1001–2024 CE; the task is framed as ordinal regression.The dataset is described as having global coverage and large-scale ordinal labels.Source: §1 · p. 2
- The benchmark frames construction-year prediction as ordinal regression and introduces Interval Accuracy and a popularity-based MAE metric.Source: §1 · p. 2
- YearCLIP combines CLIP-based ordinal training, GPS priors, and architectural reasoning prompts to provide human-verifiable prediction rationales.Its ordinal strategy is coarse-to-fine, and its prompts cover cues such as roof and wall features.Source: §1 · p. 2
2. Related Work
Related work spans building-age datasets, geolocation, geo-aware vision-language models, ordinal regression, and multimodal learning signals. The paper positions YearGuessr as extending these lines toward fine-grained building-age dating with geographic and ordinal cues.
2. Related Work
- Building Age Estimation [14] covers Western Europe with coarse epochs, while CMAB [79] spans 1985–2018; YEARGUESSR covers six continents and 1001–2024 CE.Source: §2 · p. 2
- Planet-scale localization advanced from Im2GPS and PlaNet through hierarchical cells, scene cues, segmentation, transformers, cross-view supervision, and human traces, but not fine-grained age dating.Source: §2 · p. 2
- Geo-aware VLMs incorporate coordinates, climate, map tiles, or addresses, while the evaluation also reveals popularity bias.Source: §2 · p. 3
- Ordinal regression methods range from CORAL/CORN losses [9] and probabilistic approaches [30] [55] to numeric cues in vision-language models such as OrdinalCLIP [31] and NumCLIP [16].Source: §2 · p. 3
- Weather, land-cover, and caption cues aid localization [22] [48]; the paper combines coordinate encodings with ordinal losses for YearGuessr.Source: §2 · p. 3
3. Dataset and Benchmark
YearGuessr is assembled from Wikimedia building pages into a globally distributed dataset with location and popularity metadata. The paper analyzes its coverage and formulates building-year prediction as ordinal regression with interval-based evaluation.
3.1. Dataset Construction
- The crawler collected 90,230 Wikimedia pages; deduplication and facade filtering left 55,546 unique building images.The cleaning removed 8,346 duplicates and 26,338 low-similarity images.Source: §3.1 · p. 3
- Buildings were stratified by decade and continent into 60/20/20 partitions; the corpus is CC BY-SA 4.0 and accompanying code MIT-licensed.The split contains 33,337 training, 11,122 validation, and 11,087 test samples.Source: §3.1 · p. 3
- All samples have GPS and textual descriptions, with 157-country coverage and a construction-year distribution spanning about 1,000 years.Descriptions have a median length of 2,240 characters.Source: §3.1 · p. 4
3.2. Statistics Analysis
- Coverage is skewed toward the Americas (63.3%) and Europe (22.5%), while the dataset includes many post-18th-century buildings and a significant pre-1600 group.The geographic imbalance motivates continent-specific evaluation.Source: §3.2 · p. 4
- Popularity is heavy-tailed by annual pageviews; 52.99% of buildings show no renovation, and locations are grouped into rural, semi-urban, and urban classes.Another 32.62% lack historical renovation records.Source: §3.2 · p. 4
3.3. Tasks & Metrics
- The task predicts a construction year from a facade image and optional GPS using ordinal regression, evaluated with MAE and interval accuracy at 5, 20, 50, and 100 years.Evaluation uses the fixed 11,087-image test split and reports means and standard deviations across three seeds.Source: §3.3 · pp. 4–5
4. Our YearCLIP Model
YearCLIP combines CLIP features, optional geographic context, and architectural prompts in a coarse-to-fine ordinal-regression model. Its outputs include a predicted year and interpretable cues, while the full evaluation and training settings are provided in supplementary material.
4. Our YearCLIP Model
- YearCLIP adapts CLIP with NumCLIP’s coarse-to-fine strategy, first classifying architectural style and then using similarity scores for fine-grained year regression.The approach is intended to balance style recognition with temporal precision.Source: §4 · p. 5
- A pretrained GeoCLIP location encoder fuses geographic and image embeddings, while image-only inference remains available when location data are absent.A zero-convolution layer learns the fusion weights during training.Source: §4 · p. 5
- Predefined prompts about architectural features enrich the regressor, whose input importance can be backtracked without external captioning models.The model can use cues such as roof and wall attributes to provide insight into its prediction.Source: §4 · pp. 5–6
- Table 2 presents simplified performance, interval-accuracy, and popularity analyses; the complete 43-method evaluation and training settings are supplementary.The supplied table caption identifies the results as a simplified presentation.Source: Table 2 · p. 6, §4 · p. 6
5. Results and Analysis
The benchmark comparisons show that model performance varies with popularity, geography, period, density, and renovation status. YearCLIP improves on vision baselines, while VLMs perform especially well on popular buildings and results vary across building contexts.
5.1. Main results
- Fine-tuned CLIP variants improve MAE and interval accuracy over CNN and Transformer baselines; zero-shot CLIP has MAE 78.23.Zero-shot CLIP also outperforms LLaVA-v16-13B (194.07) and MiniCPM-V2-6B (106.41) on MAE.Source: §5.1 · p. 6
- Gemini1.5-Pro leads with MAE 33.08, followed by Gemini2.0-Flash (33.91) and Grok2 (35.28); YearCLIP reaches MAE 39.52 and lowers GeoCLIP’s MAE by 13.5%.The GeoCLIP comparison is 45.69 → 39.52; YearCLIP also outperforms ConvNeXt-B (44.42) and Swin-B (47.65).Source: §5.1 · p. 6
5.2. Popularity analysis
- VLMs score higher on popular buildings: Gemini2.0-Flash rises from 24.23% to 58.41% IA5, while CNNs, Transformers, and YearCLIP often score lower.ConvNeXt-B falls from 16.57% to 12.68%, Swin-B from 15.82% to 6.78%, and YearCLIP from 20.19% to 12.39%.Source: §5.2 · pp. 6–7
5.3. Regional and Period Analysis
- MAE varies by region and period: Gemini2.0-Flash records 23.53 in the Americas versus 62.73 in Africa, and early-period errors can exceed 300.YearCLIP has its lowest regional MAE in the Americas (26.10); Gemini1.5-Pro’s MAE is 386.86 for 1000–1150 and 16.88 for 1900–1950.Source: §5.3 · p. 7, Table 3 · p. 7
5.4. Effect of Density and Renovation
- Semi-urban areas have the lowest MAE for most models, while never-renovated buildings have lower MAE than renovated or rebuilt cases.YearCLIP’s semi-urban MAE is 36.22; Gemini1.5-Pro’s MAE is 20.66 for never-renovated buildings and 57.36 for rebuilt cases.Source: §5.4 · pp. 7–8, Table 4 · p. 8
5.5. Prediction distribution
- Prediction plots show CNN and Transformer deviations especially before 1600, while VLM predictions cluster near the ideal diagonal but favor post-1800 buildings.YearCLIP’s predictions are closer to the diagonal across periods; the reported post-1800 advantage for VLMs is attributed to temporal imbalance in training data.Source: §5.5 · p. 8, Figure 5 · p. 8
5.6. Explainability
- YearCLIP’s explainable predictions use architectural cues such as roof types, materials, and structural elements to provide human-verifiable rationales.The illustrated Reason-enhanced NumCLIP prediction is within ±15 years of ground truth.Source: §5.6 · p. 8, Figure 6 · p. 8
6. Conclusion
The paper presents YearGuessr and an ordinal benchmark, using a 30+ model study to reveal landmark-memorization bias. It reports that NumCLIP-Loc halves MAE over decade classification, while identifying dataset skew and label noise as limitations.
Conclusion
- YearGuessr provides a large-scale, CC BY-SA 4.0 building-age dataset and benchmark, and the study reports landmark-memorization bias across 30+ models.The paper also reports that NumCLIP-Loc halves MAE over decade classification.Source: §6 · p. 8
- The resource may aid heritage preservation, retrofit planning, and disaster inspection, but could also help locate vulnerable sites or reinforce regional bias.Safeguards include removing non-free images, providing a responsible-use data card and public bias metrics, and documenting prompts for closed-source VLMs without redistributing weights.Source: §6 · p. 8
- Geographic and temporal skew toward modern examples limits generalization to underrepresented regions and early styles, while original-year labels add noise for substantially renovated or rebuilt buildings.Planned steps include expanding non-Western and early-period coverage and adding renovation and temporal-segmentation labels.Source: §6 · p. 8
7. Implementation details
The implementation combines frozen visual, location, and text encoders with a trainable ordinal regressor. The experiments specify the training setup and use predefined architectural reasoning prompts that can be customized.
Implementation details
- Experiments use a 6:2:2 split, ViT-B/16 encoders, batch size 64, and 50 epochs at 16-bit precision.The train, validation, and test sets contain 33,337, 11,122, and 11,087 samples; the experiments use an RTX 4090 GPU.Source: §7 · p. 13
- Gemini2.0-assisted prompts categorize architectural cues, including roof types such as spire, dome, flat, sloped, gabled, mansard, and butterfly.The accompanying table gives example roof types and descriptions.Source: §7 · p. 13, Table 5 · p. 13
- The architecture fuses CLIP facade features with an optional GPS embedding and encodes coarse styles and reasoning prompts before a trainable regressor performs coarse-to-fine ordinal regression.The system selects style and reasoning tokens and produces a readable rationale alongside its year prediction.Source: Figure 7 · p. 13
- Additional judgment criteria can be incorporated to customize the regressor’s inputs for architectural age estimation.The paper states that these enhancements may improve performance and adaptability.Source: §7 · p. 13
8. Training Pipeline
YearCLIP predicts building construction years from images and optional location data, using predefined styles and architectural cues to produce interpretable rationales. Its training pipeline combines visual and location embeddings with ordinal regression and a ranking-based contrastive loss.
8.1. Introduction
- YearCLIP accepts a 224×224 color building image and optional coordinates, with seven period-linked styles and architectural-reason prompts as predefined elements.Its components include image, location, class, and reason encoders plus a regressor.Source: §8.1 · pp. 13–14
8.2. Input Processing
- When coordinates are available, the pipeline adds location and image embeddings; without them, it uses the image embedding alone.The location encoder applies an MLP and zero-convolution before fusion.Source: §8.2 · p. 14
8.3. Pre-Defined Elements
- Seven architectural styles map to historical periods, while reason prompts encode subcategories such as roof types and materials.Each style and reason subcategory receives an embedding.Source: §8.3 · p. 14
8.4. Coarse Stage
- Cosine similarities between the input embedding and style or reason embeddings produce scores for the coarse stage.The resulting scores cover each style and reason subcategory.Source: §8.4 · p. 14
8.5. Fine Regression
- The regressor converts style and reason similarity scores into period probabilities, then predicts a weighted average of period midpoints with a learnable stability term.The input vector contains all style and reason-subcategory similarities.Source: §8.5 · p. 14
8.6. Reasoning Importance
- YearCLIP ranks reason subcategories by similarity and regressor attention, then selects the five highest-scoring reasons to explain its prediction.It also sums subcategory importance scores to estimate each reason's overall importance.Source: §8.6 · p. 15
8.7. Loss Function
- Fine-grained cross-modal ranking contrastive loss encourages ordinal consistency, with negative-sample weights determined by the distance between ground-truth labels.The loss compares image and text embedding similarities.Source: §8.7 · p. 15
9. Full Table Results
The basic-method comparison reports lower MAE for CLIP-based methods than the cited CNN and Transformer examples, with NumCLIP lowest among the listed models. The full results tables extend comparisons across model capacities, interval metrics, popularity bands, and continents.
9.1. Basic Method on Building Year Estimation
- 40.01 MAE for NumCLIP is the lowest reported value, ahead of GeoCLIP at 44.32 MAE among the listed models.The comparison also reports ResNet50 at 54.59 MAE, ResNet152 at 47.70, ConvNeXt-L at 42.34, ViT-B/16 at 48.86, and Swin-B at 47.71.Source: Figure 8 · p. 15, §9.1 · p. 15
9.2. Full Results of CLIP-based Model and VLMs
- Tables 6–8 extend the results across basic metrics, interval accuracy by year range, popularity interval, and continent.The tables cover models with varying capacities and architectures.Source: §9.2 · p. 15
10. LLM Usage Statement
The authors used LLMs for auxiliary prompt generation, dataset annotation assistance, writing, and coding. They state that research ideation, experimental design, core algorithmic contributions, analysis, and scientific insights remained the human authors’ work.
LLM Usage Statement
- Gemini 2.0 helped generate predefined architectural reasoning prompts, including categories and descriptions for roofs, materials, heights, and structural features.The prompts serve as inputs to the reasoning system.Source: §10 · pp. 15–16
- An LLM analyzed Wikipedia building descriptions to help distinguish renovations, which retain the original construction year, from rebuilds, which redefine it.This assistance was used for renovation-status annotation.Source: §10 · p. 16
- LLMs assisted with manuscript editing and coding, while the authors say core algorithmic contributions, experimental design, analysis, and scientific insights were developed by humans.The authors characterize LLM use as auxiliary and take responsibility for reviewed and validated content.Source: §10 · p. 16