Source-linked AI summary
Anthropogenic Regional Adaptation in Multimodal Vision-Language Model
Samuel Cahyawijaya, Peerat Limkonchotiwat, Tack Hwa Wong, Hitesh Laxmichand Patel, Amit Agarwal, Manuel Antonio Rufino, Carlos Rafael Catalan, Muhammad Reza Qorib, Vicky Feliren, Holy Lovenia, Aye Hninn Khine, Frederikus Hudi, David Anugraha, Alham Fikri Aji, Romrawin Chumpu, Viet-Thanh Pham, Minghan Wang, Mohamed Fazli Imam, Ruochen Zhang, Joseph Marvin Imperial, Khumaisa Nur'aini, Do Xuan Long, Musa Izzanardi Wijanarko, Joel Ruben Antony Moniz, Patrick Amadeus Irawan, Hanif Muhammad Zhafran, Isaiah Flores, Salsabila Zahirah Pranida, Jun Kevin, Jostin Jerico Rosal, Patricia Nicole Monderin, Kun Kerdthaisong, Ahmad Mustafid, My Chiffon Nguyen, Natchapon Jongwiriyanurak, Siva Worajitwannakul, Haochen Li, Adrian Xuan Wei Lim, Bin Wang, Muhammad Ravi Shulthan Habibi, Lynnette Hui Xian Ng, Mithil Bangera, Yeshil Bangera, Priyaranjan Pattnayak, Dun Li Chan, Sherissa Caren Djuniwar, Cho Chan Myei Oo, Hee Ming Shan
TL;DR
Vision-language systems lack a dedicated framework for human-centric alignment across regional contexts, especially where cultural representation is underrepresented. The paper introduces Anthropogenic Regional Adaptation and GG-EZ, combining regional data filtering with supervised fine-tuning and model merging. Across three architectures in a Southeast Asia case study, GG-EZ improves cultural relevance while largely retaining global performance.
Problem
Existing vision-language systems can show cultural insensitivity, stereotypical outputs, and reduced task performance in underrepresented regions, while no dedicated framework assesses human-centric regional alignment.
Method
GG-EZ filters regional data, fine-tunes a global model into a regional-specific model, and merges models to preserve global generalization.
Results
5-15% gains in cultural relevance metrics were achieved across SEA while maintaining over 98% of global performance across three vision-language architectures.
Takeaways & Limitations
Anthropogenic Regional Adaptation is presented as a foundational paradigm, with GG-EZ positioned as a practical baseline for regionally aligned vision-language systems.
Takeaways & Limitations
The study does not measure GRP for image generation because comparable test sets with comparable value scales are unavailable, and it anchors GRP on the KOF-GI de facto interpersonal component.
Abstract
from arXiv · showhide
While the field of vision-language (VL) has achieved remarkable success in integrating visual and textual information across multiple languages and domains, there is still no dedicated framework for assessing human-centric alignment in vision-language systems. We offer two contributions to address this gap. First, we introduce Anthropogenic Regional Adaptation: a novel paradigm that aims to optimize model relevance to specific regional contexts while ensuring the retention of global generalization capabilities. Second, we present a simple, but effective adaptation method named Geographical-generalization-made-easy (GG-EZ), which utilizes regional data filtering and model merging. Through comprehensive experiments on 3 VL architectures: large vision-language models, text-to-image diffusion models, and vision-language embedding models, and a case study in Southeast Asia (SEA) regional adaptation, we demonstrate the importance of Anthropogenic Regional Adaptation and the effectiveness of GG-EZ, showing 5-15% gains in cultural relevance metrics across SEA while maintaining over 98% of global performance and even occasionally surpassing it. Our findings establish Anthropogenic Regional Alignment as a foundational paradigm towards applicability of multimodal vision-language models in diverse regions and demonstrate a simple-yet-effective baseline method that optimizes regional value alignment while preserving global generalization.
1 Introduction
The paper identifies a gap in human-centric regional alignment for vision-language models, which can be culturally insensitive and underperform in underrepresented regions. It proposes Anthropogenic Regional Adaptation and GG-EZ to improve regional relevance while preserving global generalization.
- Underrepresented regions face cultural insensitivity, stereotypical outputs, and reduced task performance from existing vision-language systems.
- Anthropogenic Regional Adaptation evaluates human-centric alignment across regional contexts while preserving global generalization capabilities.
- GG-EZ uses regional data filtering and model merging to adapt global models to regional contexts with minimal global degradation.
- 5-15% gains in cultural relevance metrics were achieved across SEA while retaining over 98% of global benchmark performance.
- Experiments span large vision-language, text-to-image diffusion, and vision-language embedding architectures in a Southeast Asia case study.
2 Related Work
Prior regional adaptation work includes region-specific datasets and language models, but computer-vision research has focused mainly on remote sensing, limiting understanding of cultural and anthropomorphic characteristics.
- Regional-specific models can outperform one-size-fits-all models on regional context, motivating region-focused adaptation.
- Southeast Asia has dedicated datasets and language models, while computer-vision regional adaptation has concentrated more on remote sensing.
- This focus leaves limited understanding of regions’ unique cultural and anthropomorphic characteristics in computer vision.
3 Anthropogenic Regional Adaptation
The framework partitions global space into regions and distinguishes a target region from others to analyze adaptation under competing regional and global objectives. It uses a globalization factor to balance these objectives and motivates model archetype comparison through this trade-off.
- Regional partitioning: The framework represents the global context as Rglobal and partitions it into k distinct measurable regions R1, R2, . . . , Rk.
- Regional partitioning: The target region Rregional is a non-empty subset of Rglobal, while Rothers is defined by set difference to provide complete domain coverage.
- Model archetypes: Global models capture broad patterns but sacrifice regional specificity, whereas regional-specific models improve local performance but reduce global coherence.
- Global-regional optimization: The evaluation framework aggregates regional quality metrics into global and regional sets and introduces α ∈ [0, 1] to control their trade-off.
- GG-EZ framework: GG-EZ is organized into regional quality filtering and global-regional refinement, making the adaptation framework applicable across vision-language architectures.
- Global-regional optimization: The weighted objective can set α closer to 1 for globally connected regions or closer to 0 when local impacts dominate.
4 Geographical Generalization Made Easy (GG-EZ)
GG-EZ adapts global multimodal models through regional quality filtering followed by global-regional refinement, combining supervised fine-tuning with linear model merging. The framework selects regionally relevant data and balances regional performance against preservation of global capabilities.
- GG-EZ is an architecture-agnostic framework with regional quality filtering and global-regional refinement phases.It targets local textual and imagery contexts while preserving base-model capabilities across multimodal architectures.
- Regional Quality Filtering: The regional filter selects examples associated with the target region and above a reward-model quality threshold.The filtered set requires both regional membership and reward score greater than τ.
- Regional Quality Filtering: Filtered regional data can be augmented by translating high-quality English datasets into multiple target regional languages.The translation function maps English examples to languages in the regional language set.
- Global-Regional Refinement: Global-regional refinement first fine-tunes a global model on filtered and translated data, then merges the regional model with the original global model.This sequence is designed to improve regional specialization while preserving global generalization.
- Global-Regional Refinement: β controls the interpolation between regional and global models, with its optimal value selected using a weighted regional-global evaluation objective.The objective weights regional quality by α and global quality by 1 − α.
- Global-Regional Refinement: The resulting merged model is intended to achieve strong regional adaptation with minimal degradation of base-model capabilities.
5 Case Study on Southeast Asian (SEA) Adaptation
The SEA case study evaluates GG-EZ across large vision-language, diffusion, and embedding models using regional datasets, multilingual resources, and global-regional merging. Experiments assess regional cultural performance alongside global quality and generalization.
- Case Study Setup: GG-EZ is evaluated across three multimodal architectures in an SEA case study spanning eleven countries.The architectures include a 27B VLM, a 1B vision-language embedding model, and a 3.5B contextualized diffusion model.
- Case Study Setup: SEA data curation combines SEA-VL, CulturalGround imagery, and translated MAmmoTH-VL instructions across major SEA languages.The translated languages include Indonesian, Malaysian, Thai, Vietnamese, Filipino, Khmer, Lao, Chinese, and Tamil.
- Model Adaptation: SEA-VLM training uses continuous pre-training and supervised fine-tuning from Gemma-3, while SEA-ImageGen and SEA-VL Embed are fine-tuned from SDXL and SigLIP2-SO400m.The supplied implementation passages specify separate training settings for the three adapted models.
- Global-Regional Merging: Linear merging combines region-specific adaptations with base models using interpolation weights selected to balance regional and global performance.The explored β ranges differ across VL-Embed, SEA-ImageGen, and SEA-VLM.
- Evaluation: Evaluation covers regional visual question answering, culturally nuanced queries, food identification, human judgments, embedding tasks, and diffusion-model quality.The benchmarks include SEAVQA, CVQA, WorldCuisine, SEA-AYA, SEA-VL VQA, and DPGBench.
6 Result and Discussion
Across three multimodal architectures, GG-EZ improves regional adaptation while preserving or enhancing general performance. Results also show that model merging and carefully curated regional data are important, while globalization weighting affects regional–global balance.
- SEA-VLM: GG-EZ model merging improves SEA-Gemma-3 across evaluation scenarios, with the 10% variant outperforming the original Gemma-3 on all evaluation sets.The 10% variant improves averaged global performance by 1% and average SEA-specific performance by 7.5%.
- SEA-VLM: Human evaluations show SEA-Gemma-3 without merging leads regional specialization, while the 10% variant provides a stronger regional–global balance.The 10% variant achieves the highest Vietnamese and Malaysian scores, global performance of 2.42 versus 2.54 for original Gemma-3, and the highest GRP score of 2.31.
- SEA-ImageGen: GG-EZ improves SEA-SDXL image generation while preserving general quality, with merged 25% and 50% models reaching DPGBench scores of 74.75 and 74.61.The SEA-SDXL 25% model also outperforms both original SDXL and fully fine-tuned SEA-SDXL across cultural aspects in correctness and naturalness.
- SEA-VL Embed: SEA-SigLIP2 improves both SEA and non-SEA evaluation performance, while different merge weights favor global transfer or regional specialization.The 50% model reaches the highest global CVQA score of 27.52, whereas the 75% model reaches the highest SEAVQA score of 29.66 and GRP score of 27.96.
- Data Curation: Regional data curation is consequential: using 20% of the baseline dataset causes a 70% degradation in overall model performance relative to the full dataset.The ablation also finds that culture-specific data effects depend strongly on the VQA task format.
- Globalization Factor: The globalization factor α determines how Anthropogenic Regional Adaptation balances regional representation and global perspective, and misalignment can produce suboptimal behavior.The paper derives α from the KOF Globalization Index to reflect regional differences in globalization and their evolution over time.
7 Conclusion
The framework targets cultural insensitivity and reduced regional performance in vision-language systems through Anthropogenic Regional Adaptation and GG-EZ. Across three architectures in a Southeast Asia case study, it improves cultural relevance while preserving generalization.
- 5-15% gains in cultural relevance metrics across SEA contexts were achieved while maintaining or improving generalization capabilities.The evaluation covered Gemma-3 27B, SDXL, and SigLIP-2.
A Assessment for SEA languages Translation Quality
The paper evaluates English-to-Southeast Asian language translation quality using human ratings and selects models by language based on observed performance and robustness.
- Human evaluation used a five-point scale to assess grammatical accuracy and naturalness across five Southeast Asian languages.The evaluation used 100 randomly selected human-generated English captions from SEA-VL.
- Gemma-3-27B achieved competitive or top performance for Filipino, Indonesian, and Vietnamese, often matching or exceeding Gemini models in grammatical accuracy.It also maintained high naturalness scores.
- Gemma-3-27B was selected for Indonesian, Vietnamese, Standard Malay, Filipino, and Chinese because it demonstrated strong and stable performance.
- Gemini-2.5-flash was selected for Thai, Burmese, Lao, Khmer, and Tamil because the Gemini family showed superior robustness and higher translation quality for these languages.
B Reward Model Ablation
The reward-model ablation compares model agreement with human judgments and selects UnifiedReward by balancing predictive performance with computational efficiency for large-scale filtering.
- Reward models were evaluated by agreement with human evaluations using a pairwise preference task over 500 randomly sampled image pairs.Pairwise comparison avoided unreliable direct comparison of absolute scores across datasets.
- VisionReward-Image achieved the highest pairwise agreement with human judgments, but UnifiedReward was selected for its efficiency-performance trade-off.UnifiedReward supports efficient batched inference through vLLM, improving throughput for large-scale data filtering.
- UnifiedReward sacrifices slightly lower accuracy for substantially improved throughput during large-scale data filtering.
C Annotated Human Evaluation Test Sets
The paper constructs culturally grounded Southeast Asian human-evaluation sets covering visual question answering and multilingual vision-language tasks. These benchmarks emphasize image-grounded, culturally relevant, and linguistically natural evaluation.
- C.1 SEA-VL VQA: SEA-VL VQA contains approximately 1.1k visual questions across nine official Southeast Asian languages.Native speakers authored questions and two additional native speakers reviewed them.
- C.1 SEA-VL VQA: Annotators were required to ask image-grounded questions targeting culturally salient elements such as attire, cuisine, religious practices, architecture, and social activities.
- C.1 SEA-VL VQA: The annotation guidelines excluded generic, purely factual, non-image-answerable, and insufficiently Southeast Asian questions.
- C.2 SEA AYA: SEA AYA contains approximately 1.2k instances across nine vision-language tasks after translating 135 Aya Vision questions into six Southeast Asian languages.
- C.2 SEA AYA: SEA AYA translation guidelines prioritized preserving meaning, natural fluency, standard target-language usage, formality, and cultural-pragmatic norms.
- Evaluation results: SEA-Gemma-3 10% achieved the strongest performance on SEA and other regions on CVQA, outperforming several smaller, similarly sized, and larger models.
F Sample Outputs of SEA-VLM and SEA-ImageGen
This section presents sample outputs and evaluation breakdowns for SEA-adapted vision-language and image-generation models, including comparisons with global and regional baselines. The examples emphasize improved regional accuracy while retaining image naturalness and overall quality.
- Sample Outputs: SEA-SDXL 25% produces the most accurate representation among the compared image-generation models while retaining SDXL’s naturalness and overall quality.The example concerns generated nasi lemak images, where the merged model captures more detailed factors than the baselines.
- Evaluation Results: The globalization index is reported by region and year from 1993 to 2023 and is used to decide GG-EZ’s globalization factor α.Regions include EAP, ECA, LAC, MENA, NA, SA, SEA, and SSA.
- Evaluation Results: Tables 8 and 9 compare SEA-Gemma-3 10% with competitor models and SEA-Gemma-3 on CVQA and SEA-VQA.These tables provide benchmark comparisons for the best merged model across two evaluations.
- Evaluation Results: The appendix reports SEA-VLM performance by country, cuisine context, CVQA, DPGBench aspect, and SEAVQA evaluation.The breakdowns cover country-level results, contextual and adversarial World Cuisine settings, DPGBench attributes, entities, relations, and other aspects.
- Sample Outputs: Figure 5 compares global Gemma-3, SEA-Gemma-3, and merged SEA-Gemma-3 10% responses alongside their prompts.The caption states that the regional model produces the most correct image while retaining the original model’s naturalness and overall quality.
- Sample Outputs: Figure 6 compares global SDXL, SEA-SDXL, SEA-SDXL 25%, and reference natural images.The figure is used to compare regional adaptation and model merging against a global baseline and natural-image references.