Source-linked AI summary
Gradient-Guided Decoupled Adaptation for Geospatial Vision-Language Models
Dongdong Wang, Deepak Balakrishnan, Ravi Srinivasan, Shenhao Wang
TL;DR
Geo-VLMs commonly optimize diverse tasks in a shared parameter space despite heterogeneous gradient characteristics and cross-modal demands. G2DA decouples task groups, schedules modality-specific curricula, and uses bidirectional rehearsal; it outperforms representative baselines across all 24 benchmark-model settings.
Problem
Geo-VLMs commonly optimize heterogeneous geospatial tasks in a shared parameter space, although their differing vision-language demands can produce conflicting gradients.
Source: Introduction · p. 1Method
G2DA uses gradient-guided cross-modal decoupling, modality-specific curricula ordered by task-gradient similarity, and bidirectional rehearsal.
Source: Introduction · p. 1, Optimization · p. 4Results
Across all 24 benchmark-model settings, G2DA outperforms representative baselines, improving by 3.08 pp on UrBench-MCQ, 4.30 pp on XLRS-Bench-Lite, and 2.81 pp on VRS-Bench-VQA.
Source: Results · p. 5, Conclusion · p. 7Takeaways & Limitations
Organizing optimization according to task-level gradient characteristics effectively alleviates interference and provides an effective paradigm for multi-task Geo-VLM learning.
Source: Conclusion · p. 7
Abstract
from arXiv · showhide
Existing geospatial vision-language models (Geo-VLMs) typically optimize diverse geospatial tasks through a unified multi-task adaptation paradigm without explicitly accounting for the heterogeneous optimization characteristics. Our empirical observations reveal heterogeneous gradient characteristics across tasks, including vision-language differences, intra-branch gradient relationships, and task interference, which hinder effective multi-task optimization. Motivated by these observations, we propose Gradient-Guided Decoupled Adaptation (G2DA), a gradient-aware optimization framework for multi-task Geo-VLM learning. G2DA first partitions tasks into vision- and language-centric groups through gradient-guided cross-modal decoupling. It then constructs modality-specific curricula based on task gradient similarity and employs bidirectional rehearsal to mitigate the recency effects introduced by sequential optimization. We evaluate G2DA on three Geo-VLM benchmarks using six InternVL3 and Qwen3.5-VL variants, along with GeoChat and GeoLLaVA. Across all 24 benchmark-model combinations, G2DA consistently outperforms representative baselines, improving over the strongest competitor by 3.08, 4.30, and 2.81 percentage points on UrBench-MCQ, XLRS-Bench-Lite, and VRS-Bench-VQA, respectively. These results demonstrate the effectiveness of gradient-guided task organization for Geo-VLM adaptation.
Introduction
Geo-VLMs jointly optimize heterogeneous tasks in a shared parameter space, where conflicting gradients and differing vision-language demands challenge adaptation. G2DA organizes optimization around these gradient characteristics and consistently outperforms representative baselines.
Introduction
- Geo-VLM tasks have heterogeneous optimization demands across vision and language, and shared-parameter optimization can introduce conflicting gradients.Existing conflict-mitigation methods still use a unified parameter space, which may not exploit cross-modal optimization preferences.Source: Introduction · p. 1
- G2DA organizes multi-task optimization through gradient-guided decoupling, modality-specific curriculum learning, and bidirectional rehearsal.The framework is designed for heterogeneous cross-modal gradient characteristics.Source: Introduction · p. 1
- Gradient analysis identifies cross-task conflicts, differing vision-language optimization preferences, and persistent intra-component task interference.These observations characterize the optimization heterogeneity motivating the framework.Source: Introduction · p. 1
- G2DA consistently outperforms representative baselines in adapting Geo-VLMs.The introduction presents gradient-guided optimization as an effective paradigm for Geo-VLM adaptation.Source: Introduction · p. 2
Related Work
Prior work develops Geo-VLMs and multi-task optimization methods, but leaves cross-modal optimization preferences and sequential curriculum imbalance insufficiently addressed.
Geospatial Vision-Language Models
- Geo-VLMs adapt general-purpose VLMs for geospatial domains, while unified multi-task learning optimizes heterogeneous tasks under a shared objective.This approach can introduce gradient interference as task diversity increases.Source: Geospatial Vision-Language Models · p. 2
Gradient Conflict in Multi-task Learning
- PCGrad, CAGrad, GradNorm, FAMO, and STGU address task interference or scheduling, but retain a unified parameter space.These methods are not designed to exploit vision- and language-specific optimization preferences in Geo-VLMs.Source: Gradient Conflict in Multi-task Learning · p. 2
Curriculum and Rehearsal Learning
- Task-level curricula structure training sequences, but one-pass scheduling may cause imbalance or forgetting as earlier tasks are not revisited.Rehearsal revisits prior tasks, yet its use for sequential multi-task Geo-VLM curricula remains unexplored.Source: Curriculum and Rehearsal Learning · p. 2
Method Problem Formulation
G2DA frames Geo-VLM adaptation as a fixed-budget task-scheduling problem and uses gradient analysis to organize heterogeneous tasks. Its three stages decouple tasks by modality preference, schedule compatible tasks within each modality, and use bidirectional rehearsal to preserve knowledge.
Method Problem Formulation
- Under a fixed update budget, the final optimization outcome depends on how task data are scheduled, so the goal is to improve the joint objective with the same budget.The formulation holds initialization and update count constant while varying the training schedule.Source: Problem Formulation · p. 2
Gradient Characterization
- Task-gradient cosine similarity proxies pairwise optimization alignment, while vision-gradient proportion characterizes each task’s relative preference for vision or language encoders.Together, the measures describe inter-task compatibility and cross-modal optimization preference.Source: Gradient Characterization · p. 3
- Gradient analysis identifies conflicts across tasks, differing vision–language demands, and persistent gradient heterogeneity among tasks assigned to the same encoder.Negative cosine similarity means a shared update that lowers one task’s loss may increase another’s; encoder separation alone therefore leaves interference.Source: Gradient Characterization · p. 3
Optimization
- G2DA first partitions tasks into vision-centric and language-centric groups using each task’s vision-gradient proportion.The framework then organizes task compatibility within each modality and applies bidirectional rehearsal.Source: Optimization · p. 3
- Cross-modal decoupling increases local gradient cosine similarity, but substantial conflicts remain within each branch.Those remaining conflicts motivate modality-specific curriculum scheduling.Source: Optimization · p. 4
- Within each modality group, tasks with higher average compatibility are scheduled earlier, progressing from broadly aligned to relatively conflicting tasks.Sequential scheduling introduces recency effects and catastrophic forgetting, motivating rehearsal.Source: Optimization · p. 4
- Bidirectional rehearsal traverses each curriculum in ascending and reverse order, adding uniformly sampled examples from complementary stages equal in number to the current stage.The method is intended to preserve knowledge across the curriculum, mitigate optimization drift, and improve balanced performance across heterogeneous tasks.Source: Optimization · p. 4
Experiment
G2DA is evaluated across multiple Geo-VLM benchmarks, model families, and multi-task optimization baselines. It leads across all reported benchmark-model combinations, while analyses examine transfer, task behavior, gradient interactions, and component contributions.
Experimental Setup
- Experiments cover UrBench-MCQ, XLRS-Bench-Lite, and VRS-Bench-VQA, using InternVL3 and Qwen3.5-VL variants plus GeoChat and GeoLLaVA against multi-task optimization baselines.Accuracy is averaged across tasks; all VQA tasks are multiple-choice, and methods use identical optimization budgets.Source: Experimental Setup · pp. 4–5
Analyses
- Representative multi-task optimization methods do not consistently outperform joint training, while G2DA improves across general-purpose and remote-sensing-specific models, with especially pronounced gains on InternVL3 and GeoLLaVA.G2DA’s gains are generally smaller but consistent across Qwen3.5-VL scales; benchmark gains vary, with the largest on XLRS-Bench-Lite.Source: Analyses · pp. 5–6
Results
- G2DA achieves the best results in all 24 benchmark-model combinations, improving over the strongest baseline by 3.08 pp on UrBench-MCQ, 4.30 pp on XLRS-Bench-Lite, and 2.81 pp on VRS-Bench-VQA.Table 1 reports performance across three independent runs.Source: Results · p. 5, Table 1 · p. 6
Analyses
- On UrBench-MCQ, G2DA improves nearly all tasks with balanced gains across vision- and language-oriented tasks, and its largest gains occur on reasoning-intensive tasks with strong gradient conflicts.Qwen3.5-VL-27B has marginal drops on City Retrieval and Image Retrieval; Qwen3.5-VL-27B and GeoLLaVA-8B show gains in both task groups.Source: Analyses · p. 6, Table 2 · p. 7
- After training, G2DA increases average pairwise gradient cosine similarity and reduces conflicting task pairs across benchmarks compared with joint training.Conflicts fall by up to 6.0 percentage points, while gradient alignment increases by 5.1–13.1%.Source: Analyses · p. 7, Table 3 · p. 7
Ablation Study
- Gradient-guided decoupling provides the largest individual improvement; curriculum scheduling further helps on XLRS, while rehearsal recovers and improves performance after curriculum-related UrBench drops.The reported UrBench drops versus decoupling alone are 56.0→54.9 and 53.3→52.1.Source: Ablation Study · p. 7, Table 4 · p. 7
Conclusion
G2DA addresses heterogeneous optimization in multi-task Geo-VLM learning by combining gradient-guided cross-modal decoupling, branch-specific curriculum scheduling, and bidirectional rehearsal. Across three benchmarks and 24 benchmark–model settings, it consistently outperforms representative multi-task optimization baselines under identical optimization budgets.
Conclusion
- G2DA integrates gradient-guided cross-modal decoupling, branch-specific curriculum scheduling, and bidirectional rehearsal into a unified optimization framework.Source: Conclusion · p. 7
- The empirical analysis identifies inter-task gradient interference, distinct cross-modal optimization preferences, and persistent intra-encoder gradient heterogeneity in multitask Geo-VLM learning.Source: Conclusion · p. 7
- G2DA consistently outperforms representative multi-task optimization baselines across all 24 benchmark–model settings under identical optimization budgets.The experiments cover three Geo-VLM benchmarks, six general-purpose VLMs, and two representative Geo-VLMs.Source: Conclusion · p. 7
- The authors conclude that organizing optimization by task-level gradient characteristics effectively alleviates optimization interference and provides an optimization paradigm for multi-task Geo-VLM learning.Source: Conclusion · p. 7