RemoteAgent: Bridging Vague Human Intents and Earth Observation with RL-based Agentic MLLMs

1. Introduction

• EO users often lack the technical background to provide machine-friendly instructions, so practical systems must ground vague requests in visual tasks spanning image-level to pixel-wise analysis.The required visual precision varies by application scenario.Source: §1 · p. 1
• MLLMs’ text-based architecture is ill-suited to dense spatial outputs, while tool-augmented agents can overuse tools and fail to exploit MLLMs’ holistic image interpretation.Extensive supervised fine-tuning can also erode the open-ended reasoning needed to interpret ambiguous intents.Source: §1 · pp. 1–2
• VagueEO pairs standard EO tasks with simulated vague natural-language queries to train an agent to understand non-expert intents.The queries are designed to reflect non-expert users’ needs.Source: §1 · p. 2
• Reinforcement fine-tuning on VagueEO aligns the MLLM for image- and sparse region-level tasks, while MCP routes dense predictions to specialized tools.This division preserves the MLLM as the cognitive core for suitable tasks and reserves external execution for dense predictions.Source: §1 · p. 2
• RemoteAgent accurately maps free-form intents, performs intrinsic tasks competitively with less data, and outperforms MLLM baselines on extrinsic tasks with spatial precision comparable to specialized models.The evaluation covers intent recognition, native image- and sparse-region tasks, and tool orchestration for dense predictions.Source: §1 · p. 2

2. VagueEO

• The benchmark overview covers ten diverse EO tasks pairing vague, human-centric queries with standardized structural annotations.The benchmark is intended to evaluate capability-aware routing.Source: Figure 2 · p. 3, §2 · p. 3
• An LLM-driven synthesis pipeline generates diverse vague-query templates and pairs them with structural annotations from standard EO benchmarks.The queries use free-form language to require intent deduction rather than keyword matching.Source: §2 · p. 3
• VagueEO pairs everyday ambiguous queries with precise visual ground truths, spanning image-level labels, bounding boxes, and pixel-wise masks.The multi-granularity annotations provide supervision for semantic understanding and spatial reasoning.Source: §2 · p. 3
• The training set contains five intrinsic tasks with exactly 1,000 vague query-annotation pairs per category, used for reinforcement fine-tuning.The tasks are scene classification, multi-label classification, visual grounding, object counting, and geospatial region reasoning.Source: §2 · p. 3
• Testing covers ten tasks, adding five unseen tasks predominantly involving dense predictions, with 100 query-annotation pairs for each task.The supplied examples include object detection, semantic segmentation, referring expression segmentation, and change detection.Source: §2 · p. 3

3.1. Formulation

• The task formulation separates intrinsic semantic or sparse-reasoning tasks from extrinsic dense predictions requiring pixel-level precision.Intrinsic tasks include classification and visual grounding; extrinsic examples include segmentation and object detection.Source: §3.1 · p. 3
• RemoteAgent optimizes its policy with GRPO to distinguish tasks solved internally from those routed to tools, rather than maximizing likelihood with SFT.The optimization uses expected reward and is intended to preserve general reasoning capabilities.Source: §3.1 · p. 3

3.2.1. GRPO-based Optimization

• GRPO rewards functional correctness of structured outputs, while KL regularization constrains policy drift and helps retain the base model’s general capabilities.The authors state that this also preserves zero-shot ability to interpret system prompts and route dense tasks to tools.Source: §3.2.1 · p. 4

3.2.2. Unified Multimodal Reward

• For coordinate-valued outputs, Hungarian matching on bounding-box pairwise IoU supports a reward that accounts for localization quality and coverage.For numerical object-counting answers, the reward uses relative error with hard rejection of large errors; textual classification answers use canonicalized label sets and a coverage-based or recall score.Source: §3.2.2 · p. 5
• A unified reward dispatches scoring by answer format, maps scoring branches to [0, 1], and supports intrinsic tasks without task-specific losses.Dense pixel-level predictions remain the responsibility of external expert tools.Source: §3.2.2 · p. 5

3.3. Tool-Augmented Inference

• RemoteAgent delegates dense EO predictions to specialist tools through MCP while retaining the MLLM for high-level intent grounding and suitable internal tasks.The selected tool returns dense outputs such as detection boxes or segmentation masks.Source: §3.3 · p. 5

4. Experiments

• The evaluation covers intent recognition, intrinsic sparse localization, and extrinsic dense spatial predictions.More experiments are deferred to the supplementary material.Source: §4 · p. 6

4.1. Experimental Setup

• Reinforcement fine-tuning starts from Qwen2.5-VL-7B-Instruct with LoRA across all linear layers and GRPO sampling four generations per query.Training runs for 24 epochs at a constant 1 × 10−6 learning rate, with effective batch size 32 across two H100 GPUs.Source: §4.1 · p. 6
• MCP-compatible expert services and the central MLLM are deployed in a shared local environment with eight NVIDIA 4090 GPUs.Source: §4.1 · p. 6

• 95.0% mean accuracy lets RemoteAgent decisively surpass RemoteReasoner, while GeoChat and Falcon each score below 8%.The authors attribute the SFT models’ near-failure to overfitting rigid prompts and scarcity of long conversational prompts in their fine-tuning data.Source: §4.2 · p. 6
• VagueEO exposes the model to linguistic ambiguity, while RL alignment is described as avoiding the catastrophic forgetting typically induced by standard SFT.The authors characterize RL as lightweight steering that preserves intrinsic reasoning while teaching complex-intent routing.Source: §4.2 · p. 6

4.3.1. Scene Classification

• 91.34 accuracy on AID exceeds Qwen2.5-VL by over 28 points, while RemoteAgent trails specialist FUSE-RSVLM by a narrow margin.The authors interpret this result as evidence that the training strategy preserves native image-level understanding.Source: §4.3.1 · p. 6

4.3.2. Grounding & Reasoning

• 48.3 IoU on DIOR-RSVG surpasses EarthDial and Falcon, while 57.81% Acc@0.5 on region reasoning exceeds Qwen2.5-VL-7B by 16.6%.Source: §4.3.2 · p. 6, Table 3 · p. 7

4.3.3. Object Counting

• RemoteAgent achieves SOTA object-counting performance on DOTAv2 and remains competitive on HRRSD, outperforming GeoChat and RSUniVLM but trailing EarthDial slightly.Source: Table 4 · p. 7, §4.3.3 · p. 7

4.4.1. Object Detection

• RemoteAgent exceeds Falcon by over 21 points on DIOR and trails SkySense by less than one point on both DIOR and DIOR-R.The authors attribute the small deficit to ambiguous queries misrouted during intent recognition; results are compared in Table 5.Source: §4.4.1 · p. 7, Table 5 · p. 7

4.4.2. Semantic Segmentation

• 93.54 mF1 on Potsdam trails only SkySense, while 67.01 mIoU on iSAID is reported as competitive with the tool’s native capabilities.The semantic segmentation comparison is reported in Table 6.Source: §4.4.2 · p. 7, Table 6 · p. 7

4.4.3. Referring Expression Segmentation

• 71.08 mIoU and 83.64 Acc@0.5 on RRSIS-D establish state-of-the-art referring expression segmentation, exceeding RS2-SAM2 by 4.36 mIoU and SegEarth-R2 by 3.18 mIoU.The evaluation is reported in Table 7.Source: §4.4.3 · p. 7, Table 7 · p. 8

4.4.4. Building Damage Assessment

• 77.16 F1overall and 73.03 F1cls on xBD surpass DamFormer and ChangeOS, while F1loc remains behind PCDASNet.The assessment uses bi-temporal pixel-level alignment for building damage; results are reported in Table 8.Source: §4.4.4 · p. 8, Table 8 · p. 8

4.5.1. Ablation on Training Strategy

• RL restores segmentation to 71.64% mIoU versus zero-shot baselines, while SFT lowers it by 18.94% mIoU and RL exceeds SFT by 14.7 grounding points and 28% intent accuracy.The training-strategy comparison is reported in Table 9.Source: §4.5.1 · p. 8, Table 9 · p. 8

4.5.2. Time Efficiency

• 1.18 seconds is RemoteAgent’s reported total execution time, compared with 79–200 seconds for Earth-Agent configurations.The authors report a 100x speedup and attribute it to single-step tool invocation bypassing redundant reasoning cycles; Table 10 compares inference time.Source: §4.5.2 · p. 8, Table 10 · p. 8

4.5.3. Case Studies

• For “oval ground track field” and “airplanes,” the agent routes dense spatial queries to RemoteSAM for referring segmentation and SkySense for object detection.Figure 5 presents these qualitative routing cases.Source: §4.5.3 · p. 8, Figure 5 · p. 9

5.1. Remote Sensing MLLMs

• Remote-sensing MLLMs progressed from instruction-tuned captioning and visual question answering to RL-based geospatial reasoning, yet their text-centric outputs remain poorly suited to dense, precision-critical predictions.The cited RL approaches include Geo-R1 [81], RemoteReasoner [71], and RSThinker [33].Source: §5.1 · p. 9

5.2. Remote Sensing Agentic Systems

• Remote-sensing agentic systems use dynamic toolkits, MCP ecosystems, or hierarchical task abstractions, but are characterized as relying heavily on external tools even for rudimentary queries.Examples include RS-Agent [67], GeoFlow [4], EarthAgent [11], OpenEarthAgent [47], and CangLing-KnowFlow [6].Source: §5.2 · p. 9

6. Limitations & Future Work

• Limited VagueEO coverage constrains query coverage, while static tools and no self-correction or rollback leave model discovery and external-tool error recovery unaddressed.Future work targets scaling instruction data and developing open-ended dynamic tool integration.Source: §6 · p. 9

• RemoteAgent uses reinforcement fine-tuning to resolve intrinsic macroscopic tasks and routes dense predictions to specialized tools via MCP.Source: §7 · p. 10
• VagueEO grounds ambiguous, non-expert queries in Earth Observation.Source: §7 · p. 10
• Evaluations confirm data efficiency and expert-level precision for the proposed approach.Source: §7 · p. 10