ThinkGeo: Evaluating Tool-Augmented Agents for Remote Sensing Tasks

Motivation

• ThinkGeo targets a gap left by general-purpose and web-grounded agent benchmarks: evaluating tool use on spatially grounded remote-sensing tasks.Such tasks involve real EO imagery and demands including geodetic metadata, spatial resolution, temporal dynamics, and unit-aware calculations.Source: §1 · pp. 1–2

Benchmark contributions

• ThinkGeo pairs real EO imagery with an executable tool environment and structured evaluation signals for perception, planning, and geospatial reasoning.Its step-by-step and end-to-end modes assess instruction adherence, tool use, argument formatting, multi-step reasoning, and final-answer accuracy.Source: §1 · p. 2
• The benchmark comprises 486 agentic tasks and 1,778 expert-verified reasoning steps across optical RGB and SAR imagery.The task suite spans urban, environmental, transportation, aviation, industrial, change-detection, and disaster-related scenarios.Source: §1 · p. 2
• The benchmark includes 14 tools spanning perception, logic and numeric reasoning, and visualization, alongside step-wise and end-to-end evaluation modes.The study compares GPT-4o, Claude-3, Qwen-2.5, and LLaMA-3, finding persistent gaps in multimodal tool reasoning and execution-trace alignment.Source: §1 · p. 2

2. Related Work

• Prior benchmarks cover single-step API calls, general tool use, multimodal workflows, or other domains, while remote-sensing agents often lack structured trace evaluation and step-level error attribution.The cited studies report tool-use and geospatial performance gaps, including failures in tool selection, argument grounding, and spatial-unit reasoning [18] [23] [24]; [34]; [15].Source: §2 · p. 3

3. ThinkGeo Benchmark

• ThinkGeo evaluates agents on spatially grounded remote-sensing tasks that require interpreting optical imagery, planning multi-step tool use, and producing geospatially coherent outputs.Its design covers goals, use-case categories, query construction, and source datasets.Source: §3 · p. 3

3.1. Design Goals

• Queries require fine spatial fidelity and unit-based reasoning, while tool use is implicit, so agents must infer the needed modules and their order.Examples include measuring distances, counting structures within buffers, and identifying damaged-building features [21] [24]; implicit tool planning aligns with ReAct-style traces [29].Source: §3.1 · p. 3

3.2. Use Case Categories

• Seven categories cover urban planning, disaster assessment and change analysis, environmental monitoring, transportation, aviation, recreation, and industrial sites.Tasks include multi-temporal damage comparison, vehicle counting and direction, runway occupancy, recreational-field coverage, and industrial-tank measurement.Source: §3.2 · p. 4

3.3. Query Construction Pipeline

• Experts curate imagery, author natural-language ReAct queries with multi-step traces, and validate samples through expert review and script-based consistency checks.Invalid samples are manually corrected before inclusion.Source: §3.3 · pp. 4–5
• The pipeline generates one to five distinct queries per image, varying spatial relations, counting logic, or temporal comparisons, and labels difficulty using keyword and reasoning-step counts.Prompts omit explicit tool indicators so agents must recover the reasoning chain [29].Source: §3.3 · p. 5

3.4. Source RS Datasets

• The benchmark draws imagery from public remote-sensing datasets spanning multiple applications, resolutions, and annotation types; task annotations are newly created bottom-up.Sources include datasets for transportation, aviation, disaster imagery, urban and industrial scenes, environmental monitoring, maritime ships, and ground vehicles.Source: Table 2 · p. 6, §3.4 · p. 6

Tool Suite

• ThinkGeo extends AgentLego with ChangeDetection and SegmentObjectPixels, organizing tools into perception, logic, and operation categories for spatial analysis and annotation.Logic tools support numerical reasoning and spatial comparisons; operation tools support visual annotation and output formatting.Source: §4 · p. 6

Evaluation

• Step-wise evaluation measures instruction following, tool selection, argument correctness, and summary generation; end-to-end evaluation measures tool-category performance and answer correctness.An LLM judge verifies predictions using curated per-query questions, addressing phrase variation that can mislead deterministic string matching.Source: §4 · p. 6

Quantitative Analysis

• GPT-4o and GPT-4-1106 achieve the strongest overall accuracy across the ThinkGeo evaluations.The Table 3 caption also identifies the GPT-4 family as the overall best-performing group.Source: Table 3 · p. 7, §5 · p. 7
• Tool selection has the highest correlation with final answer accuracy among the step-by-step metrics.Figure 5 plots Pearson correlations between step-by-step execution metrics and final answer accuracy.Source: §5 · p. 7, Figure 5 · p. 7

Tool Call & Error

• GPT-4o and GPT4-1106 have tool-call error rates of 44.46% and 30.86%, while several open-source models invoke tools aggressively but incur high error rates.Smaller Qwen1.5-7B and Phi-3 models often fail despite limited tool use.Source: §5 · p. 7, Figure 6 · p. 7

Failure Analysis

• GPT-4o and Qwen3 failures include argument-formatting errors, redundant or misused tool calls, and outputs lacking spatial justification.The examples also identify misaligned bounding boxes and inaccurate counts for GPT-4o, and disconnected reasoning for Qwen3.Source: §5 · p. 8, Figure 7 · p. 8

Easy vs Hard Queries

• Models show a consistent gap between correct answers to easy and hard queries, with complex multi-step reasoning posing greater challenges.Figure 8 compares correct-answer counts by model and difficulty level.Source: §5 · p. 8, Figure 8 · p. 8

6. Conclusion

• ThinkGeo is a benchmark tailored to evaluate tool-augmented LLM agents on real-world remote-sensing tasks.The study reports gaps in spatial planning, temporal consistency, and domain-specific tool integration.Source: §6 · p. 8
• Although segmentation and change detection improve raw perception, true geospatial reasoning remains an open challenge.The paper calls for agents that combine perception, planning, and execution in spatially grounded remote-sensing environments.Source: §6 · p. 9

ThinkGeo

• Table A1 reports step-by-step and end-to-end results across models on the ThinkGeo SAR benchmark.Its end-to-end measures include tool-type accuracy, final-answer accuracy, and image-grounded answer accuracy.Source: Table A1 · p. 11
• The supplementary material covers SAR evaluation, error breakdowns, use-case samples, tool-usage distributions, category counts, curation effort, and model runtime.These analyses are organized across supplementary sections S1–S7.Source: ThinkGeo · p. 11
• The supplementary analysis reports human effort involved in curating, generating, and verifying the benchmark.It also analyzes evaluated models' runtime performance.Source: ThinkGeo · p. 11

S1. SAR Data Evaluation

• Final answer accuracy remains below 6% across all models on the SAR evaluation.Even the best proprietary models struggle with visual grounding in nonoptical scenarios.Source: S1. SAR Data Evaluation · p. 11
• Proprietary GPT-4 variants lead in instruction and tool usage accuracy on SAR imagery.The passage describes this as evidence of strong generalization in structured reasoning.Source: S1. SAR Data Evaluation · p. 11
• The SAR set contains 50 queries spanning 244 tool-reasoning steps and uses the same three-phase manual curation pipeline.The passage states that this maintains identical depth and consistency in annotation.Source: S1. SAR Data Evaluation · p. 11

S2. Error Analysis

• NoAction errors are prominent for Phi-3-mini (59.60%) and Mistral (55.47%), while Invalid JSON errors affect LLama3-1-8b (82.36%) and Qwen1.5-7b (92.52%).NoAction denotes extra reasoning or summaries without a final tool call or explicit answer; Invalid JSON reflects difficulty producing syntactically correct tool inputs despite selecting the correct tool.Source: S2. Error Analysis · p. 11, Table A2 · p. 11
• Final Answer (SingleStep) errors occur for Phi-3-mini-4k (58.49%), Qwen2.5-7b (69.27%), and Qwen1.5-7b (78.44%), reflecting premature answers that bypass intermediate reasoning.GPT-4o and GPT-4-1106 have remarkably low format-category error rates.Source: S2. Error Analysis · p. 11, Table A2 · p. 11

S3. More Samples by Use Case

• Example queries require agents to sequence multiple tools and combine perception, logic, and spatial reasoning on real remote-sensing imagery.The examples include ReAct-style execution chains and final answers, including a change-detection task comparing pre- and post-disaster images to quantify changes such as structural damage.Source: S3. More Samples by Use Case · p. 11

S4. Tool Usage Distribution

• Calculator, TextToBbox, and RegionAttributeDescription are the most frequently invoked tools; AddText, ObjectDetection, and OCR are rarely used.CountGivenObject and ChangeDetection are mid-frequency tools, supporting core analysis tasks.Source: S4. Tool Usage Distribution · p. 11, Figure A2 · p. 12
• Representative examples pair each user query with a ReAct-style tool-call chain and its final answer.The tool-use distribution highlights varied reliance across geospatial scenarios and multi-step reasoning.Source: Figure A1 · p. 12, S4. Tool Usage Distribution · p. 12

S5. Category-wise Sample Statistics

• Disaster Assessment & Change Analysis contributes the largest share among seven core domains, with 148 queries.Table A3 summarizes easy and hard task counts by category, including totals.Source: S5. Category-wise Sample Statistics · p. 12

• Annotators spent 100 hours selecting high-quality imagery and drafting scenario-specific queries for diverse remote-sensing datasets.Use-case guidelines and representative prompts were developed to guide query design.Source: S6. Human Annotation Effort · pp. 12–13
• Image analysis and ReAct trace construction took 280 hours, including filling missing annotations such as bounding boxes and counts.An o4-mini-powered interface supported traces containing thought steps, tool invocations, and final answers grounded in image content.Source: S6. Human Annotation Effort · pp. 12–13
• Validation and finalization took 120 hours, with manual review and script-based checks of reasoning steps, tool arguments, outputs, and answer derivability.Source: S6. Human Annotation Effort · pp. 12–13

S7. Runtime Analysis

• Proprietary models such as GPT-4o and GPT-4-1106 offer relatively stable runtime profiles, whereas open-source models often show greater variability across step-by-step and end-to-end latencies.Source: S7. Runtime Analysis · p. 13
• Step-by-step latency is longer for some models, including Qwen1.5, while its end-to-end runs often use fewer tool calls or shorter traces and require less computation.Step-by-step action prediction re-encodes the full prior interaction history, whose growing context increases load and inference time.Source: S7. Runtime Analysis · p. 13, Table A4 · p. 13
• Current agentic pipelines incur substantial latency, making reduced end-to-end response time an open research direction for real-time geospatial applications.Source: S7. Runtime Analysis · p. 13