PaperlayerView exact revision on arXiv

Source-linked AI summary

SimCRAFT: Distilling Remote Sensing Agents via Synthetic Trajectories and Contextual Retrieval-Augmented Fine-Tuning

Haoran Wang, Jing Yao, Xu Yang, Zeqing Wang, Yang Zhang, Pedram Ghamisi, Zhengchao Chen

arXiv:2608.30277v1cs.AIcs.MA

TL;DR

Manual remote-sensing workflows are expert-only, error-prone, and not reusable, while existing approaches do not teach models to plan tool calls under schema-level SOP guidance. SimCRAFT combines constraint-validated synthetic workflows with CRAFT training, and its 7B model matches GPT-4-level RS agents on SimRS-14k while generalizing to KnowFlow-Bench without additional fine-tuning.

  • Problem

    Remote-sensing workflows are expert-only, error-prone, and not reusable, while existing approaches do not teach models to plan tool calls under schema-level SOP guidance.

    Source: §1 · pp. 1–2

  • Method

    SimCRAFT combines constraint-validated SimRS-14k workflow synthesis with CRAFT training that uses retrieved SOPs and noise perturbations to teach procedural patterns.

    Source: §3.2 · p. 4, §5 · p. 8

  • Results

    83.2% PSR, 93.5% TSA, and 0.5% Hal.Rate: SimCRAFT-Qwen2.5-7B matches GPT-4-level RS agents on SimRS-14k and reaches 79.4% PSR on KnowFlow-Bench without additional fine-tuning.

    Source: §5 · p. 9

  • Takeaways & Limitations

    SimCRAFT provides a 7B open-source baseline for RS procedural planning that generalizes to KnowFlow-Bench without additional fine-tuning.

    Source: §5 · pp. 8–9

    The evaluation measures workflow-planning quality, not the correctness of downstream products such as change maps, classification maps, or statistics.

    Source: Limitations · p. 9

Abstract

from arXiv · show

The unprecedented surge in Earth observation data volume and diversity has exposed a critical bottleneck for traditional manual workflows, catalyzing the emergence of Remote Sensing (RS) Agents. However, the practical deployment of these advanced agents is severely hindered by their heavy reliance on large-scale general-purpose LLMs, which lack deep domain expertise and impose prohibitive infrastructure demands. To resolve this, we propose SimCRAFT, a model-agnostic framework that distills sophisticated RS orchestration capabilities into a compact 7B-scale model. Addressing data scarcity, we first pair a multiagent synthesis engine with a Mock Execution Engine that checks schema correctness, inter-tool dependencies, and sensor/tool compatibility, producing SimRS-14k, a large-scale, constraint-validated workflow planning corpus. Second, we propose Contextual Retrieval-Augmented Fine-Tuning (CRAFT) that finetunes the model to reason analogically by adapting retrieved Standard Operating Procedures to novel queries under a noise-robust objective, generalizing RAFT to multi-step RS workflow planning without mechanical copying. Extensive experiments demonstrate that SimCRAFT-7B significantly outperforms openweights LLMs and rivals advanced closedsource models and specialized RS agents, while reproducing across three 7B backbones. This work contributes a competitive open-weights baseline for lightweight RS intelligence, enabling efficient autonomous deployment under resource-constrained or resource-conserving conditions.

1 Introduction

SimCRAFT targets lightweight remote-sensing tool planning with training-time trajectory synthesis and SOP-guided fine-tuning. Its two phases create a constraint-validated corpus and train models to extract transferable workflow structure from noisy SOPs.

  • RS workflows remain expert-only, error-prone, and nonreusable, while GPT-4-scale orchestration imposes compute, connectivity, and privacy costs that hinder edge deployment.Source: §1 · pp. 1–2
  • Inference-time RAG yields marginal gains at 7B, while trajectory-only SFT risks instance memorization and is constrained by the lack of large RS tool-use corpora.Source: §1 · p. 2
  • The proposed direction moves SOP retrieval from inference into training, pairing each trajectory with its SOP template and noise perturbations.Source: §1 · p. 2
  • Its synthesis phase combines expert-seeded tasks, dual-mode user simulation, four-agent collaboration, and mock validation of schemas, dependencies, and sensor compatibility to create SimRS-14k.Source: §1 · p. 2
  • CRAFT adapts retrieval-augmented fine-tuning to multi-step RS planning, using irrelevant-context injection and parameter mutation to teach transferable SOP structure rather than mechanical copying.Source: §1 · p. 2
  • SimCRAFT combines training-time data synthesis and schema-aware fine-tuning to lift a 7B-scale open-source model to the level of GPT-4-driven RS agents.Source: §1 · p. 2

2 Related Work

Prior work motivates SimCRAFT’s focus on lightweight RS planning: existing agents depend on frontier models, while RS data synthesis and retrieval-augmented fine-tuning leave domain-specific workflow challenges unresolved.

2.1 Remote Sensing Agents

  • Prior RS agents address multi-step planning but rely on GPT-4-scale closed-source orchestration; SimCRAFT internalizes procedural planning in a 7B backbone.EarthAgent introduces hierarchical task abstraction, CangLing-KnowFlow retrieval-augmented replanning, and GeoGPT, RS-Agent, and GIS Copilot multi-agent collaboration; vision-language models include SkyEyeGPT, SegEarth-R1, RingMo-Agent, and RSGPT.Source: §2.1 · pp. 2–3

2.2 Data Synthesis for Agents

  • SimRS-14k addresses shallow RS interactions and long-horizon physical constraints through dual-mode user simulation and Mock Execution Engine validation.General-domain synthesis pipelines include ToolAlpaca (Tang et al., 2023), APIGen (Liu et al., 2024c), ToolACE (Liu et al., 2025), and TOUCAN (Xu et al., 2025), but rarely model RS parameter schemas and sensor compatibility explicitly.Source: §2.2 · p. 3

2.3 Retrieval-Augmented Fine-Tuning

  • CRAFT applies retrieval-augmented fine-tuning to multi-step RS workflow planning, using SOPs as procedural priors with Irrelevant Context Injection and Parameter Mutation perturbations.Prior retrieval-augmented fine-tuning has chiefly addressed factual knowledge, general instruction following, QA, or single-step API selection; CRAFT targets agent training for multi-step planning.Source: §2.3 · p. 3

3 Method

SimCRAFT combines constraint-validated RS workflow synthesis with contextual retrieval-augmented fine-tuning to distill procedural planning into a compact model. Its pipeline uses multi-agent dialogue and trajectory generation, mock validation, and noise-robust learning from retrieved SOPs.

3.1 Task Formulation

  • RS agent planning is constrained multi-step tool calling: trajectories must satisfy sensor compatibility, data dependencies, and parameter-schema legality.The task proceeds through intent clarification, retrieval of top-k SOPs, and autoregressive planning conditioned on the query, context, and execution history.Source: §3.1 · pp. 3–4

3.2 Framework Overview

  • SimCRAFT has two phases: multi-agent synthesis creates SimRS-14k, then CRAFT trains a compact model with retrieved SOPs and noise perturbations to learn their structure.Qwen-2.5-7B is the primary backbone; the framework is model-agnostic.Source: §3.2 · p. 4

3.3 Atomic Toolset

  • The expert-designed Atomic Toolset spans five RS analysis domains, with atomic tools governed by typed JSON schemas and valid value ranges.Its interface conventions follow mainstream geospatial software, including GDAL, SNAP, and Google Earth Engine.Source: §3.3 · p. 4

3.4.3 Mock Execution Engine Check

  • The Mock Execution Engine checks schema legality, registered-path dependencies, and sensor/tool compatibility at each step, admitting only trajectories that pass all checks.It also injects occasional recoverable errors to teach try-fail-correct recovery; execution failures route to reflection for replanning.Source: §3.4 · p. 4, §3.4.3 · p. 5

3.4.1 Dual-mode User Simulation

  • Dual-mode user simulation pairs expert queries with novice queries that mask parameters, while a clarification agent asks targeted follow-ups until critical information is aligned.This teaches the model when to ask and when to act rather than hallucinating missing defaults.Source: §3.4.1 · pp. 4–5

3.4.2 Multi-Agent Trajectory Synthesis

  • Four role-complementary agents coordinate through a shared state machine: planning drafts a roadmap, execution makes tool calls, reflection recovers errors, and summary aggregates results.The Atomic Toolset constrains their action space, and reflection can intercept and rewrite local errors before they propagate.Source: §3.4.2 · p. 5

3.4.3 Mock Execution Engine Check

  • 13.6% of 16,200 candidate trajectories were rejected, leaving 14,003 for SimRS-14k; 99.5% of 2,800 manually inspected trajectories passed.The 13 flagged instances remained in the corpus as residual noise.Source: §3.4.3 · p. 5

3.5.1 PKB Retrieval; 3.5.2 Noise-Robust Training

  • CRAFT retrieves top-k expert-validated SOPs from a frozen dual-encoder retriever and perturbs training contexts with unrelated workflows or valid parameter mutations.Because targets remain constraint-validated while contexts can contain incorrect fragments, the model learns to transfer procedural structure and override misleading details; unperturbed retrieval is used at deployment.Source: §3.5.1 · p. 5, §3.5.2 · p. 6

4 Experiments

Experiments show SimCRAFT-7B performing competitively on SimRS-14k and transferring to KnowFlow-Bench and ThinkGeo without ThinkGeo fine-tuning. Ablations examine backbone, data mixture, noise robustness, data scaling, and long-horizon performance.

4.1 Experimental Setup

  • The validated SimRS-14k corpus contains 14,003 trajectories; evaluation also uses 324 KnowFlow-Bench tasks held out from training across geography, time, and sensor parameters.SimCRAFT-Qwen2.5-7B uses Qwen-2.5-7B-Instruct with LoRA and is trained for three epochs; experiments run on eight NVIDIA A800 GPUs.Source: §4.1 · p. 6

4.2 Baselines; 4.3 Evaluation Metrics

  • Evaluation compares generalist LLMs, general agents, and RS agents using PSR, TSA, AEM, Hal.Rate, and Fmt.Err to assess planning quality and robustness.PSR and AEM use strict matching and are conservative lower bounds; TSA is invariant to tool ordering and argument values.Source: §4.3 · p. 7, §4.2 · p. 7

4.5 Generalization to KnowFlow-Bench

  • 79.4% PSR on KnowFlow-Bench surpasses same-backbone Inf-RAG by 29.2%, trails GPT-4 + Inf-RAG by 3.2%, and drops 3.8% across benchmarks versus 10.8% for vanilla SFT.The authors interpret this smaller cross-benchmark drop as evidence that CRAFT learns transferable tool-use structure rather than seed-task surface templates.Source: Table 2 · p. 7, §4.5 · p. 7

4.4 Main Results

  • SimCRAFT-7B matches GPT-5 and Gemini-3-Flash, exceeds 671B DeepSeek-R1 at 83.2% PSR, beats GPT-4 general agents by a +4.0% PSR margin, and reduces hallucinations to 0.5%.GPT-4-based RS agents retain a slight PSR advantage, while SimCRAFT surpasses them in execution precision and hallucination control.Source: §4.4 · p. 7

4.6 Generalization to ThinkGeo

  • Without ThinkGeo fine-tuning, SimCRAFT stays close to GPT-4 on its unseen 14-tool action space while same-size baselines drop sharply.ThinkGeo comprises 486 tasks and 1,778 expert-verified steps, with no shared provenance with the synthesis pipeline or CangLing-KnowFlow.Source: §4.6 · pp. 7–8, Table 3 · p. 8

4.7 Ablation Studies

  • Across Qwen-2.5-7B, Llama-3-8B, and Mistral-7B, CRAFT improves PSR over vanilla SFT by 9.6%–11.6%; Qwen3-8B reaches 84.7% PSR.The CRAFT PSR range is 7.5%, versus a 3.7% Zero-Shot capability gap, supporting the reported model-agnostic positioning.Source: §4.7 · p. 8
  • With 13,974 trajectories, a 50/50 novice-expert split optimizes all four metrics, reaching 83.2% PSR, 93.5% TSA, 79.6% AEM, and 0.5% Hallucination Rate.Extreme novice ratios yield PSR below 80.3%; the results are attributed to coexistence of both data distributions.Source: §4.7 · p. 8
  • At ε = 0.15, perturbations yield a peak 83.2% PSR; ε = 0.50 falls below the unperturbed baseline, while context injection and parameter mutation combine for a 4.6% gain.Their individual gains are +3.1% and +1.8%, close to their 4.9% linear sum; the sweep and component breakdown are reported in Tables 4 and 5.Source: §4.7 · p. 8, Table 4 · p. 8, Table 5 · p. 8

5 Conclusion

SimCRAFT distills remote-sensing agent procedural planning into a model-agnostic 7B open-source model using constraint-validated synthetic data and noise-robust CRAFT. It matches GPT-4-level RS agents on SimRS-14k and generalizes to KnowFlow-Bench without additional fine-tuning.

5 Conclusion

  • SimCRAFT combines a constraint-validated SimRS-14k corpus with noise-robust CRAFT to distill RS-agent procedural planning into a model-agnostic 7B open-source model.Source: §5 · p. 8
  • 83.2% PSR, 93.5% TSA, and 0.5% Hal.Rate on SimRS-14k; 79.4% PSR on KnowFlow-Bench without additional fine-tuning, matching GPT-4-level RS agents.The paper states this aligns a 100× smaller open-source model with closed-source frontier LLMs on RS procedural planning.Source: §5 · p. 9

Limitations

SimCRAFT’s evaluation covers workflow-planning quality and compatibility checks, not downstream scientific-product correctness. Its RS-specific tools and knowledge base also limit cross-domain transfer without domain-specific reconstruction.

Limitations

  • The Mock Execution Engine checks schema, dependencies, and sensor compatibility without running tools on imagery, while PSR, TSA, and AEM do not measure downstream product correctness.End-to-end physical-accuracy evaluation of products such as change maps, classification maps, or statistics remains future work.Source: Limitations · p. 9
  • The Atomic Toolset and PKB cover only RS-specific tools, so transfer to domains such as medical imaging or bioinformatics requires domain-specific reconstruction.Source: Limitations · p. 9

A Dataset Construction and Splits

SimRS-14k is a constraint-validated corpus of remote-sensing tool-use trajectories generated from expert-seeded tasks. Its construction counts, mode labels, split rules, and novice-ratio ablation subset are reported for reproducibility.

A Dataset Construction and Splits

  • 14,003 trajectories were retained from 16,200 candidates after 2,197 failed validation, forming the SimRS-14k corpus.The initial pool comprised 162 expert-seeded tasks.Source: A Dataset Construction and Splits · p. 12
  • The corpus labels 6,987 expert-mode trajectories as complete initial requests and 7,016 novice-mode trajectories as requiring clarification.The two reported mode counts sum to 14,003 trajectories.Source: A Dataset Construction and Splits · p. 12
  • The test split is stratified by seed-task type, application domain, and trajectory length, with no geographic coordinates, time windows, or sensor parameters shared with training.The stated aim is to reduce within-synthesis leakage and test reusable tool-composition patterns rather than repetition of locations, dates, or sensor settings; the nine application domains and eight technical categories are distinct from the Atomic Toolset’s five functional domains.Source: A Dataset Construction and Splits · p. 12
  • The novice-ratio ablation uses a fixed 13,974-trajectory subset—2 × 6,987—to compare expert-only, novice-only, and mixed-ratio settings at equal corpus size.The subset is twice the smaller post-filtered mode count.Source: A Dataset Construction and Splits · p. 12

B Atomic Toolset Schemas and I/O Specifications · B.1 Domain Mapping

The appendix specifies a 36-tool remote-sensing action space, its schema and I/O conventions, and its exhaustive mapping across five functional domains. It also details validation of arguments, paths, and artifact compatibility.

B Atomic Toolset Schemas and I/O Specifications

B.1 Domain Mapping

  • The five functional domains contain 12 Data & Preprocessing tools, 6 AI Interpretation tools, 4 Spatio-Temporal Analysis tools, 11 Physical & GIS Analytics tools, and 3 Visualization tools, forming the 36-tool action space.Source: §B.1 · p. 12
  • Examples span rs_data_search and rs_data_download, AI detection and segmentation, temporal trend analysis, GIS overlay, SAR/LiDAR processing, and map/chart visualization.Source: §B.1 · p. 12

B.2 Data and Preprocessing Schemas · B.3 AI Interpretation Schemas

The schemas define typed inputs, optional parameters, and artifact outputs for remote-sensing data search and preprocessing, followed by AI interpretation. They specify how registered raster artifacts flow between tools and how interpretation results are represented.

B.2 Data and Preprocessing Schemas

  • Data search takes required location and time-range queries plus optional temporal, platform, and cloud-cover filters, returning image IDs that feed the download step.Download requires those image IDs and produces registered raster artifacts for downstream tools.Source: §B.2 · p. 12
  • Geospatial preprocessing schemas cover reprojection, cropping, mosaicking, and registration, each consuming raster paths and producing transformed raster artifacts.Cropping can use a bounding box or vector mask; mosaicking supports mean, first, or blend overlap handling.Source: §B.2 · pp. 12–13
  • Radiometric calibration specifies a calibration level and optional sensor metadata, while cloud or haze removal accepts a method and returns corrected raster artifacts.Calibration levels include TOA reflectance, surface reflectance, or brightness temperature; removal methods include dark-channel or QA-band masking.Source: §B.2 · p. 13
  • Utility schemas provide image enhancement, metadata reading, file conversion, and coordinate transformation for registered imagery and spatial-filter workflows.Enhancement supports stretching, equalization, or pansharpening; metadata covers bands, resolution, CRS, and extent, while coordinate transformation returns a reusable coordinate object.Source: §B.2 · p. 13

B.3 AI Interpretation Schemas

  • Object detection requires image paths and a target class, with an optional confidence threshold, and returns target locations with confidence scores.Source: §B.3 · p. 13
  • Segmentation and classification schemas produce class-specific masks, instance masks or contours, and full-coverage land-use rasters with class maps.Semantic segmentation requires a target class and may specify multiple target classes; instance segmentation supports counting and geometry analysis.Source: §B.3 · p. 13
  • Change detection compares pre- and post-change rasters in binary or semantic mode, while scene classification returns scene labels and an optional summary artifact.Source: §B.3 · p. 13

B.4 Spatio-Temporal Analysis Schemas · B.5 Physical and GIS Analytics Schemas · B.6 Visualization Schemas

The schemas specify inputs, options, and output artifacts for spatio-temporal analysis, physical and GIS analytics, and visualization workflows.

B.4 Spatio-Temporal Analysis Schemas

  • Temporal schemas cover time-series aggregation, smoothing or gap filling, trend estimation, and disturbance or breakpoint detection.They specify required raster paths and method or operation fields, with optional parameters or requested output metrics.Source: §B.4 · p. 13

B.5 Physical and GIS Analytics Schemas

  • Physical and raster-analytics schemas compute spectral indices, invert physical parameters, and derive raster, mask, or scalar results through thresholding or band math.The inputs are registered rasters with required spectral bands or parameter-compatible products; raster math also accepts an expression and optional output type.Source: §B.5 · p. 13
  • Terrain and GIS schemas produce slope, aspect, or hillshade rasters, buffer and overlay vectors, zonal statistics, and vector-area or feature-count summaries.Zonal statistics combine raster inputs with a zone vector, while buffer and overlay operations use registered vector artifacts.Source: §B.5 · pp. 13–14
  • Sensor-specific schemas process SAR rasters, hyperspectral imagery, and LiDAR point clouds into derived, reduced, unmixed, rasterized, or classified artifacts.Supported operations include SAR speckle filtering or terrain correction, hyperspectral PCA, MNF, or spectral unmixing, and LiDAR point-to-raster conversion or ground classification.Source: §B.5 · p. 14

B.6 Visualization Schemas

  • Map-rendering schemas combine registered raster, vector, and class-map inputs into a rendered map, with optional overlays, class mapping, bands, and style.Source: §B.6 · p. 14
  • Visualization schemas also turn registered statistics files or result objects into chart artifacts and raster masks into GIS-registered vector artifacts.Chart creation requires data files and a chart type; raster-to-vector conversion optionally specifies class values, simplification tolerance, and GeoJSON or shapefile output.Source: §B.6 · p. 14

C Mock Execution Engine Validation · D Multi-Agent Synthesis Details · E CRAFT Retrieval and Fine-Tuning Details

The appendices detail how the Mock Execution Engine validates synthesized workflows, how specialized agents generate and revise trajectories, and how CRAFT retrieves and denoises SOP context during fine-tuning.

C Mock Execution Engine Validation

  • The engine injects recoverable failures to trigger replanning, but does not execute tools on imagery or assess downstream physical accuracy.For example, CloudCoverExceeded can prompt the Reflection Agent to adjust a search or preprocessing plan.Source: C Mock Execution Engine Validation · p. 14
  • The Mock Execution Engine rejects illegal arguments, references to unregistered artifacts, and sensor–tool incompatibilities before workflows become training targets.Its Dynamic Path Registry tracks outputs from successful calls, preventing fabricated intermediate files from entering trajectories.Source: C Mock Execution Engine Validation · p. 14

D Multi-Agent Synthesis Details

  • User Simulation produces complete Expert Mode requests or requests with randomly masked critical slots in Novice Mode, while Clarification tracks time range, sensor, and spatial extent.The Clarification Agent asks targeted questions until the intent is complete.Source: D Multi-Agent Synthesis Details · p. 15
  • The four-stage state machine plans, executes, reflects on validation failures or injected errors, and summarizes; validation prevents an invalid step from propagating through a trajectory.Planning drafts the roadmap, Execution emits tool calls, Reflection revises the plan when dispatched, and Summary reports the completed task.Source: D Multi-Agent Synthesis Details · p. 15

E CRAFT Retrieval and Fine-Tuning Details

  • CRAFT retrieves SOP context from a 1008-SOP PKB using a frozen dual encoder, a similarity threshold δ=0.6, and top-k=2 contexts.Retrieval behavior is aligned between training and inference.Source: E CRAFT Retrieval and Fine-Tuning… · p. 15
  • CRAFT trains the model to recover validated trajectories despite noisy SOP context, using irrelevant-context injection and parameter mutation at εctx=0.15.At inference, retrieval is unperturbed, so the model applies learned context-denoising to clean SOPs.Source: E CRAFT Retrieval and Fine-Tuning… · p. 15
  • The primary Qwen-2.5-7B-Instruct backbone is fine-tuned with LoRA (r=64, α=128, dropout 0.05) using AdamW, a 2×10−5 learning rate, cosine scheduling, and 3 epochs.Source: E CRAFT Retrieval and Fine-Tuning… · p. 15

F Bootstrap Confidence Intervals

A non-parametric bootstrap indicates that SimCRAFT-Qwen2.5-7B’s test-set estimates are stable rather than artifacts of the test split.

F Bootstrap Confidence Intervals

  • Narrow 95% confidence intervals indicate stable SimCRAFT-Qwen2.5-7B estimates rather than test-split artifacts.The intervals use the 2.5th and 97.5th percentiles of bootstrap resamples.Source: F Bootstrap Confidence Intervals · p. 15
  • The assessment uses 1,000 test-set resamples drawn with replacement.It applies a non-parametric bootstrap to SimCRAFT-Qwen2.5-7B.Source: F Bootstrap Confidence Intervals · p. 15

Related papers

Loading 2608.30277v1…