SimCRAFT: Distilling Remote Sensing Agents via Synthetic Trajectories and Contextual Retrieval-Augmented Fine-Tuning

Haoran Wang; Jing Yao; Xu Yang; Zeqing Wang; Yang Zhang; Pedram Ghamisi; Zhengchao Chen

Source-linked AI summary

SimCRAFT: Distilling Remote Sensing Agents via Synthetic Trajectories and Contextual Retrieval-Augmented Fine-Tuning

Haoran Wang, Jing Yao, Xu Yang, Zeqing Wang, Yang Zhang, Pedram Ghamisi, Zhengchao Chen

arXiv:2608.30277v1Aug 31, 2026cs.AIcs.MA

TL;DR

RS agents face deployment barriers from expert-only workflows and reliance on GPT-4-scale runtime orchestration. SimCRAFT combines constraint-validated synthetic trajectories with noise-robust contextual fine-tuning to distill procedural planning into a 7B model, which matches GPT-4-level RS agents and generalizes without additional fine-tuning.

Problem
Manual RS workflows are expert-only, error-prone, and not reusable, while existing agents depend on large-scale closed-source LLMs for runtime orchestration.
Method
SimCRAFT builds SimRS-14k through validated multi-agent synthesis and trains a 7B backbone with retrieved SOPs plus noise perturbations that promote structural rather than mechanical copying.
Results
83.2% PSR, 93.5% TSA, and 0.5% Hal.Rate on SimRS-14k, with 79.4% PSR on KnowFlow-Bench without additional fine-tuning.
Takeaways & Limitations
Constraint-validated synthetic data and context-denoising training align a 100× smaller open-source model with closed-source frontier LLMs on RS procedural planning.
Takeaways & Limitations
The framework evaluates workflow-planning quality rather than downstream product correctness, and its tools and knowledge base remain RS-specific.

Abstract

from arXiv · show

The unprecedented surge in Earth observation data volume and diversity has exposed a critical bottleneck for traditional manual workflows, catalyzing the emergence of Remote Sensing (RS) Agents. However, the practical deployment of these advanced agents is severely hindered by their heavy reliance on large-scale general-purpose LLMs, which lack deep domain expertise and impose prohibitive infrastructure demands. To resolve this, we propose SimCRAFT, a model-agnostic framework that distills sophisticated RS orchestration capabilities into a compact 7B-scale model. Addressing data scarcity, we first pair a multiagent synthesis engine with a Mock Execution Engine that checks schema correctness, inter-tool dependencies, and sensor/tool compatibility, producing SimRS-14k, a large-scale, constraint-validated workflow planning corpus. Second, we propose Contextual Retrieval-Augmented Fine-Tuning (CRAFT) that finetunes the model to reason analogically by adapting retrieved Standard Operating Procedures to novel queries under a noise-robust objective, generalizing RAFT to multi-step RS workflow planning without mechanical copying. Extensive experiments demonstrate that SimCRAFT-7B significantly outperforms openweights LLMs and rivals advanced closedsource models and specialized RS agents, while reproducing across three 7B backbones. This work contributes a competitive open-weights baseline for lightweight RS intelligence, enabling efficient autonomous deployment under resource-constrained or resource-conserving conditions.

1 Introduction

SimCRAFT targets the deployment barriers of RS agents by distilling procedural planning into a compact 7B model. It combines constraint-validated synthetic trajectories with noise-robust retrieval-augmented fine-tuning.

Manual RS workflows are expert-only, error-prone, and not reusable, motivating agents that decompose natural-language queries into executable tool-call sequences.
Runtime orchestration by GPT-4-scale models imposes high compute, connectivity, and privacy demands that hinder deployment on satellites, UAVs, and workstations.
Inference-time RAG and direct end-to-end SFT provide inadequate solutions because small models struggle to extract procedural structure and may memorize seed-task instances.
SimCRAFT synthesizes SimRS-14k with dual-mode simulation, four-agent collaboration, and Mock Execution Engine checks for schema, dependencies, and sensor compatibility.
CRAFT trains a 7B model to extract transferable SOP structure from noisy contexts rather than copy retrieved trajectories mechanically.

2 Related Work

Prior RS agents and synthetic-data methods establish retrieval, multi-agent collaboration, and verification-based trajectory generation, but commonly depend on large closed-source models. SimCRAFT instead internalizes procedural planning in a 7B backbone.

Existing RS agents address multi-step planning through hierarchical abstraction, retrieval augmentation, and multi-agent collaboration.
SimRS-14k contains 14,003 trajectories after a 13.6% rejection rate.
These RS systems rely on GPT-4-scale closed-source LLMs for runtime orchestration, whereas SimCRAFT writes procedural planning into a 7B model’s weights.
General-domain synthetic-data pipelines generate and verify tool-use trajectories, with methods spanning simulated environments, API filtering, self-evolving models, and cross-environment interactions.
RAG grounds frozen models with external knowledge, while RAFT and related methods move retrieval into training to teach noise filtering and analogy-based reasoning.

3 Method

SimCRAFT formulates RS-agent planning as constrained multi-step tool calling and trains a compact model through validated synthetic data and contextual SOP retrieval. Its pipeline combines intent clarification, procedural retrieval, autoregressive planning, and noise-denoising supervision.

3.1 Task Formulation: An RS agent produces a trajectory of atomic-tool calls with arguments and results while satisfying sensor, dependency, and parameter-schema constraints.
3.1 Task Formulation: The workflow first clarifies missing intent parameters, then retrieves top-k SOPs, and finally generates tool calls conditioned on context and execution history.
3.2 Framework Overview: SimCRAFT uses Phase I multi-agent synthesis and validation to build SimRS-14k, then Phase II CRAFT injects retrieved SOPs and perturbations during training.
3.3 Atomic Toolset: The Atomic Toolset provides verified atomic operations with strictly typed JSON schemas and valid ranges for flexible long-horizon composition.
3.4 Multi-Agent Data Synthesis: Four role-complementary agents address ordering and dependency errors, while the Mock Execution Engine validates generated trajectories before training.
3.4.3 Mock Execution Engine Check: Across 16,200 candidate trajectories, validation rejected 13.6%, leaving 14,003 SimRS-14k trajectories; expert inspection found 99.5% of 2,800 sampled instances passed.
3.5.2 Noise-Robust Training: CRAFT perturbs retrieved contexts through irrelevant-context injection and parameter mutation, forcing the model to override incorrect details while preserving validated supervision.

4 Experiments

Experiments evaluate SimCRAFT across isolated benchmarks, model backbones, data mixtures, and perturbation settings. The results show strong planning performance, transfer beyond the synthetic distribution, and consistent gains from CRAFT.

Datasets and evaluation: 14,003 validated trajectories remain after expanding 162 seed tasks, with 500 held out and 13,503 used for CRAFT fine-tuning.The held-out trajectories avoid overlap in geographic, temporal, and sensor parameters.
Evaluation protocol: Five metrics assess executable planning, tool selection, schema-exact arguments, hallucinations, and invalid JSON outputs.PSR and AEM use strict matching and therefore provide conservative lower bounds.
Main results: 83.2% PSR and 0.5% hallucination rate let SimCRAFT-Qwen2.5-7B surpass DeepSeek-R1 in PSR and exceed GPT-4-driven general agents by 4.0 percentage points.GPT-4-based RS agents retain a slight PSR advantage, while SimCRAFT leads in execution precision and hallucination control.
Generalization to KnowFlow-Bench: 79.4% PSR on KnowFlow-Bench is 29.2 points above same-backbone Inf-RAG and only 3.2 points below GPT-4 + Inf-RAG.The cross-benchmark drop is −3.8%, compared with −10.8% for vanilla SFT.
Generalization to ThinkGeo: On independently sourced ThinkGeo, SimCRAFT stays close to GPT-4 while same-size baselines drop sharply on the unseen tool space.ThinkGeo contains 486 real-imagery tasks and uses a distinct 14-tool action space without fine-tuning.
Ablations and robustness: CRAFT improves vanilla SFT by 9.6%–11.6% PSR across three 7B backbones, while Qwen3-8B reaches 84.7% PSR.At ε = 0.15, perturbations peak at 83.2% PSR; a 50/50 novice split reaches 83.2% PSR, 93.5% TSA, 79.6% AEM, and 0.5% hallucinations.

5 Conclusion

SimCRAFT distills remote-sensing procedural planning into a compact open-source 7B model by combining constraint-validated synthetic data with noise-robust contextual fine-tuning. SimCRAFT-Qwen2.5-7B matches GPT-4-level RS-agent performance on SimRS-14k and generalizes to KnowFlow-Bench without additional fine-tuning.

SimCRAFT is a model-agnostic framework for distilling RS-agent procedural planning into a 7B open-source model.
The framework combines the constraint-validated SimRS-14k corpus with the noise-robust CRAFT training paradigm.
83.2% PSR, 93.5% TSA, and 0.5% Hal.Rate on SimRS-14k show that SimCRAFT-Qwen2.5-7B matches GPT-4-level RS agents.
79.4% PSR on KnowFlow-Bench demonstrates generalization without additional fine-tuning.
The results align a 100× smaller open-source model with closed-source frontier LLMs on RS procedural planning.

Limitations

The evaluation validates workflow-level planning constraints but does not assess the physical accuracy of downstream science products. The framework is also scoped to RS-specific tools, limiting direct cross-domain transfer.

The Mock Execution Engine checks schema, dependency, and sensor compatibility rather than executing GDAL, SNAP, or GEE on real imagery.
End-to-end physical-accuracy evaluation of downstream products, such as change-detection Kappa, remains future work.
PSR, TSA, and AEM assess workflow-level properties but do not measure the correctness of change maps, classification maps, or statistics.
The RS-specific Atomic Toolset and PKB leave cross-domain transfer requiring domain-specific reconstruction.
Evaluation scope: Figure 6 examines PSR against SimRS-14k corpus size with GPT-4 and GPT-5 zero-shot reference lines.
Evaluation scope: Figure 7 reports PSR across trajectory-length buckets to examine long-horizon robustness.

A Dataset Construction and Splits

SimRS-14k is a corpus of RS tool-use trajectories generated from expert-seeded tasks, with stratified held-out evaluation designed to test reusable tool-composition patterns while reducing synthesis leakage.

SimRS-14k contains RS tool-use trajectories generated from expert-seeded tasks.
The construction records 162 expert-seeded tasks and 16,200 candidate trajectories before validation.
2,197 candidates failed validation, leaving 14,003 retained trajectories in the SimRS-14k corpus.
The corpus includes expert-mode and novice-mode trajectories, with 6,987 complete initial requests and 7,016 clarification-required cases recorded.
The test split is stratified by seed-task type, application domain, and trajectory length, and shares no geographic coordinates, time windows, or sensor parameters with training.
The held-out design reduces within-synthesis leakage and tests whether models learn reusable tool-composition patterns.

B Atomic Toolset Schemas and I/O Specifications

The Atomic Toolset defines a 36-tool RS action space across five functional domains, with schemas specifying required and optional arguments, artifact types, and registered outputs. Validation enforces argument, range, and artifact-compatibility constraints.

The Atomic Toolset specification maps 36 RS-specific tools exhaustively to five functional domains.
Required arguments are mandatory JSON keys, optional arguments are typed optional keys, and I/O fields record accepted artifacts and registered outputs.
The schema checker validates required-key presence, JSON types, enumerated values, numeric ranges, and artifact compatibility.
Path-valued arguments must reference artifacts returned by previous valid tool calls through registered output fields.
The five domains contain 12 Data & Preprocessing tools, 6 AI Interpretation tools, 4 Spatio-Temporal Analysis tools, 11 Physical & GIS Analytics tools, and 3 Visualization tools.
Representative tools cover data search and download, semantic segmentation and change detection, temporal analysis, spectral-index and GIS operations, and map rendering.

B.2 Data and Preprocessing Schemas

The schemas define typed inputs, optional parameters, and registered outputs for data access, preprocessing, AI interpretation, and coordinate operations. These interfaces enforce artifact compatibility across downstream workflow steps.

Data Access: rs_data_search queries location and time range, optionally constraining time step, platform, and cloud cover, then returns searchable image_ids.Its output is a metadata record set used by later download steps.
Data Access: rs_data_download consumes returned image_ids and registers downloaded raster artifacts for downstream tools.The optional output_dir controls where downloads are written.
Preprocessing: Preprocessing schemas cover reprojection, cropping, mosaicking, registration, radiometric calibration, atmospheric or cloud removal, and image enhancement.These tools accept registered raster inputs and return transformed raster artifacts, with operation-specific parameters and compatibility constraints.
Artifact Utilities: File conversion and metadata reading expose reusable registered artifacts and metadata for later workflow operations.Conversion accepts raster or vector files, while metadata reading returns bands, resolution, CRS, and spatial extent.
Spatial Support: Coordinate transformation accepts a place name or pixel coordinate and returns a coordinate object usable by later spatial filters.An optional reference image can support the transformation.
AI Interpretation: AI interpretation tools transform registered rasters into detection, segmentation, classification, change-detection, or scene-label artifacts.Inputs and outputs vary by task, including target classes, paired temporal rasters, masks, confidence scores, and classification maps.

B.4 Spatio-Temporal Analysis Schemas

The schemas cover temporal processing, spectral and physical analytics, GIS and vector operations, sensor-specific processing, and visualization. Each tool specifies accepted artifact types and returns a derived artifact or analysis result for subsequent steps.

Spatio-Temporal Analysis: Temporal tools compute statistics, smooth or interpolate time series, analyze trends, and detect disturbances or breakpoints.Trend analysis supports linear_regression or mann_kendall, while disturbance detection supports bfast or landtrendr.
Physical and GIS Analytics: Spectral and physical analytics derive index rasters, parameter-inversion rasters, band-math results, terrain products, and GIS statistics.Operations include spectral-index calculation, parameter inversion, threshold masks, slope, aspect, hillshade, and zonal statistics.
Physical and GIS Analytics: Vector analysis computes areas or feature counts from registered vector artifacts and returns vector-statistics objects or files.The supported operations are calculate_area and count_features.
Sensor-Specific Processing: Sensor-specific tools process SAR, hyperspectral, and LiDAR artifacts into corrected, reduced, unmixed, rasterized, or classified products.Supported operations include speckle_filter, terrain_correction, pca, mnf, spectral_unmixing, point_to_raster, and classify_ground.
Visualization: Visualization tools render maps, plot charts, and convert raster masks into vector artifacts.Map rendering accepts raster, vector, and class-map layers, while chart plotting consumes statistics files or result objects.

C Mock Execution Engine Validation

The Mock Execution Engine filters synthesized trajectories through planning-level validation and recoverable error handling before they become supervised targets. Its state-machine design supports replanning while remaining distinct from physical execution and downstream accuracy evaluation.

Scope Boundary: The engine does not invoke GDAL, SNAP, or Google Earth Engine on raw imagery and does not evaluate downstream physical accuracy.Examples of excluded outcomes include change-detection masks and crop-classification maps.
Validation Checks: Schema checks reject invalid required fields, types, enumerated values, numeric ranges, and artifact compatibility.Examples include an out-of-range cloud-cover threshold, a missing time_range, or a string supplied to a numeric argument.
Validation Checks: Dependency and path checks reject calls that reference artifacts not produced by a previous valid step.The Dynamic Path Registry prevents fabricated intermediate files from appearing in syntactically plausible trajectories.
Validation Checks: Sensor and tool compatibility checks reject operations requiring unavailable product characteristics, such as a spectral index without a near-infrared band.This check complements schema and dependency validation.
Recovery: Error injection emits recoverable runtime-like failures that trigger reflection and replanning, using ε=0.15 in Phase I.A CloudCoverExceeded failure can route the trajectory to an adjusted search or preprocessing plan.
Workflow Control: The shared state machine cycles through plan, execute, reflect, and summarize, dispatching reflection only after validation failure or injected error.This design prevents a local invalid step from propagating through the rest of a long-horizon trajectory.

F Bootstrap Confidence Intervals

Bootstrap confidence intervals quantify the stability of SimCRAFT-Qwen2.5-7B’s test-split metrics across repeated resampling. The reported intervals are intended to distinguish stable estimates from test-split artifacts.

Interpretation: Narrow confidence intervals indicate stable estimates rather than artifacts of the test split.This interpretation is reported for the evaluated metrics.