PaperlayerView exact revision on arXiv

Source-linked AI summary

In-Orbit Computing for Remote Sensing: Task-Oriented Processing and Transmission

Yi Zhang, Zesong Fei, Jingxuan Huang, Hua Dang, Jihao Luo, Wenyu Liang, Haoran Zhang, Xinyi Wang

arXiv:2610.30693v1eess.SP

TL;DR

Time-sensitive remote sensing is constrained by the latency and downlink demands of transmit-then-process systems, while onboard computing is also limited. This article proposes a task- and resource-aware framework with three space-ground processing modes; its case study illustrates distinct advantages across communication and computing bottlenecks, including lower collaborative-inference latency in the reported 80 MHz case.

  • Problem

    The amount of task-relevant information to extract onboard depends on mission bottlenecks, including downlink opportunity, response needs, ground-side information requirements, and onboard computing limits.

    Source: §I · p. 1

  • Method

    SSGC-RS selects among onboard task inference, semantic representation delivery, and space-ground split inference according to task requirements and communication-computing resources.

    Source: §I · p. 1

  • Results

    The case study illustrates advantages in task-result delivery, low-rate semantic transmission, and latency-aware collaboration; collaborative inference reaches 6.7 s at 80 MHz.

    Source: C. Space-Ground Collaborative… · p. 6, §VI · p. 7

  • Takeaways & Limitations

    The framework’s modes differ in onboard inference, transmitted representation, and ground processing, offering a task- and resource-aware approach to space-ground remote sensing.

    Source: §VI · p. 7

Abstract

from arXiv · show

The growing volume and timeliness requirements of remote sensing data are exposing the limitations of the conventional "transmit-then-process" architecture, particularly under intermittent satellite-ground connectivity, constrained downlink bandwidth, and limited onboard computing resources. This article presents a semantic-aware space-ground collaborative remote sensing (SSGC-RS) framework, in which remote sensing data are processed to different semantic depths according to task requirements and available communication-computing resources. Three operating modes are considered, ranging from fully onboard task inference to semantic representation delivery and space-ground split inference. A land-cover recognition case study illustrates how these modes address different communication and computing bottlenecks. Finally, emerging challenges and future directions for intelligent remote sensing systems are discussed.

I. INTRODUCTION

Conventional transmit-then-process systems can incur latency and spend scarce downlink capacity on information irrelevant to time-sensitive tasks. The article proposes adapting onboard processing depth to task needs and communication-computing resources.

I. INTRODUCTION

  • The transmit-then-process pipeline can add latency and consume scarce downlink resources on information irrelevant to the target task.These limitations become pronounced with scarce bandwidth, short contact windows, and stringent deadlines.Source: §I · p. 1
  • The appropriate amount of onboard processing depends on the mission bottleneck, from tight downlink or response time to richer ground-side information or limited onboard compute.The article frames the key choice as how much task-relevant information to extract in space.Source: §I · p. 1
  • SSGC-RS uses task requirements and communication-computing resources to select processing depth and the representation sent to the ground.Relevant context includes latency, downlink conditions, visibility windows, onboard computing and energy, and model characteristics.Source: §I · p. 1
  • The article examines onboard task inference, semantic representation delivery, and space-ground split inference through a land-cover recognition case study.It also discusses adaptive mode selection, trustworthy semantic extraction, and constellation-scale coordination.Source: §I · pp. 1–2

II. SYSTEM ARCHITECTURE

SSGC-RS treats onboard processing depth as an adaptive decision shaped by task context and available resources. It offers three modes that differ in transmitted information and where inference is completed.

A. Framework Overview

  • Task and resource context jointly determine onboard processing depth and the form of the downlink payload.The framework makes processing depth a decision variable rather than a fixed system configuration.Source: A. Framework Overview · p. 2

B. Task- and Resource-Aware Operating Modes

  • Onboard task inference sends task-level results when downlink or visibility is limiting; semantic delivery sends compact scene representations when ground applications need richer information.Task-result delivery loses access to visual information discarded onboard, while semantic delivery depends on preserving information needed for ground processing.Source: B. Task- and Resource-Aware Operating… · p. 2
  • Space-ground collaborative inference splits model execution when onboard compute or energy is insufficient, with intermediate features sent to the ground for the remaining inference.The split point couples onboard computation, downlink volume, and ground workload.Source: B. Task- and Resource-Aware Operating… · p. 2
  • The three modes differ in how much processing occurs onboard and what information is transmitted, with corresponding differences in ground-side operations.Ground processing may use a task result, decode or reconstruct a semantic representation, or complete back-end inference.Source: B. Task- and Resource-Aware Operating… · pp. 2–3

III. THREE OPERATING MODES FOR TASK-ORIENTED SPACE-GROUND INFERENCE

The three operating modes distribute task processing differently between satellite and ground.

III. THREE OPERATING MODES FOR TASK-ORIENTED SPACE-GROUND INFERENCE

  • The operating modes differ in how much of the task is completed on the satellite versus the ground.Source: §III · p. 3

A. Onboard Task Inference: From Raw Observation to Task-Level Results

Onboard task inference sends task-level outputs rather than raw observations, which can reduce downlink demands when the satellite can run the model. For semantic segmentation, the compact class map is useful for land-cover applications, but onboard-only results are irreversible and label-efficient models generally trail fully supervised accuracy.

A. Onboard Task Inference: From Raw Observation to Task-Level Results

  • Onboard task inference sends only the inference outcome, which is attractive when that result is much smaller than the source observation and fits onboard latency and energy budgets.The downlink may contain a class map, object locations and labels, or a change mask; a confidence measure can flag regions for further inspection.Source: A. Onboard Task Inference: From Raw… · p. 3
  • Semantic segmentation is a representative onboard task because its pixel-level class map is substantially more compact than the source image while remaining useful for land-cover applications.Source: A. Onboard Task Inference: From Raw… · p. 3
  • Fully supervised segmentation generally provides the most reliable pixel-level predictions when sufficient annotations are available, while label-efficient approaches reduce annotation and model-update costs.Weakly and semi-supervised learning and unsupervised domain adaptation use alternatives to large pixel-level annotation, but their accuracy and reliability generally remain lower.Source: A. Onboard Task Inference: From Raw… · p. 3
  • Onboard processing is irreversible: if the model misses an object or discards a region, the ground cannot recover that information without an additional transmission.Source: A. Onboard Task Inference: From Raw… · p. 4

B. Semantic Representation Delivery: From Pixel Transmission to Compact Semantics

Ground-side applications that need more than a task result require richer scene information, but conventional codecs prioritize visual fidelity rather than downstream task performance. Compact token representations can preserve more useful scene information at lower transmission rates, though their value depends on correctly identifying important regions.

B. Semantic Representation Delivery: From Pixel Transmission to Compact Semantics

  • Conventional image codecs preserve visual fidelity rather than downstream task performance, so their bit allocation may miss task-important regions.Source: B. Semantic Representation Delivery:… · p. 4
  • Discrete token communication quantizes image features into codebook indices that can be transmitted as a compact representation of the observation.Token rate can be adjusted through token resolution, coding, or selective retention.Source: B. Semantic Representation Delivery:… · p. 4
  • Semantic-aware token transmission selects regions by importance, while unified transmission treats tokens equally; VQ-GAN is a representative unified scheme.Semantic masks can retain important tokens and mask less relevant ones.Source: B. Semantic Representation Delivery:… · p. 4
  • Fig. 3 contrasts coarse-grained offloading with fine-grained, resource-aware model partitioning for space-ground split inference.Source: Figure 3 · p. 4
  • Semantic-aware token transmission depends on accurate importance estimates; overlooked regions may be discarded, causing unrecoverable information loss and lower downstream task performance.Such errors may not appear in conventional rate-distortion metrics.Source: B. Semantic Representation Delivery:… · p. 4

C. Space-Ground Collaborative Inference: Balancing Communication and Computing

Space-ground collaborative inference distributes model execution between satellite and ground to share computation and communication load. Fine-grained partitioning couples the split point to onboard computation, downlink volume, and ground workload, so its benefit depends on available resources and conditions.

C. Space-Ground Collaborative Inference: Balancing Communication and Computing

  • Collaborative inference applies when the onboard platform cannot support the whole model within its computing or energy budget, or a complex model must meet a tight deadline.The satellite runs early network layers and the ground completes the rest.Source: C. Space-Ground Collaborative… · p. 4
  • Fine-grained partitioning splits a network between satellite and ground, transmitting intermediate features at the split point.A shallow split sends larger features but reduces onboard load; a deep split reduces payload while increasing onboard computation and energy.Source: C. Space-Ground Collaborative… · p. 5
  • The split point jointly determines onboard computation, downlink volume, and ground computation, making partitioning a joint resource-allocation problem.The discrete split choice and continuous resource allocations are typically optimized in separate but coupled steps.Source: C. Space-Ground Collaborative… · p. 5
  • Collaborative inference is beneficial only when reduced onboard computation outweighs the added feature-transmission and ground-processing costs.The optimal split changes with link quality, onboard computation, and latency requirements; modes can also be combined, such as sending uncertain regions for ground refinement.Source: C. Space-Ground Collaborative… · p. 5

IV. CASE STUDY

The case study evaluates three operating modes in a common land-cover recognition setting, matching each mode to its task and resource constraint. Fig. 4 reports segmentation accuracy, reconstruction quality, and task-completion latency for the corresponding bottlenecks.

IV. CASE STUDY

  • The case study uses WHDLD, with 4,940 256 × 256 images across six land-cover categories and an 8:1:1 train-validation-test split.Source: §IV · p. 5
  • Onboard task inference is evaluated by segmentation mIoU, while semantic representation delivery is evaluated by LPIPS reconstruction quality.The study encodes images as 1,024 discrete tokens from a 1,024-entry codebook.Source: §IV · p. 5
  • The split-inference case uses 12 low Earth orbit satellites sharing one ground station and jointly optimizes the split with communication and computing resources.Fig. 4 evaluates the modes using segmentation accuracy, reconstruction quality, and task-completion latency, respectively.Source: §IV · p. 5

A. Onboard Task Inference under Downlink Constraints

The three experiments illustrate how task needs and resource bottlenecks shape remote-sensing inference: onboard task outputs suit constrained downlinks, semantic tokens retain richer scene information, and split inference balances onboard computing with feature transmission.

A. Onboard Task Inference under Downlink Constraints

  • At 0.1 bpp, onboard segmentation maps achieve mIoU 0.60 with BPG and 0.54 with JPEG2000, versus 0.28 and 0.18 for compressed source images, respectively.The map-delivery scheme approaches the uncompressed-image upper bound of 0.633 as rate increases.Source: A. Onboard Task Inference under… · p. 5, Figure 4 · p. 6

B. Semantic Representation Delivery under Rate Constraints

  • At 0.078 bpp, SQ-GAN achieves LPIPS 0.261; similar quality requires about 0.15 bpp with VQ-GAN, 0.21 bpp with BPG, and 0.55 bpp with JPEG2000.Semantic-aware selection allocates bits preferentially to visually and semantically important regions.Source: B. Semantic Representation Delivery… · p. 5, Figure 4 · p. 6

C. Space-Ground Collaborative Inference under Computation Constraints

  • At 80 MHz, collaborative inference reaches 6.7 s, reducing latency by up to 54.5% versus fully onboard and 47.9% versus fully ground execution.The preferred execution point varies with onboard computing capability and space-ground transmission conditions.Source: Figure 4 · p. 6, C. Space-Ground Collaborative… · p. 6

B. Semantic Representation Delivery under Rate Constraints

  • Onboard task inference sends task-specific outputs, while semantic representation delivery retains richer scene information for ground reconstruction and further processing.The latter depends on whether selected tokens preserve information relevant to later applications.Source: B. Semantic Representation Delivery… · p. 6

V. CHALLENGES AND FUTURE RESEARCH DIRECTIONS

Deploying the three processing modes depends on reliable onboard semantics, adaptation to changing missions and resources, and coordination across satellite and ground nodes. Key challenges include uncertainty under distribution shift and adapting models to new targets on constrained hardware.

A. Task-Adaptive and Trustworthy Semantic Extraction

  • Semantic extraction can be uncertain across regions, seasons, sensors, and object categories unlike the training data, risking errors that ground processing cannot correct when the original observation is unavailable.This is particularly consequential for task-result delivery, where the original observation may not be available for ground-side verification.Source: A. Task-Adaptive and Trustworthy… · p. 6
  • Onboard models could use region- or object-level uncertainty estimates to guide selective patch transmission, added protection for critical information, or ground-side refinement.Reliable uncertainty estimation remains difficult under distribution shifts, especially with limited onboard computing and labeled data.Source: A. Task-Adaptive and Trustworthy… · p. 6
  • Fixed-class models may not suit unexpected objects or newly specified targets, while adapting open-vocabulary models to constrained hardware without losing task accuracy remains challenging.The article identifies open-vocabulary models distilled from ground-based foundation models as a possible solution.Source: A. Task-Adaptive and Trustworthy… · p. 6

B. Dynamic Communication-Computing Orchestration under Intermittent Connectivity

Processing-mode choices can become outdated as connectivity and onboard resources change during a satellite pass. Orchestration must jointly handle mode, model partitioning, and resource allocation, including the possibility that transmission ends before all information arrives.

B. Dynamic Communication-Computing Orchestration under Intermittent Connectivity

  • A mode chosen when a task arrives may no longer fit when its data are ready, because downlink capacity, contact time, and shared onboard computing and energy resources change during a pass.Downlink capacity varies with satellite-ground geometry, and the remaining contact window continuously decreases.Source: B. Dynamic Communication-Computing… · p. 7
  • Jointly optimizing processing mode, model split point, bandwidth, transmission power, and computing allocation is difficult when future contact opportunities and task deadlines must also be considered.The decisions couple discrete choices with continuous resource variables [13].Source: B. Dynamic Communication-Computing… · p. 7
  • A contact window may close before all semantic tokens or intermediate features arrive, motivating policies that preserve task information under partial reception and account for remaining contact time.The optimization should consider task completion probability and energy consumption alongside transmission rate and processing latency.Source: B. Dynamic Communication-Computing… · p. 7

C. Constellation-Scale Closed-Loop Space-Ground Intelligence

Constellation-scale inference must coordinate distributed observations, resources, and models, while model consistency and ground-side feedback remain open coordination challenges. The conclusion presents the framework’s three modes as serving different task and resource conditions, with distinct advantages in result delivery, low-rate semantic transmission, and latency-aware inference.

C. Constellation-Scale Closed-Loop Space-Ground Intelligence

  • Across a constellation, observations, computing resources, and model components may be distributed among satellites and ground stations, with tasks spanning platforms and models updated independently.This extends collaborative inference beyond a single satellite-ground pair.Source: C. Constellation-Scale Closed-Loop… · p. 7
  • Differences in model versions, codebooks, or feature representations can affect whether one node can process another satellite’s information, making synchronization and update scheduling a coordination challenge.Scheduling must account for intermittent links, limited onboard storage, and the communication cost of distributing model parameters.Source: C. Constellation-Scale Closed-Loop… · p. 7
  • Ground feedback such as segmentation corrections, missed detections, and new annotations can support onboard model adaptation and semantic-importance updates, but deciding what to return, where, and when remains an open resource-allocation problem.Source: C. Constellation-Scale Closed-Loop… · p. 7

VI. CONCLUSION

  • The framework selects among three processing modes according to task requirements and communication-computing constraints, with case-study advantages in task-result delivery, low-rate semantic transmission, and latency-aware collaborative inference.The modes differ in onboard inference, the transmitted representation, and the processing left to the ground.Source: §VI · p. 7
Loading 2610.30693v1…