PaperlayerView exact revision on arXiv

Source-linked AI summary

Exploiting answer-invariant redundancies in satellite imagery for efficient VLM inference on edge

Ishani Janveja, Davis Zhang, Seoyul Oh, Deepak Vasisht

arXiv:2609.29029v1cs.CVcs.LG

TL;DR

Onboard satellite VLM query answering is costly because high-resolution imagery requires processing many tiles and vision tokens. Rift combines query-conditioned tile pruning with per-tile elastic prefill; on LLaVA-1.5-7B it reduced energy by 78% and improved accuracy by 28 percentage points versus a system without redundancy reduction.

  • Problem

    High-resolution satellite imagery makes onboard VLM query answering infeasible today, while downsizing can make small objects and narrow features visually ambiguous.

    Source: §1 · p. 1

  • Method

    Rift uses query-conditioned tile pruning followed by elastic prefill, which allocates a vision-token budget to each retained tile.

    Source: §1 · pp. 1–2, §2.3 · p. 3

  • Results

    78% less energy and 28 percentage points higher accuracy were reported for Rift versus a system that does not reduce redundancy.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Rift’s evaluation on LLaVA-1.5-7B demonstrates reduced-energy satellite-edge inference alongside improved accuracy.

    Source: §1 · p. 2

Abstract

from arXiv · show

Onboard vision-language models could enable satellites to answer queries directly, but exhaustive tiled inference over high-resolution imagery is slow and energy-intensive. We identify answer-invariant token redundancy (AITR): image tiles and vision tokens that can be removed without changing the final answer. We present Rift, a two-stage system that performs query-conditioned tile pruning followed by elastic prefill to reduce token budget. We evaluate it on LLaVA-1.5 7B running on Jetson AGX Orin. Compared with exhaustive tiled inference, Rift reduces energy by 78% and latency by 69%, while increasing accuracy from 45% to 73%.

1 Introduction

Satellite VLMs could answer queries onboard, but high-resolution scenes require costly tiled processing that makes current query answering infeasible. Rift prunes query-irrelevant tiles and allocates token budgets across retained tiles, reducing energy and improving accuracy in evaluation.

1 Introduction

  • Downsizing high-resolution imagery can make vessels, vehicles, and narrow flooded streets visually ambiguous, so standard practice tiles the scene and processes each tile independently.LLaVA-1.5-7B supports 336×336-pixel inputs.Source: §1 · p. 1
  • A 6700×6700-pixel scene yields about 400 tiles and 230K vision tokens, making exhaustive inference take 5–7 minutes per query on a 64 GB Jetson AGX Orin.The estimate uses 576 vision tokens per tile.Source: §1 · p. 1
  • Rift targets redundancy at tile and token levels: it prunes tiles unlikely to contain answer evidence, then dynamically allocates vision-token budgets to retained tiles before generation.Both decisions are query-conditioned and aim to reduce energy while maximizing the probability of a correct answer.Source: §1 · pp. 1–2
  • 78% less energy and 28 percentage points higher accuracy were reported for Rift versus a system that does not reduce redundancy.The evaluation used LLaVA-1.5-7B on Jetson AGX Orin and queries from xView and GLH-Bridge.Source: §1 · p. 2

2 Rift

Satellite VLM inference is prefill-heavy because queries contain large visual inputs but produce short answers, and only some tiles and tokens carry answer-relevant signal. Rift addresses this redundancy through query-conditioned tile pruning and per-tile elastic prefill.

2.1 Short Primer on VLM Inference

  • Prefill processes the full input sequence before the first output token, and satellite queries are prefill-dominated because their visual inputs are large while answers are short.Vision-token count determines latency and energy per query in this setting.Source: §2.2 · p. 2, §2.1 · p. 2

2.3 System Design

  • Rift scores each tile by target-object similarity minus similarity to common background terms, then retains tiles whose scene-normalized score exceeds threshold τ.Subtracting background similarity reduces context-driven false positives.Source: §2.3 · p. 3
  • For each retained tile, elastic prefill uses attention-based compression and chooses a token budget independently to balance expected answer accuracy against energy cost.Different query–tile pairs can tolerate different compression levels; LLaVA-1.5 uses budgets from 576 to 72 tokens.Source: §2.3 · p. 3
  • The two stages address answer-invariant redundancy at different granularities: tile pruning removes irrelevant tiles, while elastic prefill reduces tokens within retained tiles.The system makes both query-conditioned decisions before the language model generates its first output token.Source: §2.3 · p. 3
  • A gradient-boosted tree ensemble estimates whether each tile contains the target using pipeline features, including its embedding, relevance score, attention scores, and quadrant relevance scores.This estimate supports the per-tile accuracy–energy decision.Source: §2.3 · p. 3

3 Evaluation

Rift is benchmarked on Jetson AGX Orin using xView and GLH-Bridge VQA queries, with RemoteCLIP for tile pruning and LLaVA-1.5 (7B) for retained tiles. Elastic prefill is reported as cheaper and more accurate than fixed per-tile configurations, while an oracle indicates further potential gains.

3 Evaluation

  • The benchmark uses Jetson AGX Orin at 60 W on whole-image VQA queries from xView and GLH-Bridge, with RemoteCLIP for tile pruning and LLaVA-1.5 (7B) for retained tiles.Source: §3 · p. 4
  • Elastic prefill is cheaper and more accurate than fixed per-tile configurations, and Figure 3 compares their average VLM inference energy, latency, and accuracy.An oracle policy saves 33% more energy and increases accuracy by 3% over the current elastic-prefill decision policy.Source: Figure 3 · p. 4, §3 · p. 4
  • An oracle policy saves 33% more energy and increases accuracy by 3% over the current elastic-prefill decision policy, setting a ceiling for a better per-tile predictor.Source: §3 · p. 4

4 Related Work

Prior satellite query systems optimize query placement, scheduling, and transmission, while efficient VLM methods reduce vision-token costs in different ways. Rift combines query-conditioned tile and token-budget decisions and profiles candidate budgets for energy and accuracy on an edge platform.

4 Related Work

  • Prior satellite query systems optimize query placement, scheduling, and data transmission, whereas Rift reduces inference energy after a VLM query is assigned to a satellite.Rift prunes irrelevant tiles and allocates vision-token budgets across retained tiles.Source: §4 · p. 4
  • Efficient VLM methods compress tokens before prefill [18], prune within LLM layers during prefill [21], or adapt compression to input [20] [7] [19].Source: §4 · p. 4
  • The closest related method combines text-guided tile selection with vision-token pruning for remote-sensing images [10], while Rift profiles candidate token budgets for energy and accuracy rather than computational proxies.Source: §4 · p. 4

A Dataset Generation

The authors construct identification and counting VQA datasets from remote-sensing object annotations, using fixed question templates and deterministic ground-truth answers. The tasks distinguish category presence from count-range prediction, and retain zero-count examples to assess spurious predictions in target-absent images.

A Dataset Generation

  • The QA datasets derive questions from xView and GLH-Bridge object annotations using fixed templates, with ground-truth answers determined from the source annotations.Source: A Dataset Generation · p. 6
  • Object identification asks whether a target category is present, with a binary Yes-or-No answer.Source: A Dataset Generation · p. 6
  • Object counting selects among four ranges—0, 1–2, 3–9, or more than 9—and zero-count examples test whether models spuriously predict objects in target-absent images.Source: A Dataset Generation · p. 6

A.1 xView

For xView, the authors map fine-grained annotations into broader categories suitable for whole-image questions. They balance category examples and report separate question totals for identification and counting.

A.1 xView

  • The xView annotations are grouped into broader semantic categories, such as combining several aircraft labels into “aircraft.”Source: §A.1 · p. 6
  • Identification uses four categories, while counting uses three and excludes buildings or structures because dense, partially connected structures make instance boundaries visually ambiguous.Source: §A.1 · p. 6
  • The xView set contains 978 identification questions across 635 unique images, with positive and negative examples balanced within each category.Source: §A.1 · p. 7
  • Counting uses balanced count buckets and yields 372 questions: 172 aircraft, 152 boats-or-ships, and 48 rail-vehicle questions.Source: §A.1 · p. 7
  • Across both tasks, xView contains 1,350 whole-image questions over 656 unique images.Source: §A.1 · p. 7

A.2 GLH-Bridge

The GLH-Bridge set uses annotated bridge images to construct balanced identification and counting questions, totaling 594 questions over 489 unique images.

A.2 GLH-Bridge

  • The source data are publicly annotated GLH-Bridge training and validation images; its single target category, bridge, requires no semantic class aggregation.Whole-image bridge counts are computed from annotated instances.Source: §A.2 · p. 7
  • 198 identification questions are balanced between 99 positive and 99 negative examples.All 99 zero-bridge images are retained, alongside 99 subsampled positive images.Source: §A.2 · p. 7
  • 396 counting questions are balanced across four count buckets, with 99 examples per bucket.The zero-count bucket limits each bucket to 99 images.Source: §A.2 · p. 7
  • 594 whole-image questions span 489 unique images.The set combines identification and counting questions.Source: §A.2 · p. 7

B End-to-end walkthrough on a test query from xView image 1565.tif

A test aircraft-count query is traced through image tiling, query-conditioned tile scoring and pruning, per-tile token choices, and answer aggregation.

Step 1: Tiling

Step 2: Relevance scoring

  • The background-margin score compares tile similarity to “aircraft” against generic background terms, then smooths and normalizes scores; the top-right aircraft cluster ranks highest.The dense urban area in the lower two-thirds scores near the bottom of the range.Source: B End-to-end walkthrough on a test… · p. 8, Figure 5 · p. 8

Step 3: Tile pruning

Step 4: Per-tile token budgets

  • Aircraft tiles receive 72 tokens while empty urban tiles get 288, because compressing clear-target tiles barely changes counts but aggressive compression can make clutter tiles hallucinate objects.The policy assigns token budgets independently to retained tiles.Source: B End-to-end walkthrough on a test… · p. 8

Step 5: Per-tile answers and aggregation

Loading 2609.29029v1…