PaperlayerView exact revision on arXiv

Source-linked AI summary

RS-OPSD: Reliable Privileged On-Policy-Self-Distillation for Ultra-High-Resolution Remote Sensing VQA

Chengjie Jiang, Yunqi Zhou, Jiafeng Yan, Sihang Zhao, Chun Yuan, Jing Li

arXiv:2609.38072v2cs.CV

TL;DR

UHR remote-sensing VQA asks models to find small, question-relevant evidence in extremely large images, while existing approaches often use inference-time pruning, search, or tools. RS-OPSD internalizes zoom-in visual privilege through on-policy self-distillation, supported by evidence-annotated data and designs for contextual evidence and reliable supervision. It achieves SOTA on three benchmarks, with a smaller variant offering a favorable accuracy–efficiency trade-off; acquiring the required annotations remains costly and limits scalability.

  • Problem

    UHR remote-sensing VQA requires finding question-relevant evidence occupying a tiny fraction of extremely large images, and existing approaches often rely on inference-time pruning, search, or tools.

    Source: §1 · p. 2

  • Method

    RS-OPSD uses on-policy self-distillation to internalize zoom-in visual privilege, with GeoEvidence-6K, HF-SR, CPVP, and CAD supporting evidence provision and reliable supervision.

    Source: §1 · pp. 2–3, §5.2 · p. 6, §5.3 · p. 7

  • Results

    RS-OPSD achieves SOTA on XLRS-Bench, MME-RealWorld-RS, and LRS-VQA, exceeding the strongest competitors by 1.6, 3.9, and 2.0 percentage points, respectively.

    Source: §1 · p. 3

  • Takeaways & Limitations

    RS-OPD-Lite offers a favorable accuracy–efficiency trade-off with a substantially smaller student model.

    Source: §7 · p. 10

    Explicit target-region and ground-truth answer annotations are required, and their acquisition cost limits framework scalability.

    Source: E LIMITATIONS AND FUTURE WORK · p. 15

Abstract

from arXiv · show

Ultra-high-resolution (UHR) remote sensing visual question answering (VQA) requires models to resolve small visual evidence within extremely large images. Existing approaches typically rely on token pruning, visual search, or tool-augmented reasoning at inference time. We instead investigate whether the benefit of zoom-in visual privilege can be internalized into the model. We introduce RS-OPSD, a reliable privileged on-policy self-distillation (OPSD) framework for UHR remote sensing VQA. To provide high-quality privileged information with explicit question-relevant evidence, we construct GeoEvidence-6K, containing 6,750 VQA samples across seven task categories with evidence-region annotations, and develop Human Feedback-Guided Skill Refinement (HF-SR) for scalable annotation. To address context loss from tight crops and conflicting signals from imperfect teachers, RS-OPSD introduces Context-Preserving Visual Privilege (CPVP) and Correctness-Aligned Distillation (CAD). Without any additional visual search or tool calls at inference time, RS-OPSD achieves state-of-the-art (SOTA) performance on XLRS-Bench, MME-RealWorld-RS, and LRS-VQA, outperforming previous SOTA models of comparable scale by an average of 4.0 percentage points. Moreover, our 2B variant, RS-OPD-Lite, surpasses most 8B-scale models while achieving the fastest measured inference speed. Our Code, GeoEvidence-6K, and the model weights for RS-OPSD and RS-OPD-Lite are publicly available.

1 INTRODUCTION

UHR remote-sensing VQA is constrained by a where-to-look bottleneck: question-relevant evidence occupies a tiny fraction of very large images. RS-OPSD trains models to internalize zoom-in visual privilege, supported by GeoEvidence-6K and HF-SR, while CPVP and CAD address context loss and unreliable teacher supervision.

1 INTRODUCTION

  • The where-to-look bottleneck arises because question-relevant evidence occupies only a tiny fraction of extremely large images.Existing approaches use token pruning, visual search, or tool-augmented reasoning; the latter approaches add inference-time search or tool use, motivating training-time internalization of zoom privilege.Source: §1 · p. 2
  • GeoEvidence-6K provides 6,750 VQA samples across seven task categories with explicit evidence annotations, while HF-SR reuses human feedback to refine annotation skills.These resources support high-quality zoom-in evidence as privileged training information.Source: §1 · pp. 2–3
  • CPVP gives the teacher global, contextual, and fine-grained evidence, while CAD gates unreliable supervision and aligns token updates with student-response correctness.Together, the components address limited crop-only privilege and potentially harmful guidance from an imperfect teacher.Source: §1 · pp. 2–3
  • 1.6, 3.9, and 2.0 percentage points: RS-OPSD outperforms the strongest competitors on XLRS-Bench, MME-RealWorld-RS, and LRS-VQA, respectively.Using Qwen3-VL-8B as the base model, it achieves SOTA on all three benchmarks.Source: §1 · p. 3
  • RS-OPD-Lite uses a 2B student with a privileged 8B teacher during training, surpasses most 8B-scale models, and has the lowest inference latency.Its latency is 12.8% lower than the second-fastest method; both variants avoid additional visual search or tool calls at inference.Source: §1 · p. 3

2 RELATED WORK

Prior UHR remote-sensing VQA methods address large-image evidence through token pruning, visual search, or tool-augmented reasoning. These approaches respectively discard tokens, search image regions, or acquire high-resolution evidence through inference-time tools.

2 RELATED WORK

  • Token-pruning methods retain selected visual tokens to exploit image redundancy, but discarded evidence is no longer available to later reasoning.GeoLLaVA-8K combines background pruning with anchored selection; UHR-BAT uses query-guided multi-scale selection and region-faithful preservation.Source: §2 · p. 3
  • Visual-search methods identify informative regions and assemble compact evidence before reasoning, avoiding dense processing of the full scene.ZoomSearch hierarchically selects patches while preserving layout; WeaveEarth builds a Minimal Support Evidence Set with spatial metadata and topology.Source: §2 · p. 3
  • Tool-augmented methods actively acquire high-resolution evidence during inference through adaptive cropping, zooming, and stopping behaviors.ZoomEarth and GeoEyes use staged supervised and reinforcement learning to support these behaviors.Source: §2 · p. 3

3 GEOEVIDENCE-6K

GeoEvidence-6K pairs UHR remote-sensing questions with manually verified target-region annotations, and HF-SR uses accumulated human corrections to refine reusable annotation skills. The dataset covers varied image sources and local-evidence and global-context tasks.

3.1 DATASET OVERVIEW

  • The dataset draws on 1,050 images from five public sources, averaging 10,714 × 10,714 pixels, while target regions average 1,095 × 1,074 pixels.Target regions occupy roughly 1% of full-image area; image GSDs range from 0.08 m to 0.50 m.Source: §3.1 · p. 4

3 GEOEVIDENCE-6K

  • GeoEvidence-6K contains 6,750 samples across seven task categories, with five local-evidence tasks and two global-context tasks.The local tasks are counting, position, color, category, and shape; land-use and route-planning are global-context tasks.Source: §3.1 · p. 4

3.2 HF-SR: HUMAN FEEDBACK-GUIDED SKILL REFINEMENT

  • HF-SR refines reusable annotation skills by collecting human feedback on agent outputs and summarizing recurring errors and corrections into procedural rules.The agent first inspects a downsampled overview, then candidate regions at native resolution; human reviewers inspect images, regions, and annotations together.Source: §3.2 · p. 4
  • HF-SR schedules more frequent refinements during cold start and lengthens intervals as annotation skills stabilize to amortize refinement cost.The checkpoints follow cumulative Fibonacci intervals.Source: §3.2 · p. 4

4 PRELIMINARY

On-policy distillation supervises student-generated trajectories using teacher distributions at states the student actually visits. On-policy self-distillation gives the teacher privileged training information while retaining standard input for the student at inference.

4 PRELIMINARY

  • On-policy distillation samples output trajectories from the student, then compares teacher and student next-token distributions at student-visited prefixes.This uses states visited by the current student policy rather than off-policy states.Source: §4 · p. 5
  • The objective minimizes reverse KL divergence along the student trajectory, reducing train–inference distribution mismatch by supervising states the student actually visits.The passage describes this as a contrast with off-policy distillation.Source: §4 · p. 5
  • On-policy self-distillation conditions the student on standard input and the teacher on additional privileged information available only during training.The privileged teacher evaluates the same student-generated trajectory, and its distribution guides the student.Source: §4 · p. 5

5 RS-OPSD

RS-OPSD addresses weak crop-only privilege and imperfect teacher supervision with complementary visual views and reliability- and correctness-based filtering.

5.1 MOTIVATION

  • 45.7 average for LoRA fine-tuning on GeoEvidence-6K exceeded direct crop-privilege OPSD's 45.2, while crop privilege provided only a 6.06-point advantage over full-image input.In the same-model diagnostic, full-image accuracy was 63.85% and evidence-crop accuracy was 69.91%; nearly 30% of training samples were answered incorrectly by the privileged model.Source: §5.1 · p. 6

5.2 CPVP: WHAT SHOULD THE TEACHER SEE?

  • CPVP gives the teacher global, contextual, and fine-grained views, pairing a tight evidence crop with a wider crop that includes surrounding context.The contextual crop keeps the evidence crop's center and doubles its width and height.Source: §5.2 · p. 6

5.3 CAD: WHEN SHOULD THE STUDENT TRUST IT?

  • CAD admits supervision only when the privileged teacher stays aligned with the ground-truth response throughout its probe, then reinforces sampled tokens for correct student answers and suppresses them for incorrect ones.The privileged teacher evaluates the same student-generated trajectory; the student is also regularized toward a frozen reference model receiving the standard input.Source: §5.3 · p. 7

6 EXPERIMENTS

RS-OPSD leads all three UHR remote-sensing VQA benchmarks, with RS-OPD-Lite offering a smaller, faster alternative. Ablations and training analyses examine the framework’s designs and optimization choices.

6.1 EXPERIMENTAL SETTINGS

  • RS-OPSD trains on GeoEvidence-6K for 150 steps with batch size 96 and an EMA teacher; evaluation uses three benchmarks and their official protocols.Baselines span closed- and open-source VLMs, remote-sensing VLMs, and specialized UHR methods.Source: §6.1 · pp. 7–8

6.2 COMPARISON WITH STATE-OF-THE-ART METHODS

  • RS-OPSD scores 53.1 on XLRS-Bench, 61.5 on MME-RealWorld-RS, and 33.3 on LRS-VQA, averaging 49.3 and beating the strongest rival by 4.0 points.The comparison table reports quantitative results across the benchmarks.Source: Table 2 · p. 8, §6.2 · p. 8
  • RS-OPD-Lite averages 44.2 with a 2B student, surpasses most 8B-scale models, and runs at 1.09 s/sample, 12.8% faster than the second-fastest method.RS-OPSD runs at 1.58 s/sample, faster than the Qwen3-VL-8B baseline.Source: §6.2 · p. 8

6.3 ABLATION STUDIES AND ANALYSIS

  • LoRA raises the average from 40.8 to 45.7, crop-only OPSD reaches 45.2, and CPVP, CAD, and their combination reach 47.2, 47.0, and 49.3.The ablation table evaluates the core RS-OPSD designs.Source: Table 3 · p. 9, §6.3 · p. 9
  • With λKL=10^-3, cgrad=5 yields 49.3 versus 48.3 at cgrad=20 or 100; λKL=10^-3 achieves 49.0 in the KL-coefficient analysis.The authors use λKL=10^-3 and cgrad=5 in the main experiments.Source: Table 4 · p. 9, §6.3 · p. 9
  • Across three epochs, student and EMA-teacher accuracy rise as their gap narrows, while nearly all task categories improve during training.Suppressive token activation decreases, and reinforced and overall active tokens drop sharply in the third epoch.Source: Figure 5 · p. 9, §6.3 · pp. 9–10

7 CONCLUSION

RS-OPSD internalizes zoom-in visual privilege during training, supported by GeoEvidence-6K and HF-SR, and combines CPVP with CAD. Its training procedure filters privileged guidance, while RS-OPD-Lite uses a smaller student with a frozen teacher.

7 CONCLUSION

  • RS-OPSD internalizes zoom-in privilege during training and combines GeoEvidence-6K, HF-SR, CPVP, and CAD to achieve SOTA performance with efficient inference.The conclusion also reports a favorable accuracy–efficiency trade-off for RS-OPD-Lite.Source: §7 · p. 10

A ALGORITHM OF RS-OPSD

  • For each sample, the student rolls out from Global Evidence; the CPVP teacher evaluates that trajectory, and CAD filters supervision by teacher reliability and student-answer correctness.The student is optimized with the CAD objective and reference-model regularization, while the teacher is updated by EMA.Source: A ALGORITHM OF RS-OPSD · p. 12
  • RS-OPD-Lite uses a Qwen3-VL-2B student with a Qwen3-VL-8B privileged teacher that remains frozen during training.It follows the same training procedure otherwise.Source: A ALGORITHM OF RS-OPSD · p. 12

B DATASET QUALITY AND INTEGRITY

GeoEvidence-6K has no exact image-level overlap with the three evaluation benchmarks and reports high annotation acceptance. Its question patterns and task categories span local-evidence and global-context work.

B DATASET QUALITY AND INTEGRITY

  • GeoEvidence-6K has no exact image-level overlap with XLRS-Bench, MME-RealWorld-RS, or LRS-VQA; its final cross-validation acceptance rate is 97.8%.Ten domain experts contributed, and all samples were manually reviewed before final cross-validation.Source: B DATASET QUALITY AND INTEGRITY · pp. 12–13
  • Question vocabulary includes attribute, counting, spatial-localization, and geometric-reasoning terms, while question lengths have mean 56.87, median 54, and 95th percentile 84 tokens.Figure 6 summarizes question vocabulary and token-length distribution.Source: Figure 6 · p. 13, B DATASET QUALITY AND INTEGRITY · p. 13
  • Five local-evidence categories each comprise about 15.6% of the dataset, while route planning and land use each comprise about 11.1%.The authors describe this composition as balanced supervision across local perception and global scene understanding.Source: Figure 6 · p. 13, B DATASET QUALITY AND INTEGRITY · p. 13

C IMPLEMENTATION DETAILS

Training uses one on-policy student rollout per sample and complementary teacher views. Visual preprocessing preserves aspect ratio, while inference requires no extra search or tool calls.

C IMPLEMENTATION DETAILS

  • Each training sample uses one student on-policy rollout, reused for privileged-teacher evaluation, CAD computation, and reference-model regularization.Training is implemented with the verl framework.Source: C IMPLEMENTATION DETAILS · p. 13
  • Training images are resized with aspect ratio preserved and a 2,048-pixel longest-side cap; annotated samples give the student a boxed Global Evidence image and the teacher contextual and fine-grained views.Evaluation uses original unmarked MME-RealWorld-RS and LRS-VQA images; some XLRS-Bench images natively include red circles.Source: C IMPLEMENTATION DETAILS · pp. 13–14
  • The student prompt uses one full image and the question, while the privileged teacher receives Global, Contextual, and Fine-grained Evidence views with the same question and options.Figure 7 summarizes the prompt templates; prompts use the Qwen chat template.Source: Figure 7 · p. 14, C IMPLEMENTATION DETAILS · p. 14
  • Latency is measured on one NVIDIA H100 at batch size 1, and RS-OPSD remains compatible with vLLM without specialized serving infrastructure.The model adds no architectural modifications, visual search, or external tool calls at inference time.Source: C IMPLEMENTATION DETAILS · p. 14

D DETAILED QUANTITATIVE COMPARISON RESULTS

RS-OPSD leads the reported comparisons across the three UHR remote-sensing VQA benchmarks, with gains spanning perception and reasoning. RS-OPD-Lite combines competitive accuracy with the fastest measured inference speed.

XLRS-Bench

MME-RealWorld-RS and LRS-VQA

  • 61.5 on MME-RealWorld-RS exceeds ZoomSearch by 3.9 points; RS-OPSD’s 33.3 LRS-VQA average exceeds WeaveEarth by 2.0 points.It leads on MME-RealWorld-RS Position (76.8) and Color (70.4), and achieves the highest LRS-VQA average despite not leading every subset.Source: D DETAILED QUANTITATIVE COMPARISON… · p. 15

Overall accuracy and efficiency

  • 49.3 average across three benchmarks gives RS-OPSD a 4.0-point lead over the strongest competing method.RS-OPD-Lite averages 44.2 with a 2B student and records the lowest average inference time, 1.09 s/sample.Source: D DETAILED QUANTITATIVE COMPARISON… · p. 15

E LIMITATIONS AND FUTURE WORK

RS-OPSD depends on explicit target-region and answer supervision, whose annotation cost limits scalability. The authors propose self-supervised extensions and identify larger-teacher-to-smaller-student transfer as a further direction.

Limitations

  • Explicit target-region annotations for CPVP and ground-truth answers for CAD impose substantial annotation cost and limit scalability.The answer annotations assess teacher reliability and student-rollout correctness.Source: E LIMITATIONS AND FUTURE WORK · p. 15

Future work

  • Future work may explore automatic discovery of question-relevant visual privilege and teacher-reliability estimation without ground-truth answers.The authors propose combining model-intrinsic evidence discovery with self-verification to scale toward larger unlabeled UHR datasets and broader high-resolution vision-language tasks.Source: E LIMITATIONS AND FUTURE WORK · p. 16
  • RS-OPD-Lite suggests visual privilege may also benefit distillation from a larger teacher to a smaller student.The proposed direction retains a lightweight student at inference time.Source: E LIMITATIONS AND FUTURE WORK · p. 16

Related papers

Loading 2609.38072v2…