PaperlayerView exact revision on arXiv

Source-linked AI summary

HeatTok: Enhancing Remote Sensing Image Understanding via Thermodiffusion-based Tokenization

Yingying Yan, Jiaqi Tang, Wei Wei, Qianzhou Wang, Jinjian Wu, Botong Geng, Jianmin Chen, Yuyang Xia, Lei Zhang

arXiv:2608.22485v1cs.CV

TL;DR

Patch-based tokenizers fragment remote-sensing geo-objects, while adaptive methods lack precise object-level tokens and dedicated geometric encoding for irregular regions. HeatTok thermodiffusion-merges homogeneous regions and adds Gaussian positional cues; it reports state-of-the-art performance on VRSBench and EarthVQA under a reasonable token budget.

  • Problem

    Patch-based tokenization mixes semantic classes and fragments irregular geo-objects, while existing adaptive methods lack precise object-level tokens and adequate geometric representations.

  • Method

    HeatTok uses thermodiffusion-guided merging to form semantically complete irregular tokens and G-MRoPE to encode their Gaussian center, scale, and orientation.

  • Results

    HeatTok better preserves object-level semantic integrity and achieves state-of-the-art performance on VRSBench and EarthVQA under a reasonable token budget.

  • Takeaways & Limitations

    Object-aligned irregular tokens combined with explicit Gaussian geometry provide HeatTok’s reported approach to preserving semantic integrity and conveying spatial extent, orientation, and location.

  • Takeaways & Limitations

    The authors acknowledge two main limitations regarding HeatTok’s broader applicability.

Abstract

from arXiv · show

Current visual tokenizers in Multimodal Large Language Models (MLLMs) predominantly rely on patch-based partitioning, which causes severe semantic mixture and object fragmentation in remote sensing imagery due to the irregular contours of geo-objects. Moreover, existing adaptive methods struggle to extract precise object-level tokens and lack dedicated geometric positional encodings for irregular regions. In this paper, we propose HeatTok, a semantic-aware tokenizer driven by thermodiffusion aggregation. Inspired by the physical principles of heat conduction, HeatTok adaptively merges adjacent homogeneous regions to generate semantically independent, object-aligned irregular tokens. To enable MLLMs to perceive these irregular shapes, we design the Gaussian Multimodal Rotary Positional Embedding (G-MRoPE), which models token spatial distributions via 2D Gaussians and explicitly injects center, scale, and orientation cues. Extensive evaluations on the VRSBench and EarthVQA datasets demonstrate that HeatTok effectively preserves object-level semantic integrity and achieves state-of-the-art performance under a reasonable token budget. The code is available: https://github.com/YingyingYan1/HeatTok.

1 Introduction

Remote-sensing MLLMs rely on visual tokenizers whose patch-based partitions misalign with irregular geo-objects, motivating HeatTok’s object-aligned thermodiffusion tokens and Gaussian geometric encoding.

  • MLLM visual tokenizers convert image elements into model-processable token sequences, directly affecting visual comprehension.
  • Fixed square patches mix semantic classes and fragment individual buildings, roads, and other geo-objects across tokens.Their regular grids fail to follow true object contours in remote sensing imagery.
  • Existing adaptive methods improve semantic alignment but remain constrained by rectangular blocks or insufficient geometric representations for irregular tokens.
  • HeatTok merges similar FastSAM region proposals on a region graph to create compact, semantically independent tokens aligned with individual geo-objects.Thermodiffusion aggregation elevates representations from pixel-level to object-level.
  • G-MRoPE models irregular token regions with 2D Gaussians and injects their center, scale, and orientation into positional embeddings.
  • 76.37% accuracy on VRSBench and 78.88% on EarthVQA demonstrate HeatTok’s reported effectiveness under a reasonable token budget.

2 Related Work

Adaptive tokenization research has moved beyond fixed patches through merging, deformable sampling, and irregular-region methods, but geometric encoding for irregular tokens remains incomplete.

  • 2 Related Work: Patch-based tokenization causes semantic fragmentation, semantic mixture, and computational redundancy, motivating adaptive visual tokenizers.
  • Dynamic Grouping and Merging: Dynamic grouping and merging compress or reallocate encoded grid tokens but still rely on regular patch primitives.
  • Deformable Sampling: Deformable sampling learns token positions and scales, while GPSToken uses 2D Gaussians to parameterize token regions.
  • Subobject and Superpixel Tokenizers: Subobject and superpixel tokenizers replace square patches with boundary- or content-aligned irregular regions to better preserve semantic integrity.
  • Subobject and Superpixel Tokenizers: These irregular-region methods typically use implicit positional aggregation or centroids rather than explicit geometric parameterization and encoding.
  • HeatTok addresses the remaining gap by combining thermodiffusion-generated irregular tokens with Gaussian geometric representations.

3 Methodology

HeatTok combines thermodiffusion-guided merging with Gaussian geometric positional encoding to produce semantically complete, irregular tokens aligned with remote-sensing objects. A parallel global-context branch supplements these object-centric tokens with coarse scene layout information.

  • 3.3 Overall Architecture: HeatTok retains a downsampled global-context branch alongside the semantic branch to provide scene-level layout and long-range spatial topology.The two branches are concatenated into a unified visual context for multimodal reasoning.
  • 3.1 Thermodiffusion-Guided Token Merging: HeatTok converts over-segmented FastSAM proposals into semantically complete irregular regions through thermodiffusion-guided merging on a region adjacency graph.Adjacent regions are represented as graph nodes and merged after semantic consistency converges.
  • 3.1 Thermodiffusion-Guided Token Merging: Initial temperatures encode neighborhood color homogeneity, while diffusivity depends on inter-region color similarity and region complexity.Homogeneous regions act as heat sources; boundary regions have temperatures approaching zero, and complex textures diffuse heat faster.
  • 3.1 Thermodiffusion-Guided Token Merging: The merging process yields geometric unions that tightly conform to geo-object boundaries, reducing fragmentation while preserving irregular shapes.Regions are merged when their converged temperatures satisfy the semantic consistency threshold.
  • 3.2 Gaussian Multimodal Rotary Positional Embedding: G-MRoPE models each irregular token with a 2D Gaussian whose parameters represent center, extent, and orientation, then injects these cues into M-RoPE.The Gaussian uses spatial centers, axis standard deviations, and a correlation coefficient governing ellipse orientation.

4 Experiments

Experiments evaluate HeatTok within a unified Qwen2.5-VL framework on VRSBench and EarthVQA, comparing tokenization paradigms, semantic integrity, geometric encoding, and efficiency. HeatTok achieves the best reported benchmark performance while improving object-level representations and spatial understanding.

  • 4.1 Experimental Settings: VRSBench and EarthVQA evaluate remote sensing visual question answering across object attributes, counting, scene understanding, and reasoning, using accuracy, token count, and throughput.
  • 4.1 Experimental Settings: All methods replace Qwen2.5-VL’s Patch Embed tokenizer while keeping the backbone and training protocol unchanged for fair comparison.
  • 4.1 Experimental Settings: HeatTok more accurately preserves continuous irregular boundaries, whereas grid, deformable, and superpixel methods produce fragmentation, blocky contours, or localized cells.
  • 4.2 Main Results: HeatTok achieves 76.37% on VRSBench and 78.88% on EarthVQA, with a competitive token budget and favorable throughput.
  • 4.3 Ablation Study: Thermodiffusion improves Color from 72.06% to 77.17% and Shape from 75.12% to 75.98%, but spatial metrics slightly decline without geometric encoding.
  • 4.3 Ablation Study: G-MRoPE raises Position accuracy from 65.38% to 67.34%, Size from 61.72% to 63.34%, and Direction from 53.24% to 55.35%.

5 Conclusion

The conclusion presents HeatTok as an object-aligned tokenizer for remote sensing MLLMs, combining thermodiffusion aggregation with geometric positional encoding. Experiments on VRSBench and EarthVQA report improved semantic integrity and state-of-the-art performance under a reasonable token budget.

  • HeatTok generates irregular, object-aligned tokens that preserve the semantic integrity of complex geo-objects.
  • G-MRoPE injects Gaussian geometric priors so MLLMs perceive the spatial extent and location of irregular tokens.
  • Experiments on VRSBench and EarthVQA show better object-level semantic integrity and state-of-the-art performance under a reasonable token budget.

A Methodological Details

HeatTok generalizes heat diffusion to an irregular FastSAM region graph, using semantic affinity to smooth features and merge adjacent regions into semantically complete tokens. Its workflow combines graph construction, thermodiffusion updates, threshold-based merging, and empirically selected hyperparameters.

  • Graph Construction: FastSAM generates fine-grained semantic regions that become vertices in a spatial adjacency graph.Edges connect spatially adjacent regions, while each vertex carries a temperature representing its semantic state.
  • Thermodiffusion Process: Forward Euler discretization provides the iterative update used to propagate semantic features across the irregular graph.The method extends classical heat diffusion from a continuous domain to irregular graph structures constructed from region proposals.
  • Region Merging: Adjacent regions are merged when their final temperature difference falls below the merging threshold, producing a semantically complete region set.Algorithm 1 returns the merged regions as the final token set R.
  • Thermodiffusion Process: Thermodiffusion updates each region’s temperature by aggregating neighbor fluxes weighted by inter-region semantic affinity.The graph formulation replaces the continuous Laplacian with neighbor-wise diffusion using edge-specific weights κ_ij.
  • Hyperparameters: The method uses five hyperparameters controlling initialization, diffusion, complexity modulation, and final merging granularity.The reported optimal settings include σ_T = 5.0 and τ_m = 0.03; larger σ_T broadens activation, while larger τ_m produces fewer tokens.

B.2 Ablation Study on Thermodynamic Parameters

The thermodynamic parameter ablations show that overly aggressive diffusion causes under-segmentation, while removing texture-aware modulation damages boundaries. The Optimal configuration provides the best balance between object preservation and region separation.

  • Ablation Setup: Figure 6 compares tokenization results under different hyperparameter configurations.
  • Ablation Findings: Set A’s excessive diffusion, color tolerance, and temperature activation severely under-segmented small distinct objects into the background.The resulting tokens lose fine-grained details around small buildings near the stadium.
  • Ablation Findings: Set B’s removal of the texture complexity constraint caused severe boundary degradation and loss of morphological fidelity.Set B uses α = 0.0, disabling texture-aware diffusivity modulation.
  • Ablation Findings: The Optimal configuration preserved large-object semantic integrity while separating adjacent distinct regions with high boundary adherence.Its parameters were κ_0 = 1.0, σ_C = 5.0, σ_T = 5.0, α = 0.5, and τ_m = 0.03.

C Ablation on Region Proposal Dependency

HeatTok’s gains persist across alternative region proposal generators, indicating that thermodiffusion aggregation—not a specific proposal frontend—drives the main performance improvement. FastSAM with thermodiffusion offers the strongest reported efficiency–accuracy trade-off.

  • Thermodiffusion Ablation: Without thermodiffusion, raw FastSAM proposals reached only 69.84% on VRSBench and 70.60% on EarthVQA.The ablation attributes the drop to semantic fragmentation and incoherent object-level tokens.
  • Thermodiffusion Ablation: Thermodiffusion aggregation produced significant accuracy improvements across all tested proposal generators.This supports the conclusion that the principal performance gain comes from the aggregation module rather than a particular frontend.
  • Proposal Generator Comparison: HeatTok led overall sample-weighted accuracy on both datasets: 76.37% versus 76.28% on VRSBench and 78.88% versus 78.39% on EarthVQA.SAM-H slightly outperformed HeatTok on specific fine-grained subsets but used 624.0M parameters and 1468.4 ms frontend latency.
  • Proposal Generator Comparison: FastSAM combined with thermodiffusion achieved 15.3 ms frontend latency and 92.0M parameters in the reported comparison.SLIC was lighter but had lower boundary precision and 45.2 ms CPU-bound latency.

D Generalization to Fundamental Vision Tasks

HeatTok generalizes beyond VQA to visual grounding and semantic segmentation. Its tokenization produces more accurate object localization and preserves land-cover boundaries more effectively than standard patch tokenization in the reported visualizations.

  • Evaluation Scope: The extended evaluation covers visual grounding and semantic segmentation in addition to VQA.
  • Visual Grounding: On VRSBench visual grounding results, HeatTok produced bounding boxes better aligned with object boundaries than standard Patch Embed tokenization.The reported comparison links this alignment to more precise spatial localization.
  • Semantic Segmentation: On EarthVQA segmentation visualizations, HeatTok produced more accurate boundaries and better preserved land-cover region integrity.The result demonstrates extension to dense prediction tasks.

E Additional Qualitative Results

Additional qualitative evaluations show that HeatTok's object-level semantic integrity supports more accurate multimodal grounding and robust reasoning in remote sensing VQA.

  • E Additional Qualitative Results: HeatTok's qualitative evaluations assess how tokenization strategies affect MLLM reasoning in remote sensing VQA.The cases cover distinct geographical scenarios, including curved park layouts and meandering rivers.
  • E Additional Qualitative Results: The case studies link HeatTok's preserved object-level semantic integrity to accurate multimodal grounding and robust reasoning.
  • E Additional Qualitative Results: The evaluations compare visual question-answering results with corresponding tokenization visualizations across two geographical scenarios.

F Limitations and Future Work

The paper identifies limited generalization beyond remote sensing as a key boundary: HeatTok is motivated by remote sensing imagery, and performance gains on natural-image benchmarks are not presumed.

  • F Limitations and Future Work: HeatTok is fundamentally motivated by remote sensing imagery with extensive homogeneous regions and multi-scale, irregular geometric structures.
  • F Limitations and Future Work: The authors do not presume uniform performance gains across general natural-image benchmarks.Extending the framework to high-resolution natural imagery remains future work.
  • F Limitations and Future Work: Although the formulation is described as task- and domain-agnostic, its broader applicability to high-resolution natural images remains unvalidated.

F.2 Robustness and Failure Cases

Robustness analyses show HeatTok merging fragmented proposals into coherent regions and preserving object structure, while severe haze degrades performance and competing tokenizers can induce VQA errors.

  • F.2 Robustness and Failure Cases: Thermodiffusion corrects over-segmentation by merging fragmented proposals into coherent regions, but performance degrades under severe haze.
  • F.2 Robustness and Failure Cases: HeatTok preserves crisp object boundaries, geometric continuity, and avoids over-segmentation in visual tokenization comparisons.
  • F.2 Robustness and Failure Cases: In a meandering-river case, rigid patch partitioning fragments the river and leads to misclassification as dried-up channels with unrelated construction hallucinations.
  • F.2 Robustness and Failure Cases: Dynamic grouping methods can misclassify water as muddy pools and trigger hallucinated housing upgrades when rectangular primitives fail to follow river trajectories.
  • F.2 Robustness and Failure Cases: Deformable sampling may discard macro-level river continuity, causing failure to perceive water bodies and hallucinated water-supply needs.
  • F.2 Robustness and Failure Cases: Superpixel over-segmentation can fracture homogeneous water regions, producing visual noise and incorrect perceptions such as fragmented muddy water.
  • F.2 Robustness and Failure Cases: HeatTok delineates continuous river and road geometries, enabling correct assessment of clean water and road-improvement needs without hallucinations.
Loading 2608.22485v1…