When Generated Images Look Right and Retrieve Wrong: Coverage-Guided Cross-Scale Re-Indexing for Knowledge-Faithful Generative Perception

Guangyuan Dong; Chuang Liu; Yangchen Zeng; Haoyu Wang; Xiaoyang Yu; Pinlong Zhao; Yuchao Hou; Ziwei Li; Zheng Lin

Source-linked AI summary

When Generated Images Look Right and Retrieve Wrong: Coverage-Guided Cross-Scale Re-Indexing for Knowledge-Faithful Generative Perception

Guangyuan Dong, Chuang Liu, Yangchen Zeng, Haoyu Wang, Xiaoyang Yu, Pinlong Zhao, Yuchao Hou, Ziwei Li, Zheng Lin

arXiv:2608.20810v1Aug 21, 2026cs.MMcs.AIcs.CVcs.GR

TL;DR

Single-caption conditioning can lose scale-specific concepts, making generated images difficult to retrieve even when pixel fidelity is high. CERES builds a multi-scale semantic index, routes concepts by scale, and verifies coverage by re-indexing with a frozen VLM; it achieves state-of-the-art pansharpening results and stronger concept-query retrieval, with stated theory and evaluation scope boundaries.

Problem
Existing VLM-conditioned generators collapse multi-scale scenes into one globally pooled embedding, risking loss of scale-specific concepts and weakening re-indexability despite pixel fidelity.
Method
CERES uses a three-level semantic pyramid, co-occurrence-aware routing, scale-routed U-Net generation, frozen-VLM re-indexing, soft-Jaccard coverage, and independent DINOv2 verification.
Results
CERES dominates larger baselines on pansharpening and downstream perception, while improving concept-query retrieval Recall@5 by +14 pp and MRR by +0.19 over the strongest baseline.
Takeaways & Limitations
The results support treating generated visual content as queryable knowledge and optimizing the indexing topology rather than only increasing generator size.
Takeaways & Limitations
Coverage gains concentrate in spectrally ambiguous classes and sub-patch ultra-fine structures below the SigLIP-2 token grid, while the theory provides sufficient-condition sanity checks rather than performance guarantees.

Abstract

from arXiv · show

Multimodal information systems increasingly route generated visual content back through the same vision-language index that informed its production, so the output must remain retrievable by the queries it was meant to serve. When the scene contains entities at vastly different scales, existing language-guided generators condition on a single, globally pooled text embedding and quietly drop scale-specific concepts, breaking concept-query retrieval even when pixel fidelity is high. We formalise this failure as semantic collapse and propose CERES, a closed-loop multimodal indexing framework that builds a three-level semantic pyramid, mines implicit concepts via a co-occurrence-aware router, performs scale-routed cross-attention into a lightweight U-Net generator, and verifies coverage by re-indexing the generated image with the same frozen VLM. A continuously differentiable soft-Jaccard coverage objective returns dense gradients to the 0.39M-parameter generator under explicit non-degeneracy conditions, and coverage is verified by an independent DINOv2 linear probe trained only on external scene and object labels. On four pansharpening benchmarks across seven settings, CERES delivers the new state of the art with the largest gains where scale variation is most extreme. It also improves concept-query retrieval Recall@5 by +14.0 points and image-text mean reciprocal rank by 0.19 over the strongest baseline, showing that the closed loop preserves queryable content rather than self-referential feature consistency.

1 Introduction

CERES frames semantic collapse as the loss of scale-specific concepts during generation and addresses it with a closed-loop, multi-scale indexing and coverage-verification framework.

Motivation: Existing VLM-conditioned generators use one globally pooled embedding, which can discard microscopic concepts in hierarchically composed scenes despite high pixel fidelity.The paper names this measurable failure semantic collapse.
Approach: CERES replaces single-scale conditioning with a three-level semantic pyramid and a co-occurrence-aware router for explicit and implicit concepts.The index represents local, meso-scale, and global content before generation.
Approach: Scale-routed cross-attention and FiLM modulation inject concepts into decoder layers matched to their spatial scales.This implements the paper’s second required ingredient: routing concepts to corresponding resolutions.
Closed-loop verification: The generated image is re-encoded by the same frozen VLM, and a soft-Jaccard verifier supplies differentiable coverage feedback to the trainable generator.The framework also uses an independent DINOv2 probe trained on external labels to verify coverage beyond the optimization target.
Contributions: CERES combines state-of-the-art fidelity with stronger concept-query retrieval and downstream-perception gains across four pansharpening benchmarks and three perception modes.The contribution statement also reports evaluation with a 60-query retrieval bank and an independent label-grounded probe.

2 Related Work

The related work spans multimodal indexing, dense VLM prediction, fine-grained grounding, knowledge-aware generation, closed-loop adaptation, and remote-sensing generative perception.

Multimodal indexing and cross-modal retrieval: Cross-modal retrieval methods index visual regions and textual tokens into shared semantic spaces using attention, graph reasoning, and semantic aggregation.Recent work also studies semantic capacity balancing and personalized dense retrieval.
VLMs for low-level dense prediction: Dense VLM prediction methods extend image-level matching toward pixel-text matching, dense labeling, feature-invariant restoration, and universal restoration.The cited methods include DenseCLIP, MaskCLIP, CLIPDenoising, and DA-CLIP.
Fine-grained multimodal grounding: Fine-grained grounding research improves spatial reasoning, local geometry alignment, visual imagery, and object-based control.These directions connect semantic intent with spatial structure across 3D understanding and long-horizon action.
Knowledge-aware multimodal generation: Knowledge-aware generation injects structured concepts through region-phrase matching, concept graphs, feature modulation, latent diffusion, and instruction-tuned editing.FiLM and SPADE are cited as mechanisms for modulating dense features with external semantics.
Closed-loop adaptation and memory: Closed-loop adaptation and memory methods reuse intermediate states, execution traces, or verifiable rewards as feedback for long-horizon systems.The related work includes contribution-aware memory, memory-quality optimization, attribution, and reinforcement learning with verifiable rewards.
Generative perception in remote sensing: Remote-sensing generative perception provides the pansharpening setting, where methods address extreme intra-class and inter-class scale variation.The cited approaches include detail injection, transformer fusion, spatial-spectral attention, non-local convolution, and diffusion-based methods.

3 The CERES Framework

CERES builds a differentiable, multi-scale semantic index, routes its units through a resolution-aware U-Net, and closes the loop by re-indexing generated images for coverage verification.

3.1 Problem Formulation: CERES represents scenes with global, meso, and local texts plus K scale-tagged semantic units.Semantic collapse is defined as low coverage despite bounded pixel reconstruction error.
3.3 Stage I: Cross-Scale Semantic Indexing: The explicit index uses soft concept membership and aggregates dense features through a differentiable pseudo-RGB projection.The projection maps the multispectral input into a proxy used by the frozen encoder.
Optimization: SigLIP-2 remains frozen while only the generator and lightweight projection or modulation components are trained.The trainable portion contains 0.39 M parameters.
3.3 Stage I: Cross-Scale Semantic Indexing: The implicit branch mines compound concepts from spatial co-occurrence, stops routing under a decreasing cumulative threshold, and merges or filters candidate units.For r>3, a second-order Markov factorization reduces storage from O(KR) to O(K3) at 0.94 unit-Jaccard accuracy versus exact r=3 enumeration.
3.4 Stage II: Scale-Routed Generation: Scale tags route local, meso, and global units to progressively lower-resolution decoder layers in a four-level 128→64→32→16 U-Net.Resolution-matched injection is intended to prevent global units from dominating high-resolution layers.
3.5 Stages III & IV: Re-Indexing and Soft Coverage: The generated image is re-encoded by the frozen VLM, and recovered mask statistics are converted into a semantic pyramid using fixed typed prompt templates.The same index is therefore reused after generation rather than replaced by a separate evaluator.
3.5 Stages III & IV: Re-Indexing and Soft Coverage: Soft-Jaccard coverage compares input and recovered concept presence scores and remains differentiable with non-zero gradient at non-degenerate scores.An independent DINOv2 probe trained on external labels is used to test whether coverage gains are circular.

4 Theory: Sufficient Conditions

The theory gives sufficient conditions for a non-vanishing closed-loop coverage gradient and bounds its connection to spectral distortion, while explicitly treating these propositions as sanity checks rather than performance guarantees.

Proposition 1 establishes sufficient conditions for a non-vanishing closed-loop gradient under Assumption 1.The assumptions include encoder regularity, a non-degenerate generator Jacobian, non-saturated recovered presence, and a bounded projection operator.
Under Assumption 1, the coverage-loss chain rule passes through the soft-max, sigmoid response map, frozen dense encoder, and generator.The resulting lower bound depends on L, M, σmin, the Jaccard denominator, and the response-map temperature.
Without the non-degeneracy conditions, saturation, a rank-deficient generator step, or a degenerate Jaccard denominator can collapse the bound.On WV-III validation, the conditions held on 96.7% of training steps; the remaining steps were carried by Lrec.
Proposition 2 links semantic coverage to spectral angular distortion under assumptions on the centroid system and pseudo-RGB projection.The associated numerical illustration gives 5.18° from the bound and 2.75° observed on WV-III, showing that the linkage is an order-of-magnitude bound rather than a worst-case guarantee.
The propositions are sanity checks rather than performance guarantees, and Proposition 1 motivates delaying Lcov until reconstruction warm-up.The non-saturation condition fails for 41.3% of concepts at initialization but for fewer than 5% after warm-up, so Lcov activates at epoch 200.
The router has a finite termination guarantee of at most Rmax=min(N,⌈log ϵ/log maxj αj⌉) steps.For maxj αj=0.7 and ϵ=10^-3, the bound is Rmax≤19.

5 Experiments

CERES is evaluated across pansharpening quality, downstream perception, concept-query retrieval, and ablations, with additional calibration and failure-mode analyses. Across these tests, it improves both image quality and the preservation of queryable semantic content.

Main Pansharpening Results: CERES reaches the new state of the art on every reduced-scale metric and all but one full-scale metric across seven pansharpening settings.Paired Wilcoxon tests report p<0.05 versus the second-best baseline.
Main Pansharpening Results: +4.64% Q2n on the unseen WV-II sensor is CERES’s largest relative gain, with cross-sensor transfer gains of +3.33% and +5.38% / +3.24%.The WV-II result is 0.879 vs. 0.840; the additional values correspond to reverse WV-II→WV-III and heterogeneous 4-band↔8-band transfers.
Downstream Perception: CERES improves every downstream perception cell, while DINOv2-measured coverage rises from 0.63 to 0.85 and small-object AP gains reach +13.6 to +16.7.The cross-encoder correlation between CDINO and DOTA mAP is ρ=0.83.
Ablations: Ablations show that soft-Jaccard coverage adds +0.006 Q2n and +0.015 HQNR, while removing the presence gate drops HQNR by 0.051.Explicit-only and implicit-only configurations each lose 0.014–0.016 Q2n, indicating that both branches contribute.
Coverage Calibration: Coverage calibration is stable across thresholds and templates, but disagreement concentrates in spectrally ambiguous classes and sub-patch ultra-fine structures below the SigLIP-2 token grid.Both probes rank CERES first on every WV-III tile, while the cross-encoder Spearman correlation is 0.91.

6 Discussion

CERES’s coverage loop captures information that consistency-only and pixel-cycle alternatives miss, while adding measurable training-time computation but no extra learnable verifier.

The coverage signal captures information that neither soft consistency nor pixel-level cycle losses recover.
25.7 G FLOPs and 25.5 ms per 64×64 PAN input quantify the full closed-loop training cost on an RTX 4090.Generation-only deployment costs 13.6 G FLOPs and 16.3 ms because coverage verification is optional at inference.
The 0.39 M-parameter trainable budget is two orders of magnitude smaller than the strongest learning baseline.
The recipe extends to product catalogues and medical imaging pipelines, requiring domain-specific refitting only for centroid construction.

7 Conclusion

The paper frames semantic collapse as loss of re-indexability and addresses it with CERES, whose frozen-index closed loop improves pansharpening and downstream perception across scales.

Semantic collapse is the silent loss of scale-specific indexed concepts under single-caption conditioning, formalised as a measurable re-indexability failure.
CERES pairs a soft, differentiable cross-scale concept index with a soft-Jaccard coverage verifier and an external label-grounded DINOv2 probe.
+14 pp Recall@5 and +0.19 MRR over the strongest baseline quantify the concept-retrieval improvement.
+16.7 AP on small vehicles is the largest small-object detection gain, where single-caption baselines drop concepts.
The 0.39 M-parameter generator remains closed against a frozen SigLIP-2 index and dominates 11× larger baselines on pansharpening and downstream perception.

A Detailed Architecture and Training Loop

CERES trains a four-stage pipeline that builds explicit and implicit scale-tagged semantic units, routes them through a U-Net, re-indexes outputs, and optimizes coverage with staged losses.

Training loop: CERES chains a frozen SigLIP-2 encoder, trainable U-Net generator, reused frozen encoder, and soft-Jaccard verifier across four stages.
Training loop: The training loop requires multispectral input, panchromatic input, ground truth, and a frozen VLM, then applies warm-up scheduling before coverage and semantic-alignment losses.
Stage I: semantic indexing: Stage I combines differentiable soft assignment to learned concept centroids with a co-occurrence-aware implicit router for scale-tagged units.
Stage II: generation: The four-level U-Net injects scale-routed units at decoder levels through cross-scale attention followed by FiLM modulation.
Worked example: For aircraft, the system routes a scale-1 unit to the 128×128 decoder, recovers presence 0.86, and ranks the query tile first versus seventh for the strongest open-loop baseline.The aircraft unit has confidence 0.88, attention 0.23 on aircraft patches versus 0.03 on background patches, and FiLM parameters (1.08, −0.02).

B.2 Proof of Proposition 2

The proposition links semantic coverage to spectral angular distortion under separability and smoothness assumptions, with a non-vacuous numerical bound but weaker behavior on cross-sensor data.

The proof bounds spectral angular distortion by combining centroid separability, prediction coverage, and Lipschitz continuity of the dense encoder.
With L=1.4, M=1.0, d=1024, K=200, and ε_intra=0.005 rad, the bound is SAM ≤5.18° while observed WV-III SAM is 2.75°.
Linear separability holds on 98.4% of held-out tiles, but the larger WV-II residual ε_intra makes the bound correspondingly looser.
The router terminates when r ≥ log ε / log max_j α_j; for α=0.7 and ε=10^-3, R_max≤19.

C.1 Bidirectional cross-sensor Q2n

CERES improves cross-sensor pansharpening and oriented-object detection, with especially strong gains under heterogeneous sensor transfer and for small objects. Additional analyses show sensitivity to vocabulary size, routing depth, thresholds, and warm-up design.

+3.24% to +5.38% Q2n across all four bidirectional cross-sensor directions, with the largest gain on heterogeneous 4-band→8-band transfer.
CERES improves every DOTA object class, with gains of +13.6 to +16.7 AP for small classes, +8.1 to +9.3 for medium classes, and +5.3 to +5.6 for large classes.
CERES provides the largest independent DINOv2 coverage gain among the compared open-loop adaptations.
Implicit relational concepts remain harder to recover than explicit concepts, while CERES raises WV-III recovery from 0.76/0.45 to 0.91/0.79 for explicit/implicit subsets.
Q2n improves rapidly from K=50 to K=100, plateaus through K=250, and degrades beyond K>300 as the centroid vocabulary overfragments.
Routing depth saturates at R=3 on 8-band datasets and R=2 on 4-band datasets, while the Markov approximation reduces storage from 1.4 TB to 32 MB with 0.94 unit-Jaccard similarity at depth 3.
Delaying coverage-loss activation to Phase 3 yields 0.924 Q2n, whereas activating it from epoch 0 lowers final Q2n to 0.917 and causes unstable early training.

D.6 Joint factorial ablation of the injection mechanism

The factorial and language-pathway interventions identify complementary roles for spatial attention, FiLM modulation, presence gating, and text anchors. Together, the results support the closed-loop design and show robustness across seeds, templates, and centroid-pool choices.

+0.053 mean Q2n for the attention×FiLM interaction shows that the two injection mechanisms are strongly super-additive.Either mechanism alone falls below the no-injection baseline.
+0.026 mean HQNR is the presence gate’s main effect, consistent with suppressing rewards for absent concepts.
+0.016 HQNR for the three-way attention×FiLM×gate interaction indicates that gating is most valuable when both injection mechanisms are active.
Removing, coarsening, or mismatching language–concept correspondences substantially reduces local and meso retrieval and coverage while changing global-query performance only modestly.Shuffled anchors yield lower local Recall@5 than zeroed anchors, 0.45 versus 0.52.
Verifier-template expansion from 4 to 36 templates raises CDINO from 0.82 to 0.85 and mean Recall@5 from 0.69 to 0.73, then plateaus at 12 templates.
Across ten k-means restarts, WV-III Q2n varies by only 0.924 ± 0.001, while biased centroid pools selectively reduce Recall@5 for concepts absent from the pool.Absent concepts lose −0.09 Recall@5 on average, compared with −0.01 for concepts present in the pool.

E Prompt Templates, DOTA Protocol, Reproducibility

The appendix documents a fixed verifier, source-image-disjoint DOTA evaluation, external coverage probes, shared hyperparameters, compute costs, multi-scale activations, and a small cross-domain pathology pilot.

E.1 Prompt template family: The verifier maps scale-specific mask statistics to 12 parameter-free typed prompt templates shared across datasets and sensors.
E.1 Prompt template family: Replacing the 12 templates with an LLM-generated 36-template superset changes CDINO by less than 0.01, indicating robustness to prompt phrasing.
E.2 DOTA tile protocol: DOTA uses 20,000 overlapping tiles split by source image into 16,000 training and 4,000 test tiles, preventing leakage through spatial overlap.
E.3 Coverage probe construction: The independent DINOv2 linear probe is trained with external labels and frozen during downstream evaluation.
E.4 Hyperparameters: The same hyperparameter configuration is shared across all four datasets and three seeds.
E.5 Per-module compute cost: Training uses 11.4 GB peak memory, while generation-only deployment requires 1.9 GB and coverage certification adds one encoder pass with 0.6 GB transiently.
E.6 Unit activations: Unit supports scale from median areas of 0.6% to 24.7%, and the recovered-mask residual is the signal minimized by the coverage loss.
F Cross-Domain Pilot: Histopathology Super-Resolution: +13 pp mean Recall@5 in the small histopathology pilot accompanies coverage rising from 0.70 to 0.83, but the single-organ-site study does not claim clinical validity.