Source-linked AI summary
Collapse of Patches: Ranking Image Patches for Efficient Visual Modeling
Wei Guo, Shunqi Mao, Zhuonan Liang, Xuanhua Yin, Heng Wang, Weidong Cai
TL;DR
Masked image modeling often treats patches as uniformly correlated, despite their differing roles in reducing image uncertainty. The paper learns patch collapse orders and reports benefits for generation and classification, including a ViT maintaining high accuracy while processing 22% of image content.
Problem
Many masked image modeling methods randomly mask patches with uniform probability, overlooking their nonuniform perceptual importance.
Source: §1 · p. 1Method
CoMAE learns target-patch dependencies through selective masking during reconstruction, then PageRank on a weighted dependency graph produces collapse orders.
Source: §1 · p. 1Results
Collapse-order supervision benefits autoregressive generation and masked classification; a ViT maintains high accuracy while processing just 22% of image content.
Source: §1 · p. 2Takeaways & Limitations
Patch collapse offers a modeling perspective for efficient visual systems across image generation and classification.
Source: §1 · p. 2, §5 · p. 7The fixed-size patch representation does not reflect object saliency; the authors leave salient local features for future exploration.
Source: §5 · p. 10
Abstract
from arXiv · showhide
Observing certain patches in an image reduces the uncertainty of others. Their realization lowers the distribution entropy of each remaining patch feature, analogous to collapsing a particle's wave function in quantum mechanics. This phenomenon can intuitively be called patch collapse. To identify which patches are most relied on during a target region's collapse, we learn an autoencoder that softly selects a subset of informative patches during reconstruction. Graphing these learned dependencies for each patch's PageRank score reveals the optimal patch order to realize an image. We show that respecting this order benefits various masked image modeling methods. First, autoregressive image generation can be boosted by finetuning with the ordered generation sequence. Second, we introduce a new setup for image classification by exposing Vision Transformers only to high-rank patches in the collapse order. Seeing 22% of such patches is sufficient to achieve high accuracy. With these experiments, we propose patch collapse as a novel image modeling perspective that promotes vision efficiency.
1 Introduction
Patch collapse treats image understanding and generation as sequential reductions in patch uncertainty; the paper learns collapse orders and applies them to generation and classification.
1 Introduction
- Different patches reduce residual-image uncertainty with different effectiveness, so a rooster’s beak can constrain the remaining image more than a background field.The paper connects this nonuniformity to efficient visual processing through quick scans of important image regions.Source: §1 · p. 1
- Many masked image modeling methods randomly mask patches with uniform probability, overlooking their nonuniform perceptual importance.The introduction contrasts this assumption with biological visual systems’ preferential attention to salient locations.Source: §1 · p. 1
- CoMAE selectively masks patches during reconstruction, learning continuous weights that identify which patches contribute most to each target patch’s collapse.The selected subsets vary across target patches, indicating distinct collapse dependencies.Source: §1 · pp. 1–2
- PageRank on a weighted patch-dependency graph yields collapse orders that rank patches by collapse independence.Visualizations show high-rank patches outlining important image shapes, while ImageNet inter-class collapse orders exhibit moderate similarity.Source: §1 · p. 2
- 22% of image content lets a ViT maintain high accuracy, while collapse-order supervision benefits autoregressive generation and masked classification.The paper reports improved sample quality for autoregressive generation and high accuracy for classification using prioritized patches.Source: §1 · p. 2
2 Related Works
- Prior stochastic MIM assumes uniform patch correlations, while adaptive MIM targets salience without explicitly modeling uncertainty reduction across patches.CoMAE uses adaptive masking to model patch dependencies and derive global patch orders.Source: §2 · p. 2
- Collapse order offers a data-salient generation strategy that can integrate with next-x methods and operate on image patches independently of the model architecture.The paper suggests combining image-space pruning with token-pruning methods.Source: §2 · p. 3
3 Method
The method infers patch dependencies from reconstruction and converts them into a PageRank collapse order. It then uses that order to guide autoregressive generation and selective-input ViT classification.
3 Method
- The method estimates patch dependencies, ranks patches with PageRank, then uses that order to guide autoregressive generation and masked ViT classification.The resulting ranking is defined as the patch realization sequence for reducing image uncertainty.Source: §3 · p. 3
3.1 Collapse Masked Autoencoder
- CoMAE learns target-specific soft patch-selection weights, injects more noise into less-selected patches, and uses contrastive regularization to diversify masks and reduce reconstruction loss.Without the contrastive loss, masks remain similar and reconstruction loss plateaus higher.Source: §3.1 · pp. 3–4
3.2 Patch Ranking
- CoMAE selection weights form a patch dependency graph, and PageRank scores rank patches by collapse independence.Higher PageRank patches have greater influence on other patches.Source: §3.2 · p. 4
3.3 Collapsing Autoregressive Image Generation
- CMAR retrains the stochastic MAR generator with CoMAE-derived collapse-order supervision, retaining 10% random-rank sequences to reduce overfitting risk.For each sample, it learns to generate randomly masked low-rank patches from high-rank patches.Source: §3.3 · p. 4
3.4 Collapsing Image Classification
- CViT trains on ImageNet with low-rank patches masked at rates from 0–99% and drops masked patches from the input sequence.A cosine annealing schedule favors lower mask rates.Source: §3.4 · p. 4
4 Experiments and Results
Experiments test CoMAE’s learned masks and collapse-ordered generation and classification on ImageNet. Results support polarized, target-specific patch dependencies and benefits for generation and sparse-patch classification.
4 Experiments and Results
- Experiments use center-cropped 256×256 ImageNet-1k images; generation is limited to MAR-B, while classification fine-tunes pretrained ViT-Base.The MAR-B restriction reflects the authors’ computational resources.Source: §4 · p. 5
4.1 Properties of CoMAE
- As reconstruction loss converges, CoMAE masks polarize rather than remain uniform, indicating a subset of patches drives each target’s collapse.Lower mask entropy reflects more polarized selection.Source: §4.1 · p. 5
- Different target patches rely on different subsets, and collapse-order visualizations rank patches outlining major objects highly.The observed rankings align with human visual scanning order.Source: §4.1 · p. 5
4.2 Benchmarking CMAR
- CMAR gains 4% in tFID with 0.3% IS degradation, while CViT leads in almost all classification metrics.The comparisons cover collapse-order generation and classification across mask rates.Source: Table 2 · p. 6, Table 5 · p. 7, §4.2 · p. 7
- CMAR samples look slightly more realistic than MAR+C, while baseline MAR shows object-composition defects or content confusion.Figure 5 frames MAR in purple, MAR+C in blue, and CMAR in green.Source: Figure 5 · p. 6, §4.2 · p. 7
4.3 Properties of CMAR
- CMAR uses CFG scale 3.0 rather than MAR’s 2.9; lower CFG favors lower FIDs, while higher CFG increases IS at the expense of FID.The authors attribute CMAR’s setting shift to stronger reliance on class-label conditioning.Source: §4.3 · p. 7
- Descending collapse-rank generation performs significantly better than ascending-rank generation.Table 4 identifies descending collapse ranks as CMAR’s best synthesis order.Source: Table 4 · p. 6, §4.3 · p. 7
4.5 Properties of CViT
- At 78% patches omitted, CViT accuracy is 70.6%; it also achieves the highest unmasked accuracy and exceeds RViT at each masking level.The reported 78% omission point is where CViT accuracy drops.Source: §4.5 · p. 7
5 Conclusion
The paper models local feature uncertainty reduction as patch collapse and learns collapse orders with CoMAE and PageRank. Respecting those orders benefits autoregressive generation and classification, including high accuracy with only 22% of image patches.
5 Conclusion
- Respecting collapse order benefits masked image modeling in autoregressive image generation and image classification.The paper reports improved FID and IS for MAR and high accuracy for a ViT processing only 22% of image patches.Source: §5 · p. 7
- CoMAE reconstructs a target patch from other tiles, and PageRank on the learned dependency graph identifies a patch ordering for image realization.The paper describes this collapse order as maximizing uncertainty reduction.Source: §5 · p. 7
- For classification, a ViT can maintain high accuracy while seeing only 22% of image patches.Source: §5 · p. 7
- Limited computational resources led the authors to study only one CoMAE variant and leave other variants for future work.Source: §5 · p. 7
1 Proof of Optimal Collapse Ranking
Under a linear influence model and approximate submodularity, the paper derives a patch ranking that minimizes a linearized cumulative-entropy proxy. The ranking scores correspond to PageRank, and descending scores yield an optimal prefix sequence for that objective.
1 Proof of Optimal Collapse Ranking
- Under the stated assumptions, ranking patches by r minimizes the linearized proxy of cumulative conditional entropy at each prefix.The assumptions are a linear influence model and monotone, approximately submodular entropy.Source: §1 · p. 8
- Under submodularity, greedy maximization of marginal gain Δ(j | S) gives a (1 − 1/e)-approximation to minimizing Hc; with the linear model’s assumption Δ(j | S) ≈ rj, descending rj yields an optimal prefix sequence for cumulative entropy minimization.Source: §1 · p. 8
- The proof shows that the ranking vector’s fixed-point equation matches the PageRank formulation for P.The result follows after normalizing β to sum to one.Source: §1 · p. 8
2 Additional Qualitative Results
The qualitative results show similar collapse-order patterns within image classes and across classes, alongside high-fidelity CMAR samples across a broad range of classes.
2 Additional Qualitative Results
- Collapse orders show similar patterns among instances from the same class, with closely aligned patch indices.The heatmaps show these aligned indices as vertical lines.Source: §2 · p. 8
- Inter-class collapse orders also have similar heat-line locations, suggesting moderate consistency across classes and image instances.The authors suggest this points to a common structure behind different images’ realization.Source: §2 · p. 8
- CMAR synthesizes high-fidelity images across a broad range of classes in the additional qualitative samples.Source: §2 · p. 8
3 Model Architectures
CoMAE uses ViT-style encoder and decoder components to reconstruct a target patch from the other patch tokens. The encoder produces selection weights, while the decoder predicts the target patch.
3 Model Architectures
- The encoder drops the target patch, adds RoPE, processes the sequence through 12 self-attention blocks, and maps its [cls] token to selection weights.Its embedding dimension is 256, and tanh keeps each weight between 0 and 1.Source: §3 · p. 8
- The decoder injects Gaussian noise into the original patch tokens, drops the target patch, and uses a [tgt cls] token to reconstruct it.It processes the sequence through 12 self-attention blocks with embedding dimension 64, then predicts the patch from the first [tgt] token.Source: §3 · p. 8
- Both CoMAE’s encoder and decoder follow ViT and pool information into a [cls] token.Source: Figure 8 · p. 10
4 Mask Polarization during Training
During reconstruction training, CoMAE’s masking entropy decreases as it concentrates masking on a small subset of patches that most influence each target patch. It gives those informative patches higher signal-to-noise weights and corrupts less relevant regions more strongly.
4 Mask Polarization during Training
- As reconstruction is optimized, CoMAE concentrates masking on a small set of patches with the greatest influence on each target patch.The model assigns higher signal-to-noise weights to informative patches and stronger corruption to less relevant regions.Source: §4 · p. 10
5 Limitations and Future Improvements
The paper’s patch-based collapse order uses fixed-size regions that do not reflect object saliency, and the authors describe the work as a preliminary feasibility study. Limited computation also restricted model variants and task coverage, while alternative CoMAE designs remain unexplored.
5 Limitations and Future Improvements
- Patch-based collapse orders use fixed-size regions whose shapes do not reflect object saliency.The authors describe the current modeling as a simple, preliminary feasibility study and leave salient local features for future exploration.Source: §5 · p. 10
- Limited computation prevented training more CoMAE, CMAR, and CViT variants in this work.The authors identify deeper CoMAE encoders, longer training for larger CMAR models, and segmentation and detection as future directions.Source: §5 · p. 10
- An alternative CoMAE design using a pretrained autoregressive decoder instead of training a decoder from scratch remains unexplored.The authors also propose applying CoMAE to representation-autoencoder spaces to study representation collapse.Source: §5 · p. 10
6 Ethical Statement
The authors note that high-rank patches have shown greater influence on image synthesis and classification, and suggest both exploiting them for concentrated visual attacks and protecting them to improve robustness.
6 Ethical Statement
- High-rank patches have shown greater influence on synthesis and classification, motivating both concentrated visual attacks and defenses that strengthen model robustness on those patches.The authors state that they will release implementation code and model weights to aid defense design.Source: §6 · p. 10