HiPerViT: A Hierarchical Perceiver-Vision Transformer Architecture for Multi-Scale Texture Recognition

• Texture recognition depends on higher-order spatial statistics, local structure, long-range context, and cross-scale organization rather than object shape alone.Source: §1 · p. 1
• Standard CNN and ViT representations do not expose higher-order co-occurrence statistics as first-class information, while multimodal pretraining can be costly and impractical in specialized domains.Source: §1 · pp. 1–2
• The architecture is motivated by whether texture gains arise from a specific fusion design or from exposing higher-order statistics explicitly.Source: §1 · p. 2
• HiPerViT combines global and local views, compact bilinear co-occurrence statistics, and Perceiver-style latent distillation to integrate statistical and spatial tokens.Source: §1 · p. 2
• HiPerViT achieves competitive or best-reported accuracy across six texture benchmarks while remaining computationally compact.It matches or exceeds transformer baselines across all six datasets under matched protocols.Source: §1 · pp. 2–3
• Performance gains remain stable across fusion orderings and second-order capture depths, supporting explicit statistical availability over a privileged fusion topology.Source: §1 · pp. 2–3

• Texture methods progressed from handcrafted co-occurrence statistics to CNN aggregation, bilinear pooling, multiscale architectures, and transformer variants.Source: §2.1 · p. 3, §2.2 · p. 4, §2.3 · p. 4
• Handcrafted descriptors captured texture regularities but often generalized poorly across scales, lighting conditions, and domains.Source: §2.1 · p. 3
• CNN solutions frequently required task-specific modifications, indicating that their inductive biases were not inherently tuned to texture statistics.Source: §2.2 · p. 4
• Vanilla ViTs provide global attention but develop strong shape priors and often neglect fine-grained texture cues.Source: §2.4 · p. 5
• Multimodal pretraining can improve texture sensitivity but requires large models, substantial computation, and language data.Source: §2.4 · p. 5
• HiPerViT bridges global transformer reasoning and local statistical structure through a multiscale ViT, Perceiver bottleneck, and compact bilinear pooling.Source: §2.3 · p. 4, §2.4 · p. 5

• Early visual processing emphasizes localized orientation- and frequency-selective micro-patterns, providing a biological motivation for multiscale feature extraction.Source: §3 · p. 5
• Mid-level visual areas integrate local filter responses with co-occurrence statistics, curvature, surface properties, and broader contextual cues.Source: §3 · p. 5
• The biological framework treats texture perception as coordinated interaction among specialized modules rather than a monolithic sequence.Source: §3 · p. 6
• Higher cortical processing merges local structure with semantic associations and compresses distributed inputs into compact representations robust to variability.Source: §3 · p. 6
• HiPerViT mirrors hierarchical cortical processing through a multiscale ViT backbone, pairwise co-occurrence coding, and a Perceiver-style latent bottleneck.Source: §3 · p. 6

Statistical Token Injection

• Statistical Token Injection converts compact second-order feature statistics into an explicit token that participates directly in transformer self-attention.Count Sketch bilinear projection approximates pairwise co-occurrences without constructing the full outer-product representation.Source: §4.1 · p. 6

Multi-Scale Feature Extraction

• Global and local views capture complementary macro-structural and fine-grained textural information using a shared-weight ViT backbone.The shared backbone processes a downsampled global image and a high-resolution local crop while reducing memory relative to separate branches.Source: §4.2.1 · p. 8

Second-Order Statistical Branch

• Compact bilinear pooling transforms stage-specific token features into concatenated second-order descriptors that are integrated alongside aggregated multi-scale spatial tokens.The second-order branch approximates outer-product statistics through Tensor Sketch, producing the compact vector ZSRM.Source: §4.2.2 · pp. 8–9, §4.4 · p. 10

Topology Robustness

• Reversing interaction and distillation or deferring SRM integration to late fusion yields equivalent accuracy, indicating robustness to fusion topology.The ablations support the statistical prior, rather than a particular wiring arrangement, as the key design ingredient.Source: §4.2.3 · p. 9, §4.4 · p. 10

Cross-Order Interaction

• Cross-order self-attention lets spatial tokens re-weight second-order correlations while contextualizing the statistical token with localized image structure.This bidirectional interaction makes the statistical signal context-dependent rather than requiring it to be linearly separable alone.Source: §4.5 · p. 11, §4.9 · p. 12

Datasets and Evaluation Protocol

• The evaluation covers six texture benchmarks spanning controlled and in-the-wild conditions, with dataset-specific established splitting protocols.The benchmarks are DTD, FMD, KTH-TIPS2-b, GTOS-Mobile, USPTex, and 1200Tex.Source: §5.1 · p. 12

Comparison with State-of-the-Art

• Published SOTA values are heterogeneous because they come from different architectures, pre-training modalities, and task-specific tuning procedures.The authors therefore frame identical-protocol comparisons as the more rigorous test of architectural claims.Source: §5.2 · p. 13
• HiPerViT establishes new vision-only state-of-the-art on all six benchmarks, with substantial gains on five.Reported gains include +3.05 pp on DTD, +10.48 pp on GTOS-Mobile, and +10.1 pp on 1200Tex; USPTex reaches 100.00%.Source: §5.2 · p. 13, Table 1 · p. 14
• HiPerViT outperforms all alternative transformer architectures across all datasets under identical training protocols.Table 1 separates previously published results from controlled comparisons against transformer baselines.Source: §5.2 · p. 13, Table 1 · p. 14
• Table 1 compares external vision-only SOTA results with transformer models trained under a shared protocol.Its two subtables support side-by-side interpretation of published results and controlled baselines.Source: §5.2 · p. 13, Table 1 · p. 14

Multi-scale inputs

• Combining coarse and fine input scales consistently outperforms single-scale setups, supporting complementary global and local context.Source: §5.3 · p. 14, Figure 2 · p. 15

Hierarchical fusion

• Fusing early- or mid-level features with deeper semantic stages outperforms shallow-only or deep-only combinations.Source: §5.3 · p. 14, Table 2 · p. 15

Component ablation

• The full model outperforms strongest partial variants across controlled texture benchmarks, although USPTex provides limited headroom for complex additions.Source: §5.4 · p. 15, Table 3 · p. 16
• Perceiver-only aggregation improves DTD accuracy from 83.35 to 91.44 over the backbone, indicating strong bottleneck aggregation for diverse texture cues.Source: §5.4 · p. 15

Data efficiency

• At 10% of DTD training data, the SRM token improves over no statistical token by +1.64 pp, but its advantage narrows with more data.Source: §5.4.1 · pp. 16–17, Table 4 · p. 17

Fusion topology

• Parameter-matched fusion topologies produce essentially indistinguishable performance, indicating that the SRM prior matters more than a specific integration ordering.Source: §5.5 · pp. 17–18

SRM capture-depth robustness

• Final classification accuracy is statistically invariant to SRM capture depth, with all differences ≤0.11 pp and no paired t-test significant at p ≥0.37.Source: §5.6.1 · pp. 18–19, Table 6 · p. 18

Raw descriptor quality

• Raw SRM linear-probe accuracy rises from 46.17% for Early vectors to 79.73% for Late vectors, a 33.56 pp gap.Source: §5.6.2 · p. 19, Table 7 · p. 19

Mechanistic interpretation

• Despite this depth-dependent raw separability, the interact→distill fusion head achieves equivalent classification performance by leveraging complementary spatial tokens.Source: §5.6.2 · p. 19, §5.6.3 · p. 20

Representation geometry

• SRM vectors from different depths occupy distinct subspaces, with Early–Late CKA of 0.304 and Pearson correlation r = 0.004.Source: §5.6.3 · pp. 19–20, Table 8 · p. 19

Robustness under corruption

• Depth invariance persists under blur, JPEG, contrast, and Gaussian-noise corruption, with noise showing less than 0.5 pp spread.Source: §5.6.4 · p. 20, Table 9 · p. 20

Cue-conflict analysis

• The analysis tests whether explicit texture information changes perceptual bias at the expense of shape reliance.This directly examines whether the statistical token alters the model’s cue preference.Source: §5.7 · p. 20
• Cue-conflict images preserve one class’s shape while adopting another class’s texture, enabling separate measurement of shape bias, texture bias, and conflict accuracy.The evaluation uses AdaIN-generated images and compares predictions with shape and texture labels.Source: §5.7 · p. 21
• TBI remains below 2% for both HiPerViT and the baseline ViT, with HiPerViT slightly higher in shape bias and overall conflict accuracy.The cue-conflict analysis indicates that adding second-order statistics does not override shape-dominant cues under ImageNet-100 supervision.Source: §5.7 · p. 21, Table 10 · p. 21

Conditional texture utilization

• The interact→distill topology adaptively emphasizes spatial or statistical features according to task statistics rather than applying a fixed texture enhancement.The coefficients α and β shift with the task: α dominates under object-centric supervision, whereas β contributes more on texture benchmarks.Source: §5.7 · p. 21

Representation specialization

• The transfer study frames HiPerViT as balancing generic representation reuse against domain-specific optimization.This trade-off follows from retaining general backbone features while specializing the fusion output.Source: §5.8 · pp. 22–23

Evaluation design

• The study compares frozen backbone, frozen multi-stage, fine-tuned backbone, and post-distillation latent representations using linear probing between DTD and GTOS-Mobile.These stages span increasing architectural specialization.Source: §5.8 · p. 22

Cross-dataset transfer

• 80.14% target accuracy for DTD→GTOS latent features contrasts with 90.06% for fine-tuned backbone features and 90.01–90.74% for frozen DINOv2 features.The result indicates reduced cross-domain separability after interact→distill despite preserved transferability before fusion.Source: §5.8 · p. 22
• GTOS→DTD transfer falls to 72.71% for pre-fusion features and 64.10% for latent features, consistent with over-specialization from narrower source diversity.GTOS has fewer classes and a narrower visual vocabulary than DTD.Source: §5.8 · p. 22

Representation specialization

• The fusion bottleneck induces a source-conditioned projection of feature correlations, making latent features distribution-aligned rather than domain-agnostic embeddings.Backbone features retain more transferability, whereas latent features reflect task-adaptive statistical alignment.Source: §5.8 · pp. 22–23

Computational evaluation

• The benchmark reports GFLOPs, inference latency, and peak GPU memory across different ViT backbones.These metrics characterize the computational trade-offs of the architecture.Source: §5.9 · p. 23, Figure 3 · p. 23

Backbone efficiency

• ViT-S/16 delivers strong performance at 4.0 GFLOPs and 214 MiB peak memory, supporting resource-constrained deployment.ViT-B/16 reduces FLOPs by over 60% while maintaining nearly identical accuracy, whereas ViT-L/16 costs 27.0 GFLOPs for modest gains.Source: §5.9 · p. 23

Comparison with prior SOTA

• ConvNeXt-XXL with RADAM exceeds 350 GFLOPs yet does not outperform HiPerViT+ViT-B on FMD or GTOS-Mobile.The comparison shows that higher computational cost does not guarantee better reported texture-recognition performance.Source: §5.9 · p. 23

Overall efficiency

• Across datasets, HiPerViT combines strong accuracy with computational efficiency rather than relying on brute-force backbone scaling.The reported design uses multi-scale input, cross-stage fusion, and bilinear pooling.Source: §5.9 · p. 23

Evaluation scope

• HiPerViT is applied without domain-specific architectural tailoring to scientific datasets with specialized acquisition conditions and limited samples.The evaluation covers biotechnology, medicine, and agriculture.Source: §5.10 · pp. 23–24

Medicine

• The SICAPv2 comparison concerns automated prostate-biopsy analysis, where Gleason grading and grade-4 pattern identification are time-consuming and subject to inter-observer variability.Reported comparison accuracies include 76.22% and 85.13% from prior publications.Source: §5.10 · p. 25

Cross-domain evaluation

• Across three real-world datasets, HiPerViT remains competitive and exceeds the best reported results despite shifts among hyperspectral microscopy, histology, and greenhouse imagery.Table 12 consolidates results spanning biotechnology, medicine, and agriculture.Source: Table 12 · p. 25, §5.10 · p. 26

Implications

• The results exceed pipelines using stronger sensing modalities or additional task-specific engineering, supporting transfer beyond dataset-specific exploitation.The reported gains are attributed within the paper to hierarchical multi-scale extraction, token distillation, and explicit second-order interactions.Source: §5.10 · p. 26

• HiPerViT improves prior state-of-the-art performance across five of six benchmarks while avoiding excessively large backbones.The architecture combines multiscale extraction, compact bilinear statistics, and Perceiver-style latent distillation.Source: §6 · p. 26, §6.3 · p. 27
• The model gains +10.48 pp on GTOS-Mobile, +10.27 pp on 1200Tex, and +3.05 pp over ViT-L/14 on DTD.Smaller ViT-B/S backbones paired with the HiPerViT head also reduce computational requirements while retaining competitive accuracy.Source: §6 · p. 26
• HiPerViT shows depth- and topology-invariance, with performance remaining stable across statistical-extraction depths and interaction–distillation orderings.These results support the interpretation that explicit second-order information matters more than a particular fusion wiring.Source: §6.1 · p. 27, §6.3 · p. 27
• The study lacks counterfactual token baselines, broader domain evaluation, dynamic-texture experiments, and stronger statistical validation.The authors specifically identify random, shuffled, or capacity-matched learned-token controls and more seeds with confidence intervals as useful additions.Source: §6.2 · p. 27
• The approach is most beneficial when pairwise correlations are discriminative, pretraining omits second-order structure, and supervision is limited.Cue-conflict evaluation indicates that the statistical token does not override backbone priors when the task lacks relevant statistical cues.Source: §6.3 · p. 27

• STI reintroduces second-order statistical structure into self-supervised Vision Transformer representations through lightweight statistical tokens.Source: §7 · p. 28
• HiPerViT achieves state-of-the-art or competitive vision-only performance across six texture benchmarks and three applied scientific domains, with gains up to +10.48 pp.Source: §7 · p. 28
• Accuracy varies by less than 2 pp across ViT-S/B/L backbones once STI and the Perceiver head are integrated.The analysis also reports substantially reduced FLOPs relative to higher-capacity backbone configurations.Source: §7 · p. 28
• The results support making second-order statistics explicitly available rather than prioritizing a particular interaction–distillation ordering.Source: §7 · p. 28
• Explicit statistical inductive biases can benefit tasks where micro-structural information is discriminative without requiring multimodal pretraining or excessive compute.Source: §7 · p. 28