PaperlayerView exact revision on arXiv

Source-linked AI summary

HiPerViT: A Hierarchical Perceiver-Vision Transformer Architecture for Multi-Scale Texture Recognition

João Pedro C. A. de Sá, Odemir Martinez Bruno

arXiv:2609.10917v1cs.CV

TL;DR

Texture recognition is difficult because vision models often favor object shape while leaving higher-order spatial statistics implicit. HiPerViT inserts compact bilinear statistics as explicit tokens and integrates them with spatial features through multiscale processing and Perceiver-style distillation. Across six benchmarks, it improves over strong vision-only baselines, with gains that remain stable across capture depths and fusion topologies.

  • Problem

    Texture recognition requires higher-order spatial statistics, but standard vision representations do not expose co-occurrence structure explicitly.

  • Method

    HiPerViT combines global and local views, compact bilinear statistical tokens, multiscale features, and Perceiver-style latent distillation.

  • Results

    HiPerViT achieves consistent vision-only improvements across six texture benchmarks, including gains of up to +10.48 pp.

  • Takeaways & Limitations

    Stable gains across fusion topologies and backbone capture depths support explicit statistical tokenization as a robust texture-recognition design principle.

  • Takeaways & Limitations

    The study lacks counterfactual token baselines, broader non-texture evaluation, and experiments on temporal or dynamic textures.

Abstract

from arXiv · show

Texture recognition remains challenging for modern vision models because discriminative evidence is often carried by higher-order spatial statistics rather than by object shape alone. While Vision Transformers provide strong long-range modeling capacity, their standard object-centric representations do not explicitly expose such statistical structure, which limits texture sensitivity in fine-grained recognition settings. We present HiPerViT, a compact vision-only architecture that injects an explicit second-order statistical prior into a transformer-based recognition pipeline. The method combines global and local image views with a compact bilinear descriptor encoded as a statistical token, and integrates this token with first-order spatial representations through Perceiver-style latent distillation. This design enables direct interaction between spatial tokens and second-order feature co-occurrence statistics, providing the model with explicit access to texture-relevant information without requiring multimodal pretraining or ensemble construction. Across six texture recognition benchmarks, HiPerViT achieves consistent improvements over strong vision-only baselines under the reported evaluation protocols, including gains of +3.05 percentage points on DTD, +10.48 on GTOS-Mobile, and +10.10 on 1200Tex. Beyond benchmark performance, our analyses show that these gains are largely invariant to the backbone depth used to extract second-order statistics and to the ordering of interaction and distillation stages. This pattern suggests that the primary source of improvement is not a specific fusion topology, but the explicit availability of second-order statistical information as a first-class representational signal. These results support explicit statistical tokenization as an effective and robust design principle for texture-centric visual recognition.

1. Introduction

Texture recognition requires sensitivity to higher-order spatial statistics, but standard vision models do not expose co-occurrence structure as an explicit representation. HiPerViT addresses this gap with statistical tokens and shows robust gains across texture benchmarks and design variations.

  • Texture recognition depends on higher-order spatial statistics, local structure, long-range context, and cross-scale organization rather than object shape alone.
  • Standard CNN and ViT representations do not expose higher-order co-occurrence statistics as first-class information, while multimodal pretraining can be costly and impractical in specialized domains.
  • The architecture is motivated by whether texture gains arise from a specific fusion design or from exposing higher-order statistics explicitly.
  • HiPerViT combines global and local views, compact bilinear co-occurrence statistics, and Perceiver-style latent distillation to integrate statistical and spatial tokens.
  • HiPerViT achieves competitive or best-reported accuracy across six texture benchmarks while remaining computationally compact.It matches or exceeds transformer baselines across all six datasets under matched protocols.
  • Performance gains remain stable across fusion orderings and second-order capture depths, supporting explicit statistical availability over a privileged fusion topology.

2. Background

Texture recognition evolved from handcrafted statistical descriptors to learned multiscale and transformer-based representations. HiPerViT combines transformer global context with hierarchical features and explicit second-order modeling to address transformers’ tendency to favor shape over texture.

  • Texture methods progressed from handcrafted co-occurrence statistics to CNN aggregation, bilinear pooling, multiscale architectures, and transformer variants.
  • Handcrafted descriptors captured texture regularities but often generalized poorly across scales, lighting conditions, and domains.
  • CNN solutions frequently required task-specific modifications, indicating that their inductive biases were not inherently tuned to texture statistics.
  • Vanilla ViTs provide global attention but develop strong shape priors and often neglect fine-grained texture cues.
  • Multimodal pretraining can improve texture sensitivity but requires large models, substantial computation, and language data.
  • HiPerViT bridges global transformer reasoning and local statistical structure through a multiscale ViT, Perceiver bottleneck, and compact bilinear pooling.

3. Biological Motivation

Biological vision motivates hierarchical, multiscale processing in which early local responses are integrated into mid-level co-occurrence representations and compressed into efficient higher-level codes. HiPerViT mirrors this organization with multiscale features, compact bilinear pooling, and latent compression.

  • Early visual processing emphasizes localized orientation- and frequency-selective micro-patterns, providing a biological motivation for multiscale feature extraction.
  • Mid-level visual areas integrate local filter responses with co-occurrence statistics, curvature, surface properties, and broader contextual cues.
  • The biological framework treats texture perception as coordinated interaction among specialized modules rather than a monolithic sequence.
  • Higher cortical processing merges local structure with semantic associations and compresses distributed inputs into compact representations robust to variability.
  • HiPerViT mirrors hierarchical cortical processing through a multiscale ViT backbone, pairwise co-occurrence coding, and a Perceiver-style latent bottleneck.

4. Model Architecture and Methodology

HiPerViT injects compact second-order feature statistics as an explicit token into a hierarchical, multi-scale transformer pipeline, enabling cross-order interaction with spatial representations. Its Perceiver-style latent bottleneck compresses these combined signals while preserving texture-relevant information.

  • Statistical Token Injection: Statistical Token Injection converts compact second-order feature statistics into an explicit token that participates directly in transformer self-attention.Count Sketch bilinear projection approximates pairwise co-occurrences without constructing the full outer-product representation.
  • Multi-Scale Feature Extraction: Global and local views capture complementary macro-structural and fine-grained textural information using a shared-weight ViT backbone.The shared backbone processes a downsampled global image and a high-resolution local crop while reducing memory relative to separate branches.
  • Second-Order Statistical Branch: Compact bilinear pooling transforms stage-specific token features into concatenated second-order descriptors that are integrated alongside aggregated multi-scale spatial tokens.The second-order branch approximates outer-product statistics through Tensor Sketch, producing the compact vector ZSRM.
  • Topology Robustness: Reversing interaction and distillation or deferring SRM integration to late fusion yields equivalent accuracy, indicating robustness to fusion topology.The ablations support the statistical prior, rather than a particular wiring arrangement, as the key design ingredient.
  • Cross-Order Interaction: Cross-order self-attention lets spatial tokens re-weight second-order correlations while contextualizing the statistical token with localized image structure.This bidirectional interaction makes the statistical signal context-dependent rather than requiring it to be linearly separable alone.

5. Experiments and Analysis

HiPerViT is evaluated across six texture benchmarks using established protocols and controlled transformer comparisons. It reports strong vision-only results, while the authors note that published SOTA comparisons are heterogeneous and that identical-protocol comparisons provide the cleaner architectural test.

  • Datasets and Evaluation Protocol: The evaluation covers six texture benchmarks spanning controlled and in-the-wild conditions, with dataset-specific established splitting protocols.The benchmarks are DTD, FMD, KTH-TIPS2-b, GTOS-Mobile, USPTex, and 1200Tex.
  • Comparison with State-of-the-Art: Published SOTA values are heterogeneous because they come from different architectures, pre-training modalities, and task-specific tuning procedures.The authors therefore frame identical-protocol comparisons as the more rigorous test of architectural claims.
  • Comparison with State-of-the-Art: HiPerViT establishes new vision-only state-of-the-art on all six benchmarks, with substantial gains on five.Reported gains include +3.05 pp on DTD, +10.48 pp on GTOS-Mobile, and +10.1 pp on 1200Tex; USPTex reaches 100.00%.
  • Comparison with State-of-the-Art: HiPerViT outperforms all alternative transformer architectures across all datasets under identical training protocols.Table 1 separates previously published results from controlled comparisons against transformer baselines.
  • Comparison with State-of-the-Art: Table 1 compares external vision-only SOTA results with transformer models trained under a shared protocol.Its two subtables support side-by-side interpretation of published results and controlled baselines.

5.3. Impact of Multi-Scale and Hierarchical Fusion

HiPerViT benefits from combining complementary image scales and backbone stages through structured fusion, while the statistical token is especially useful when supervision is limited.

  • Multi-scale inputs: Combining coarse and fine input scales consistently outperforms single-scale setups, supporting complementary global and local context.
  • Hierarchical fusion: Fusing early- or mid-level features with deeper semantic stages outperforms shallow-only or deep-only combinations.
  • Component ablation: The full model outperforms strongest partial variants across controlled texture benchmarks, although USPTex provides limited headroom for complex additions.
  • Component ablation: Perceiver-only aggregation improves DTD accuracy from 83.35 to 91.44 over the backbone, indicating strong bottleneck aggregation for diverse texture cues.
  • Data efficiency: At 10% of DTD training data, the SRM token improves over no statistical token by +1.64 pp, but its advantage narrows with more data.

5.5. Fusion Topology Robustness and Order Invariance

HiPerViT’s final accuracy is robust to both fusion ordering and the backbone depth supplying second-order statistics. Raw statistical descriptors vary substantially with depth, but the fusion head integrates them effectively, including under corruption.

  • Fusion topology: Parameter-matched fusion topologies produce essentially indistinguishable performance, indicating that the SRM prior matters more than a specific integration ordering.
  • SRM capture-depth robustness: Final classification accuracy is statistically invariant to SRM capture depth, with all differences ≤0.11 pp and no paired t-test significant at p ≥0.37.
  • Raw descriptor quality: Raw SRM linear-probe accuracy rises from 46.17% for Early vectors to 79.73% for Late vectors, a 33.56 pp gap.
  • Mechanistic interpretation: Despite this depth-dependent raw separability, the interact→distill fusion head achieves equivalent classification performance by leveraging complementary spatial tokens.
  • Representation geometry: SRM vectors from different depths occupy distinct subspaces, with Early–Late CKA of 0.304 and Pearson correlation r = 0.004.
  • Robustness under corruption: Depth invariance persists under blur, JPEG, contrast, and Gaussian-noise corruption, with noise showing less than 0.5 pp spread.

5.7. Cue-Conflict Behavior and Conditional Texture Utilization

HiPerViT’s explicit statistical token does not make ImageNet-100 predictions more texture-biased, while its fusion weights adapt across object-centric and texture-recognition settings.

  • Cue-conflict analysis: The analysis tests whether explicit texture information changes perceptual bias at the expense of shape reliance.This directly examines whether the statistical token alters the model’s cue preference.
  • Cue-conflict analysis: Cue-conflict images preserve one class’s shape while adopting another class’s texture, enabling separate measurement of shape bias, texture bias, and conflict accuracy.The evaluation uses AdaIN-generated images and compares predictions with shape and texture labels.
  • Cue-conflict analysis: TBI remains below 2% for both HiPerViT and the baseline ViT, with HiPerViT slightly higher in shape bias and overall conflict accuracy.The cue-conflict analysis indicates that adding second-order statistics does not override shape-dominant cues under ImageNet-100 supervision.
  • Conditional texture utilization: The interact→distill topology adaptively emphasizes spatial or statistical features according to task statistics rather than applying a fixed texture enhancement.The coefficients α and β shift with the task: α dominates under object-centric supervision, whereas β contributes more on texture benchmarks.

5.8. Cross-Dataset Transfer and Representation Specialization

Cross-dataset transfer reveals that HiPerViT preserves backbone-level universality under favorable source diversity but its distilled latent representation becomes more task-specialized.

  • Representation specialization: The transfer study frames HiPerViT as balancing generic representation reuse against domain-specific optimization.This trade-off follows from retaining general backbone features while specializing the fusion output.
  • Evaluation design: The study compares frozen backbone, frozen multi-stage, fine-tuned backbone, and post-distillation latent representations using linear probing between DTD and GTOS-Mobile.These stages span increasing architectural specialization.
  • Cross-dataset transfer: 80.14% target accuracy for DTD→GTOS latent features contrasts with 90.06% for fine-tuned backbone features and 90.01–90.74% for frozen DINOv2 features.The result indicates reduced cross-domain separability after interact→distill despite preserved transferability before fusion.
  • Cross-dataset transfer: GTOS→DTD transfer falls to 72.71% for pre-fusion features and 64.10% for latent features, consistent with over-specialization from narrower source diversity.GTOS has fewer classes and a narrower visual vocabulary than DTD.
  • Representation specialization: The fusion bottleneck induces a source-conditioned projection of feature correlations, making latent features distribution-aligned rather than domain-agnostic embeddings.Backbone features retain more transferability, whereas latent features reflect task-adaptive statistical alignment.

5.9. Computational Efficiency

HiPerViT achieves strong texture-recognition performance with substantially lower computational cost than larger comparison models, and smaller backbones remain competitive.

  • Computational evaluation: The benchmark reports GFLOPs, inference latency, and peak GPU memory across different ViT backbones.These metrics characterize the computational trade-offs of the architecture.
  • Backbone efficiency: ViT-S/16 delivers strong performance at 4.0 GFLOPs and 214 MiB peak memory, supporting resource-constrained deployment.ViT-B/16 reduces FLOPs by over 60% while maintaining nearly identical accuracy, whereas ViT-L/16 costs 27.0 GFLOPs for modest gains.
  • Comparison with prior SOTA: ConvNeXt-XXL with RADAM exceeds 350 GFLOPs yet does not outperform HiPerViT+ViT-B on FMD or GTOS-Mobile.The comparison shows that higher computational cost does not guarantee better reported texture-recognition performance.
  • Overall efficiency: Across datasets, HiPerViT combines strong accuracy with computational efficiency rather than relying on brute-force backbone scaling.The reported design uses multi-scale input, cross-stage fusion, and bilinear pooling.

5.10. Real-world applications

HiPerViT transfers to biotechnology, medicine, and agriculture without domain-specific architectural changes, remaining competitive or exceeding reported results across substantial domain shifts.

  • Evaluation scope: HiPerViT is applied without domain-specific architectural tailoring to scientific datasets with specialized acquisition conditions and limited samples.The evaluation covers biotechnology, medicine, and agriculture.
  • Medicine: The SICAPv2 comparison concerns automated prostate-biopsy analysis, where Gleason grading and grade-4 pattern identification are time-consuming and subject to inter-observer variability.Reported comparison accuracies include 76.22% and 85.13% from prior publications.
  • Cross-domain evaluation: Across three real-world datasets, HiPerViT remains competitive and exceeds the best reported results despite shifts among hyperspectral microscopy, histology, and greenhouse imagery.Table 12 consolidates results spanning biotechnology, medicine, and agriculture.
  • Implications: The results exceed pipelines using stronger sensing modalities or additional task-specific engineering, supporting transfer beyond dataset-specific exploitation.The reported gains are attributed within the paper to hierarchical multi-scale extraction, token distillation, and explicit second-order interactions.

6. Discussion

HiPerViT combines multiscale representations, statistical tokens, and latent distillation to improve texture recognition efficiently. Its gains remain robust across fusion topologies and statistical-extraction depths, while the study identifies limits in causal controls, domain coverage, temporal evaluation, and statistical rigor.

  • HiPerViT improves prior state-of-the-art performance across five of six benchmarks while avoiding excessively large backbones.The architecture combines multiscale extraction, compact bilinear statistics, and Perceiver-style latent distillation.
  • The model gains +10.48 pp on GTOS-Mobile, +10.27 pp on 1200Tex, and +3.05 pp over ViT-L/14 on DTD.Smaller ViT-B/S backbones paired with the HiPerViT head also reduce computational requirements while retaining competitive accuracy.
  • HiPerViT shows depth- and topology-invariance, with performance remaining stable across statistical-extraction depths and interaction–distillation orderings.These results support the interpretation that explicit second-order information matters more than a particular fusion wiring.
  • The study lacks counterfactual token baselines, broader domain evaluation, dynamic-texture experiments, and stronger statistical validation.The authors specifically identify random, shuffled, or capacity-matched learned-token controls and more seeds with confidence intervals as useful additions.
  • The approach is most beneficial when pairwise correlations are discriminative, pretraining omits second-order structure, and supervision is limited.Cue-conflict evaluation indicates that the statistical token does not override backbone priors when the task lacks relevant statistical cues.

7. Conclusion

The paper introduces Statistical Token Injection, a lightweight mechanism that adds explicit second-order structure to self-supervised Vision Transformer representations. HiPerViT combines this mechanism with multiscale extraction and latent distillation, achieving competitive or state-of-the-art vision-only results while supporting the broader view that explicit statistical availability matters more than integration wiring.

  • STI reintroduces second-order statistical structure into self-supervised Vision Transformer representations through lightweight statistical tokens.
  • HiPerViT achieves state-of-the-art or competitive vision-only performance across six texture benchmarks and three applied scientific domains, with gains up to +10.48 pp.
  • Accuracy varies by less than 2 pp across ViT-S/B/L backbones once STI and the Perceiver head are integrated.The analysis also reports substantially reduced FLOPs relative to higher-capacity backbone configurations.
  • The results support making second-order statistics explicitly available rather than prioritizing a particular interaction–distillation ordering.
  • Explicit statistical inductive biases can benefit tasks where micro-structural information is discriminative without requiring multimodal pretraining or excessive compute.
Loading 2609.10917v1…