VisionHOPE: Visual Backbones as Self-Modifying Learning Systems

• Across CNNs, ViTs, SSMs, and TTT, visual computation becomes more input-adaptive, yet the trained backbone still largely prescribes the adaptation rules.CNNs use local kernels, ViTs use global self-attention, SSMs use input-dependent transitions, and TTT adapts an inner learner.Source: §1 · pp. 1–2
• VisionHOPE aligns chunks with image rows and columns across four directional scans to process two-dimensional feature maps.Each direction maintains independent states, and directional outputs are fused channel-wise.Source: §1 · p. 2
• VisionHOPE couples content, key, value, learning-rate, and retention memories that co-evolve along visual scans, modifying both what it remembers and how it learns.Source: §1 · p. 2
• A soft cap on self-referential injection and a spectral clamp on the retained transition jointly guarantee non-expansive memory dynamics.The unconstrained self-referential update creates a stability barrier for visual backbones.Source: §1 · p. 2
• VisionHOPE achieves competitive results on ImageNet-1K, COCO, and ADE20K, demonstrating its potential as a general-purpose visual backbone.Source: §1 · p. 3

2 METHODOLOGY

• The methodology reviews SRNL foundations, proves non-expansion for token-wise and chunk-wise recurrences, and embeds the controlled module in a residual visual backbone.Its stages are theoretical foundations, stability-matched step-size control, and application of independent SRNL instances along four scans.Source: §2 · p. 3

2.1 NESTED LEARNING FOUNDATIONS

• Nested Learning models a system as interconnected learning processes that compress context flows into internal states.VisionHOPE builds on this self-referential construction.Source: §2.1 · p. 3

2.1.1 ASSOCIATIVE MEMORY AND DELTA GRADIENT DESCENT

• DGD uses an L2 regression target and corrects the write using the memory response at the current key, while retention scales the prior memory.This correction is intended for context flows with highly correlated elements.Source: §2.1.1 · pp. 3–4

2.1.2 SELF-REFERENTIAL NESTED LEARNING

• SRNL couples content, key, value, learning-rate, and retention memories, each generating quantities that help determine subsequent updates.VisionHOPE uses a linear L2 recurrence, and the explicit differentiable recurrence permits outer gradients to pass through the memory trajectory without an iterative inner solver.Source: §2.1.2 · pp. 4–5

2.1.3 CHUNK-WISE LINEAR RECURRENCE

• Chunking holds boundary states fixed while generating token-dependent quantities in parallel, then updates working memories in token order and carries them to the next chunk.The boundary states refresh only at chunk boundaries.Source: §2.1.3 · p. 5

2.2 STABILITY-MATCHED STEP-SIZE CONTROL

• The step-size control combines a soft cap on self-referential injection with a spectral clamp on the retained transition, guaranteeing non-expansive memory dynamics for token-wise and chunk-wise updates.The bounds control both potential sources of one-step amplification, including chunk-boundary transitions.Source: §2.2 · p. 6

2.3 VISIONHOPE ARCHITECTURE

• VisionHOPE runs independent SRNL instances over four directional scans with row- or column-aligned chunks, restores the outputs spatially, and fuses them with learned channel-wise scales.The operator is incorporated into a block with projections, depthwise convolution, and an FFN in two pre-normalized residual branches.Source: §2.3 · pp. 6–7, Figure 2 · p. 7

3.1 IMAGE CLASSIFICATION

• Across hierarchical and plain backbone layouts, VisionHOPE achieves the best or tied-best ImageNet-1K Top-1 accuracy in every scale panel, with accuracy reported as a percentage.The hierarchical and plain comparisons are at 224×224 resolution.Source: §3.1 · p. 8, Table 1 · p. 8, Table 2 · p. 8

3.2 OBJECT DETECTION AND INSTANCE SEGMENTATION

• VisionHOPE achieves the best or tied-best box and mask AP across all three evaluated model scales on COCO.The comparison uses Mask R-CNN under the 1× schedule.Source: §3.2 · p. 8, Table 3 · p. 9

3.3 SEMANTIC SEGMENTATION

• VisionHOPE achieves the best mIoU among the compared backbones on ADE20K semantic segmentation.The evaluation transfers ImageNet-pretrained hierarchical backbones to UPerNet.Source: §3.3 · pp. 8–9, Table 4 · p. 9

3.4 EFFICIENCY ANALYSIS

• Vim and VisionHOPE have linear token-count computation and memory scaling versus DeiT’s quadratic terms; at similar FLOPs, P-VisionHOPE-B has higher throughput and lower memory across tested resolutions.Figure 3 compares FLOPs, throughput, and memory from 224 × 224 to 1280 × 1280.Source: §3.4 · p. 9, Figure 3 · p. 10

3.5 ABLATION STUDY

• Ablations favor coupled SRNL memories and DGD and broader directional coverage; removing the soft injection cap diverges, while clamp substitutions stay stable but underperform full control.Removing the spectral clamp leaves accuracy unchanged in this setting, possibly because its activation rate is extremely low.Source: §3.5 · p. 10, Table 5 · p. 10

4 CONCLUSION

• VisionHOPE couples five memories so stored content and the learning rule co-evolve within each image; its stability control and four directional scans support non-expansive dynamics and linear token scaling.Experiments on ImageNet-1K, COCO, and ADE20K show strong performance in classification and dense prediction.Source: §4 · p. 10

A.1 CONVOLUTIONAL AND TRANSFORMER BACKBONES

• Convolutional and transformer backbones use local aggregation or global attention, but their learned aggregation mappings or interaction-generating mechanisms remain fixed during image processing.The discussion cites CNNs and ConvNets including ConvNeXt and InternImage, and transformer designs including Swin, EfficientViT, and RMT.Source: §A.1 · p. 15

A.2 VISUAL STATE-SPACE MODELS

• Visual SSMs adapt state transitions to input tokens, but their outer-parameterized transition-generation rule remains unchanged within an image.The discussion includes bidirectional processing in Vim and four-directional selective scanning in VMamba.Source: §A.2 · p. 15

A.3 TEST-TIME TRAINING IN VISION

• VisionHOPE differs from visual TTT by co-evolving content, key, value, learning-rate, and retention memories, rather than leaving update dynamics largely outer-parameterized.The coupled memories evolve as visual context accumulates.Source: §A.3 · pp. 15–16

A.4 NESTED LEARNING AND SELF-MODIFYING SYSTEMS

• Nested Learning couples five memories, but incorporating its Continuum Memory System into discriminative vision would require an additional online objective because visual scans lack a canonical token-wise target.VisionHOPE therefore focuses on NL’s self-referential component.Source: §A.4 · p. 16

B.1 GRADIENT DESCENT AS A DOT-PRODUCT ASSOCIATIVE MEMORY

• Gradient descent can be formulated as a dot-product associative-memory update: the key specifies the write address, while the negative gradient supplies the write signal.A proximal objective combines the mapping objective with a penalty on changes to the previous memory state.Source: §B.1 · p. 16

B.2 PROXIMAL DERIVATION OF DELTA GRADIENT DESCENT

• The proximal L2 formulation yields the DGD recurrence, with the key norm kept explicit to cover unnormalized keys.The objective is strictly convex for a positive proximal parameter, giving a unique minimizer.Source: §B.2 · pp. 16–17

B.3 PROPERTIES OF THE STABILITY-MATCHED STEP-SIZE CONTROL

• A soft cap on self-referential injection and a spectral clamp on the retained transition ensure non-expansion of token-wise memory dynamics.The resulting memory-state norms remain bounded by their initial values; finite-precision implementation uses slightly more conservative limits to provide margins against roundoff.Source: §B.3 · pp. 17–18

B.4 CHUNK-WISE STABILITY OF VISIONHOPE

• All five memories share a right-multiplicative gain within a fixed chunk because the transition and injection matrices do not depend on the memory.The gain is defined relative to that chunk’s boundary state.Source: §B.4 · p. 18
• Under stability-matched step-size control, every within-chunk gain and memory state satisfies the stated non-expansion theorem.The theorem applies to each token within a fixed chunk.Source: §B.4 · p. 18
• Applying the chunk bound repeatedly keeps every memory-state norm along the scan bounded by its initial value.The end-of-chunk state becomes the boundary state for the next chunk.Source: §B.4 · p. 19

B.5 VISIONHOPE ARCHITECTURE DETAILS

• Each direction uses independent five-memory SRNL states, while the multi-head implementation reuses outer-projected queries across the four scans.Head-wise reads are concatenated, combined with a channel-wise skip, and fused across directions.Source: §B.5 · p. 19

C.1 MODEL CONFIGURATIONS

• Hierarchical models distribute blocks across four downsampled stages, whereas plain models retain one feature resolution after the convolutional stem.At 224 × 224 resolution, row and column chunk lengths coincide; the hierarchical and plain model configurations are specified in Tables 6 and 7.Source: §C.1 · p. 19, Table 6 · p. 20, Table 7 · p. 20
• The stability analysis requires only positive step sizes and retention factors in [0, 1), so the chosen scalar-map parameterization does not alter its non-expansion guarantees.The reported settings use γη = 0.025 and αinit = 0.9.Source: §C.1 · p. 20

Image classification

• ImageNet models use a 300-epoch AdamW schedule with warmup and cosine decay; hierarchical models receive 10 additional minimum-learning-rate epochs and MESA training, unlike plain models.MESA begins at epoch 75 and adds teacher-provided soft class targets to the classification objective.Source: §C.2 · p. 20

Object detection and instance segmentation

• Mask R-CNN models start from corresponding ImageNet-pretrained hierarchical backbones and are fine-tuned with AdamW and standard multi-scale augmentation.The 1× and 3× schedules run for 12 and 36 epochs, respectively.Source: §C.2 · p. 20

Semantic segmentation

• UPerNet models start from corresponding ImageNet-pretrained hierarchical backbones and train for 160K iterations with AdamW and standard ADE20K augmentation.Training uses linear warmup followed by polynomial learning-rate decay.Source: §C.2 · p. 21

C.3 EFFICIENCY ANALYSIS DETAILS

• Per-layer compute accounting separates projections from attention or recurrent token interactions, with VisionHOPE’s recurrence and chunk-boundary updates counted across four routes.Its row-major chunks span feature-map rows and column-major chunks span columns.Source: §C.3 · p. 21
• VisionHOPE’s activation-memory estimate includes token representations, directional token-dependent quantities, and chunk-boundary storage; each chunk gain is shared across five memories.The accounting also includes directional state-space activations for Vim and explicit attention-matrix storage when materialized for DeiT.Source: §C.3 · p. 21
• At 224 × 224 to 1280 × 1280, P-VisionHOPE maintains linear computation and memory growth; versus DeiT its efficiency advantage widens, and versus Vim it has higher throughput at every tested resolution.Compared with Vim, it also uses less memory at high resolutions while maintaining a comparable FLOP profile.Source: Figure 4 · p. 22, §C.3 · p. 22

C.4 COCO RESULTS UNDER THE 3× SCHEDULE

• Under COCO’s 3× Mask R-CNN schedule, VisionHOPE retains competitive Tiny- and Small-scale object-detection and instance-segmentation performance.Table 8 reports bounding-box and mask AP, including AP50 and AP75.Source: Table 8 · p. 22, §C.4 · p. 22

Step-size control behavior

• For P-VisionHOPE-S with the injection-clamp variant, the clamp activates on 49.68% of token-wise inference updates and replaces the raw step with ηinj when active.The soft injection cap smoothly limits self-referential injection while preserving graded modulation of the raw step.Source: §C.5 · p. 22

Effect of MESA

• MESA improves Top-1 accuracy across the compared model families, while VisionHOPE exceeds MILA and H-ViT3 at every scale with or without MESA.Table 10 reports hierarchical-model ImageNet-1K accuracy for Tiny, Small, and Base scales.Source: Table 10 · p. 23, §C.5 · p. 23

Step-size control behavior

• In full P-VisionHOPE-S, the spectral clamp condition holds for 0.0044% of inference updates and 0.0011% of updates across the first ten warmup epochs.The full model and the variant without this clamp have matching accuracies; the clamp changes candidate steps only when they exceed the retained-transition limit.Source: §C.5 · p. 23
• The soft spectral cap remaps every positive candidate step, including those within the limit, and this additional remapping plausibly explains its lower accuracy.Unlike that cap, the spectral clamp leaves below-limit candidate steps unchanged.Source: §C.5 · p. 23

Configuration sensitivity

• The default chunk length C = 14 yields a 2.86× inference speedup over C = 1 while maintaining comparable Top-1 accuracy.The configuration ablations vary internal width, chunk length, raw-step scale, and memory-head dimension.Source: §C.5 · p. 23, Table 9 · p. 23

Global perception

• P-VisionHOPE-S distributes receptive-field influence broadly across the two-dimensional input while retaining a clear central response, unlike the more localized or directional patterns described for other backbones.The comparison uses ImageNet-1K-pretrained models at 512 × 512 resolution and averages maps over the validation set.Source: §C.6 · pp. 23–24, Figure 5 · p. 24

Within-image learning dynamics

• VisionHOPE’s executed step forms structured, content-dependent spatial patterns, with larger values often aligning with prominent objects and distinctive visual structures.The maps restore four scan orders to the 14 × 14 feature grid and average across heads and directions; darker regions indicate larger executed steps.Source: §C.6 · p. 24

Strengths

• VisionHOPE unifies within-image adaptation of stored content, update representations, and learning dynamics through five coupled memories, with non-expansion guarantees for token-wise and chunk-wise recurrences.The operator supports hierarchical and plain backbones, with computation and activation memory scaling linearly with visual token count.Source: D STRENGTHS & WEAKNESSES · p. 24

Weaknesses and future work

• The current evaluation focuses on general-purpose visual recognition; VisionHOPE has not been evaluated as an MLLM encoder or empirically validated on medical-imaging and remote-sensing tasks.The authors identify these settings as future evaluation targets.Source: D STRENGTHS & WEAKNESSES · p. 24
• Figure 6 maps final-block executed steps onto 14 × 14 image-coordinate grids, averaged across heads and four scan directions.Darker regions denote relatively larger executed steps.Source: Figure 6 · p. 25