Source-linked AI summary
GramLoop: Training-Free Gram-Gated Replay for Robust Dense Prediction
Yang Chen, Canyu Shen, Xinzhe Rao, Yuanyi Yan, Yunlu Chen, Meng Tang, Teng Long, Vincent Tao Hu
TL;DR
Frozen DINOv3 dense-prediction models can lose robustness under distribution shift, motivating inference-time refinement without changing model weights, adapters, or heads. GramLoop replays a short transformer window and gates patchwise updates by final-layer cosine-Gram consistency. It improves all five shifted benchmarks over paired DINOv3 while preserving clean ADE20K performance.
Problem
Distribution shifts degrade dense-prediction systems, while refining a frozen backbone at inference time without updating its backbone or task-specific head remains an open question.
Method
GramLoop replays a short transformer window, propagates proposals through the frozen suffix, and uses patchwise final-layer cosine-Gram drift to gate endpoint updates.
Results
GramLoop improves over the paired DINOv3 baseline on all five shifted benchmarks and preserves clean ADE20K performance.
Takeaways & Limitations
Final-layer Gram gating refines the shared frozen token trajectory across object detection and semantic segmentation without changing model parameters or task heads.
Abstract
from arXiv · showhide
We aim to improve frozen DINOv3 dense-prediction models under distribution shift by adding inference computation inside the visual backbone, without changing model weights, task adapters, or prediction heads. The challenge is that repeated transformer-block computation must refine dense features without disrupting the pairwise patch relations that DINOv3 uses to preserve spatial structure. We introduce GramLoop, a training-free framework that replays a short transformer window and controls each replay through final-layer cosine-Gram consistency. Each proposal is propagated through the frozen suffix, measured against the standard DINOv3 trajectory, and accepted through a patchwise gate at the replay-window endpoint. Across object detection and semantic segmentation under corruptions, perturbations, and natural shifts, GramLoop improves all five shifted benchmarks over the paired DINOv3 baseline. On COCO-O, it improves mAP by +0.252 and Effective Robustness by +0.250, while preserving clean ADE20K performance. Code will be released.
Introduction
DINOv3 offers powerful frozen features for dense prediction, but clean performance does not ensure robustness under distribution shift. GramLoop addresses this by refining the frozen backbone trajectory at inference time while preserving patch relations and leaving task components unchanged.
- DINOv3 scales self-supervised vision-transformer pretraining to models with up to 7B parameters and 1.7B images, supporting general-purpose dense features.
- Clean benchmark strength does not guarantee robustness, as corruptions, real-world perturbations, and natural shifts degrade segmentation and detection systems.
- GramLoop asks whether a frozen visual backbone’s representation trajectory can be refined at inference time without updating the backbone or task-specific head.
- GramLoop replays a short transformer window and gates patchwise updates using final-layer cosine-Gram drift measured after propagation through the frozen suffix.
- The framework improves COCO-O, COCO-P, COCO-C, ADE20K-C, and ADE20K-P while preserving clean performance, with diagnostics and ablations explaining the gains.
Related Work
Related work frames GramLoop within self-supervised vision transformers, test-time adaptation, recurrent-depth computation, and dense prediction. Its distinctive scope is inference-time modification of the frozen backbone trajectory while keeping the released adapter and task head unchanged.
- DINO, DINOv2, and DINOv3 develop increasingly scalable self-supervised vision-transformer representations for image- and pixel-level tasks.
- Test-time adaptation methods use unlabeled test inputs to update parameters or normalization statistics in response to distribution shift.
- Recurrent-depth transformers and related equilibrium models reuse parameters or solve implicit fixed-point transformations to increase effective computation.
- Dense-prediction research applies transformer backbones to structured spatial outputs such as semantic labels, depth, categories, and boxes.
- GramLoop leaves the released adapter and task head unchanged and modifies only the frozen backbone trajectory at inference time for segmentation and detection.
Method
GramLoop replays a short frozen DINOv3 transformer window and uses final-layer patchwise cosine-Gram drift to gate endpoint updates before the unchanged task head. It evaluates proposals through the frozen suffix, attenuating updates that disrupt patch relations while retaining additional computation for structurally compatible proposals.
- Framework: GramLoop adds training-free replay to a frozen DINOv3 dense-prediction backbone without updating the backbone, task adapter, or prediction head.The unchanged task head receives predictions after the modified backbone token trajectory.
- Diagnostic: In a COCO-P impulse-severity-10 example, high-drift regions receive smaller gates and the post-gating drift remains small on the shared drift color scale.Figure 2 orders panels as corrupted input, proposal drift, gate, and post-gating drift; the gate scale is [0, 1].
- Window Replay: A short transformer window is replayed from each accepted endpoint state, producing a proposal that is evaluated through the frozen suffix.The initial window pass belongs to the ordinary trajectory; only the prescribed additional replays are counted.
- Gram Guidance: Final-layer Gram drift compares each replay proposal with the standard DINOv3 trajectory and measures changes in each patch’s relations to the entire image.The gate uses normalized patch tokens and exact global cosine Gram matrices after final normalization.
- Patchwise Gate: Patchwise gates are recomputed per valid patch and replay, with high-drift proposals receiving smaller endpoint updates through per-token interpolation.Gates are normalized by median valid-patch drift, use ϵ = 10^-6 for numerical stability, and vary smoothly between retaining the current state and accepting more proposal displacement.
- Gated Endpoint Update: The accepted endpoint state is recurrently fed into the next replay, after which the frozen suffix is recomputed and the unchanged task head produces predictions.Class and register tokens share an image-level scalar gate, while patch tokens receive spatially varying gates.
Experiments
GramLoop is evaluated across controlled corruptions, real-world-inspired perturbations, and natural domain shifts for semantic segmentation and object detection. It consistently improves shifted-benchmark robustness, with gains generally increasing at higher severities, while diagnostics support patchwise final-layer gating and loop-end updates.
- Evaluation setup: GramLoop is evaluated on five shifted benchmarks spanning semantic segmentation and object detection under corruptions, perturbations, and natural domain shifts.The suites include ADE20K-C, ADE20K-P, COCO-C, COCO-P, and COCO-O.
- Evaluation metrics: mAP and Effective Robustness are reported for COCO-O, while mIoU and mean performance under corruption are reported for semantic-segmentation evaluations.COCO-O mAP averages domain-level AP across six domains; mPC averages the task metric over non-clean shift–severity conditions.
- Main results: GramLoop improves the paired DINOv3 baseline on COCO-O, COCO-P, COCO-C, ADE20K-P, and ADE20K-C while preserving clean ADE20K performance.It also improves clean ADE20K by +0.165 mIoU, and qualitative examples show a corrected semantic region and a recovered object at a shared 0.40 detection threshold.
- Robustness checks: All four bootstrap confidence intervals exclude zero, supporting consistency of the improvements across sampled corruption and perturbation families.Each paired cluster-bootstrap replicate retains all five severity levels of every sampled family.
- Severity analysis: Gains generally concentrate at severities 4–5 across the four suites, indicating greater benefits under stronger distribution shifts.Figure 3 places severity levels 1–5 from left to right, with each benchmark’s largest gain at severity 4 or 5.
- Validation transfer: Fixed cross-validation hyperparameters improve three of four ADE20K subsets and all four complements, reducing concern that one validation partition drives the aggregate gain.The validation set contains 2,000 images, with four fixed 500-image subsets and corresponding 1,500-image complements.
- Ablations: Final-layer Gram gating with loop-end updates is the strongest evaluated placement configuration.The ablation compares where Gram consistency is measured and where the accepted update is applied; loop end and final layer correspond to blocks 23 and 39.
- Diagnostics and cost: At matched mean update strength, patchwise gating lowers final-layer cosine-Gram discrepancy across all 17 COCO-P perturbations and 85 severity conditions.GramLoop uses 1.96× GPU-equivalent inference time with unchanged peak memory, making runtime the principal reported trade-off.
Conclusion
GramLoop improves robustness of frozen DINOv3 dense-prediction models under distribution shifts while preserving clean ADE20K performance. Its strongest evaluated configuration uses final-layer global Gram evidence to gate updates at the replay-window endpoint.
- GramLoop improves COCO-O, COCO-P, COCO-C, ADE20K-P, and ADE20K-C over the paired DINOv3 baseline while preserving clean ADE20K performance.
- Final-layer global Gram evidence applied at the replay-window endpoint is the strongest evaluated configuration.
- Future work targets adaptive replay timing and counts, alongside lower-cost reliability probes that preserve gains under distribution shifts.
Supplementary Material
The supplementary material details GramLoop’s frozen inference procedure, evaluation setup, task interfaces, qualitative visualizations, diagnostics, and computational cost. It also specifies how Gram consistency and gated replay are analyzed across shifted conditions.
- Method: The inference procedure anchors each image with an initial forward pass, propagates replay proposals through the frozen suffix, and gates endpoint updates using final-layer Gram consistency.After K gated replays, the suffix and unchanged task head produce the final prediction from retained multi-level features.
- Evaluation configuration: Reported experiments use a frozen 40-block DINOv3 ViT-7B/16 backbone, replay blocks [21, 23] for K = 2 passes, and apply the gate at block 23.Exact global cosine-Gram matrices are computed at the final layer with ϵ = 10^-6.
- Implementation: No model parameters are updated, and the inference path is deterministic for fixed inputs, checkpoints, and configuration.Seed 100 is retained only for defensive random-number-generator initialization and provenance.
- Task interfaces: Taps before the replay window retain standard-forward features, whereas later taps are recomputed from the accepted loop endpoint and carry the refined trajectory.Patch tokens are the direct backbone inputs to both task heads; class and register tokens are handled according to task-specific interfaces.
- Evaluation: ADE20K-C and COCO-C retain original annotations after input corruption, while COCO-P uses its released condition-specific annotations for paired detection evaluation.
- Qualitative results: Qualitative figures use stored predictions from the same frozen task heads, with green contours or boxes identifying GramLoop-correct or newly recovered objects under specified matching rules.Visualization-only category, crop, score, and IoU filters do not enter reported metrics.
- Diagnostics: The impulse-noise diagnostic compares proposal drift, gate maps, and post-update Gram statistics across released severities and a deterministic severity-10 stress test.
- Runtime: For L = 40, W = 3, S = 16, and K = 2, GramLoop evaluates 94 equivalent backbone blocks versus 40 for the baseline.The multiplier counts frozen transformer-block evaluations and excludes the shared task head and Gram algebra.