Source-linked AI summary
Phase Marginalization for Patch-Grid Instability in Vision Transformers
Oğuzhan Ercan
TL;DR
Vision Transformers can produce phase-sensitive dense predictions because patch-grid shifts alter token evidence near boundaries. The paper formalizes phase as a nuisance variable and uses training-free Uniform Phase Marginalization, which improves the canonical baseline across measured tasks while requiring additional phase-dependent forward passes.
Problem
Fixed ViT patchification creates a discrete spatial phase that can change token evidence and destabilize pixel-level predictions, especially near semantic boundaries.
Method
Uniform Phase Marginalization evaluates structured patch-grid phases, inverse-aligns their dense outputs, and aggregates them without updating the encoder, dense head, or normalization statistics.
Results
Uniform K = 4 improves over K = 1 across measured segmentation, depth, and local matching settings, including segmentation gains of +0.75 to +1.39 mIoU and NYU Depth v2 RMSE reduction from 0.6506 to 0.6277.
Takeaways & Limitations
Phase Marginalization provides a diagnostic and practical post-hoc baseline for accounting for patch-tokenization aliasing in dense ViT prediction, with K = 4 suggested as a measured cost-accuracy point.
Takeaways & Limitations
Compute-matched TTA comparisons and K-scaling analyses are limited to Cityscapes, while inference cost grows approximately linearly with the number of phases.
Abstract
from arXiv · showhide
Vision Transformers operate on fixed patch grids, which can introduce phase-dependent instability for dense prediction: changing the patch partition can change the token evidence available to a pixel, especially near boundaries. We formalize patch-grid phase as a nuisance variable and propose Phase Marginalization, a post-hoc marginalization method that evaluates structured patch-grid phases, inverse-aligns dense outputs, and aggregates them in the original image coordinate system. The central variant, Uniform Phase Marginalization with K = 4, is training-free and improves over the canonical K = 1 baseline across measured segmentation, depth, and local matching settings. In a controlled Cityscapes experiment, Uniform Phase Marginalization provides a modest compute-matched advantage over generic shift-based four-forward test-time augmentation (TTA) (+0.31 mean Intersection-over-Union over the strongest tested generic row). A scaling study further shows that K = 4 is a practical cost-accuracy trade-off: K = 8 is essentially unchanged and K = 16 adds little accuracy at much higher latency. These results position patch-grid phase as a measurable nuisance variable and Phase Marginalization as a simple diagnostic and post-hoc marginalization baseline for dense ViT prediction.
1 Introduction
The paper treats patch-grid phase as a measurable nuisance in dense ViT prediction and proposes training-free Phase Marginalization to reduce phase-dependent instability. Across measured tasks, Uniform K = 4 improves over canonical inference, with a modest compute-matched advantage over tested shift-based TTA and K = 4 serving as a practical default.
- Patchification creates a discrete spatial phase that can change token memberships and pixel-level evidence, especially near semantic boundaries.
- Patch-grid phase instability is defined as disagreement among dense predictions from different sub-patch phases after mapping them to common pixel coordinates.
- Phase Marginalization evaluates a frozen predictor under structured patch-grid phases, inverse-aligns dense outputs, and averages them without retraining or redesigning the backbone.
- The method samples patch-grid offsets rather than generic image transformations, enabling a controlled comparison with shift-based TTA under the same four-forward budget.
2 Related Work
Phase Marginalization addresses patch-tokenization aliasing by keeping the model fixed and marginalizing over discrete patch-grid phases at inference time. It is distinguished from architectural redesign, training-based adaptation, and generic transformation-based TTA.
- Patch-grid phase is a distinct source of aliasing arising from the non-overlapping patch grid used to tokenize ViT inputs.
- Architectural alternatives modify tokenizers, early feature extractors, windows, or token designs, whereas Phase Marginalization keeps the model fixed.
- Generic TTA averages predictions over image transformations, while Uniform Phase Marginalization averages predictions across patch-grid phases with inverse alignment.
3 Method
The method treats patch-grid phase as a nuisance variable and marginalizes it by aligning dense predictions from structured phase-shifted inputs before aggregation. Uniform Phase Marginalization is training-free and leaves the encoder, dense head, and normalization statistics unchanged.
- Patch-grid phase: A patch-grid phase is an offset ϕ = (d_x, d_y) defining where the patch grid begins relative to image coordinates.The canonical single-phase baseline uses ϕ = (0, 0), denoted K = 1.
- Phase alignment: For each phase, reflective padding and cropping shift the patch grid, while inverse alignment maps the resulting dense logits back to the original image coordinates.The aligned outputs can then be compared or aggregated at corresponding pixels.
- Phase instability: Phase instability is diagnosed with per-pixel variance across aligned phase features, where lower variance indicates less dispersion at the same pixel or descriptor location.Task metrics separately evaluate whether marginalization improves predictions.
- Uniform Phase Marginalization: Uniform Phase Marginalization averages inverse-aligned logits over a discrete set of patch-grid phases without updating the encoder, dense head, or normalization statistics.This makes it a post-hoc marginalization procedure around a frozen dense prediction model.
- Uniform Phase Marginalization: For K = 4 and even patch size P, the method samples four canonical quadrants of the sub-patch phase space.The paper evaluates this choice as a cost-accuracy trade-off on Cityscapes.
- Secondary variants: Learned and adapted variants use aligned K-phase feature stacks with learned phase attention, optional encoder tuning, or reliability-based weighting as secondary audits.These variants are distinct from the main Uniform Phase Marginalization method and include specialized alternatives such as pre-transformer patch-embedding averaging.
- Relation to generic TTA: Phase Marginalization differs from generic TTA by sampling patch-grid offsets and aligning predictions to the original pixel grid rather than sampling image transformations.The distinction remains even when both methods use the same number of forward passes.
4 Experiments
The experiments compare Uniform Phase Marginalization with canonical inference, generic shift-based TTA, phase-count variants, and adapted methods across segmentation, depth, and local matching. Results support K = 4 as a strong training-free setting, while showing task- and variant-specific differences.
- Setup and scope: The evaluation covers semantic segmentation, NYU Depth v2 depth estimation, and HPatches local feature matching, including cross-domain synthetic-to-real tests.Compute-matched TTA and K-scaling analyses are limited to Cityscapes, while learned and adapted variants change the training or aggregation setup.
- Cross-task results: Uniform K = 4 improves over K = 1 in every measured row, with segmentation gains of +0.75 to +1.39 mIoU and NYU Depth v2 RMSE decreasing from 0.6506 to 0.6277.HPatches matching accuracy also improves for both DINOv3 and DINOv2, although its submetrics are mixed rather than uniformly monotonic.
- Comparison with generic TTA: 53.53 mIoU is achieved by Uniform Phase Marginalization, exceeding integer-shift TTA at 53.22 mIoU by +0.31 mIoU under the same four-forward Cityscapes budget.Random subpatch-shift TTA reaches 53.00 mIoU; the comparison concerns the tested generic shift-based variants rather than TTA universally.
- Phase-count scaling: K = 4 adds +0.63 mIoU over K = 1, while K = 8 is essentially unchanged and K = 16 adds about +0.05 mIoU over K = 4 as latency rises from 87.52 to 347.16 ms/image.Latency grows approximately linearly with K because the dominant cost is the number of backbone forwards.
- Boundary diagnostics: Uniform K = 4 improves Boundary@5px and Boundary@3px over K = 1 for both shown Cityscapes backbones.Partial Encoder Tuning with Spatial Phase Attention improves further, but is an adapted variant rather than the training-free method.
- Variants and failure modes: Learned and adapted variants are task-dependent: some help on segmentation, but selected learned variants underperform Uniform K = 4 on GTA5→Cityscapes, HPatches, or depth.Pre-Transformer Patch-Embedding Averaging reduces GTA5→Cityscapes target mIoU to 44.48 for DINOv3/F, below both K = 1 and Uniform K = 4.
5 Limitations
The main scope limitation is that the compute-matched TTA comparison and K-scaling analysis are limited to Cityscapes, while inference cost increases with the number of phases.
- Scope and cost: The compute-matched TTA comparison and K-scaling analysis are limited to Cityscapes, and Uniform Phase Marginalization requires K forward passes with approximately linear inference-cost growth.External architecture and denoising families require separate matched protocols, and learned or adapted variants are task-dependent.
6 Conclusion
Patch-grid phase is an explicit nuisance variable in dense ViT prediction. Post-hoc marginalization improves fixed predictors without retraining, with K = 4 offering a practical measured trade-off.
- Patch-grid phase is an explicit nuisance variable induced by ViT tokenization and exposed directly by dense prediction.
- Post-hoc marginalization over structured phases improves fixed dense predictors without retraining the backbone.
- K = 4 provides a practical cost-accuracy point, while larger phase sets offer diminishing returns at substantially higher latency.
- Phase Marginalization serves as both a diagnostic for patch-tokenization aliasing and an inference-time baseline for mitigating it.
A Cross-Domain Segmentation Audit
The cross-domain segmentation audit groups rows by training and evaluation datasets. Its arrow notation clarifies the direction of transfer, while adapted attention rows are distinct from the plain learned aggregation method.
- Rows are grouped by evaluation setting in the cross-domain segmentation audit.
- GTA5→Cityscapes denotes models trained on GTA5 and evaluated on Cityscapes.
- Partial Encoder Tuning with Spatial Phase Attention rows are not plain Learned Phase-Feature Attention Aggregation.
B HPatches Metric Breakdown
The HPatches breakdown separates viewpoint, illumination, and repeatability metrics. Uniform K = 4 improves matching accuracy for both backbones, although submetrics are mixed rather than uniformly monotonic.
- Table 8 separates HPatches viewpoint and illumination subsets and includes repeatability metrics.
- HPatches submetrics are mixed rather than uniformly monotonic despite the matching-accuracy improvement.
- Uniform K = 4 improves matching accuracy for both evaluated backbones.
C Related-Method Taxonomy
The related-method taxonomy organizes approaches by intervention and deployment assumptions rather than performance ranking. It distinguishes fixed-predictor inference-time evaluation from methods requiring architectural changes or training.
- The taxonomy groups methods by the intervention they require and their deployment assumptions.
- Methods are separated according to whether they can be evaluated as fixed-predictor, inference-time procedures.
- The taxonomy is conceptual and should not be read as a performance ranking.
D Reproducibility Notes
The evaluation uses the MarginSeg framework, with planned release of code and result manifests, while external architecture comparisons require carefully matched evaluation conditions.
- The evaluation is implemented in the MarginSeg framework, with code and result manifests slated for release.
- Compute-matched rows use the study’s full evaluation runs.
- Fair empirical comparisons to external architecture families require matched training data, heads, checkpoints, evaluation code, and inference-cost accounting.