PaperlayerView exact revision on arXiv

Source-linked AI summary

Decoding Register Tokens Preserves Global Scene Details

David Jacob Drexlin, Julius Hense, Jonas Dippel

arXiv:2610.13299v1cs.CVcs.LG

TL;DR

Patch-based decoders discard register outputs, leaving image appearance to be recovered from patches alone. This paper decodes registers alongside all patches and predicts registers from sampled patches for generation; register inputs lower reconstruction FID on ImageNet-1K, and recovered registers lower generation FID on identical DINOv2 patch samples.

  • Problem

    Patch-based decoders discard non-spatial register outputs, leaving appearance recovery to patches even though registers may make global image properties easier to use.

    Source: §1 · p. 1

  • Method

    The paper decodes four encoder registers jointly with all patch tokens and uses an MHAP predictor to recover registers from sampled patches for generation.

    Source: §1 · p. 2

  • Results

    ImageNet-1K reconstruction FID is 9.5% lower with DINOv2 and 5.2% lower with DINOv3 than their patch-only references; DINOv2 generation FID is 1.641 versus 1.819 on identical sampled patches, a 9.8% reduction.

    Source: §1 · p. 2, §5 · p. 9

  • Takeaways & Limitations

    Registers expose global color and texture cues, while register-only reconstructions lose substantial object and spatial detail.

    Source: §4.1 · p. 5

Abstract

from arXiv · show

Pretrained vision transformers learn structured image representations spanning spatial patch tokens and non-spatial tokens. Recent analyses suggest that non-spatial register tokens encode global scene properties such as illumination, blur, and style. Yet patch-based image decoders discard these outputs, leaving the decoder to recover image appearance from patches alone. We investigate whether retaining register features improves decoding when the complete spatial patch representation is already available. We study this question using representation autoencoders (RAEs) with frozen register-equipped encoders. Our decoder processes four register tokens together with all patch tokens through joint self-attention. Appearance probes, paired reconstruction-error analysis, and fixed-decoder register substitutions examine which global image properties are accessible from registers and how their values affect reconstructed pixels. For generation, we train a multi-head attention pooling (MHAP) predictor to recover register features from sampled patches, keeping the pretrained diffusion model and its samples unchanged. On ImageNet-1K, our ViT-B decoders achieve mean reconstruction FIDs of 0.472 with DINOv2 and 0.442 with DINOv3, reductions of 9.5% and 5.2% relative to their respective patch-only baselines. The corresponding PSNRs are 19.73 and 20.54 dB. For generation, our DINOv2 models achieve a mean FID of 1.641, 9.8% below the patch-only baseline on identical sampled patches.

1 INTRODUCTION

Patch-based decoders discard register outputs, leaving appearance recovery to patches even when registers may make global properties easier to use. The paper tests register-aware decoding and register recovery for generation, reporting improved reconstruction and generation comparisons.

1 INTRODUCTION

  • Patch-based RAEs discard four register features from register-equipped DINO encoders, although registers have been associated with illumination, blur, and style.Recognizable objects alone do not ensure matching lighting, color relationships, or surrounding textures.Source: §1 · p. 1
  • The central question is whether registers improve decoding when the complete spatial patch representation is already available.The decoder processes four register outputs and all patch tokens through joint self-attention, with a frozen encoder and no additional decoder loss.Source: §1 · p. 2
  • 9.5% and 5.2% lower ImageNet-1K reconstruction FID with DINOv2 and DINOv3, respectively, than their patch-only references.The decoders retain all spatial patches and add four encoder registers.Source: §1 · p. 2
  • Mean generation FID is 1.641 with recovered registers versus 1.819 for patch-only decoding on identical DINOv2 patch samples, a 9.8% reduction.The pretrained diffusion model and its samples remain unchanged.Source: §1 · p. 2

2 RELATED WORK

The related work situates this study among representation autoencoders, methods for decoding pretrained representations, and analyses of register tokens. The paper's distinction is to decode registers alongside patches and recover them after diffusion sampling.

2 RELATED WORK

  • RAEs learn decoders over frozen vision-backbone features; this work exposes four register outputs from the same encoder while retaining all spatial patches.Other described extensions include text-to-image generation, multiple encoder layers, intermediate-feature alignment, and learned local variation.Source: §2 · p. 2
  • Unlike methods that guide or modify diffusion training, this register-recovery pathway operates after sampling and leaves pretrained diffusion weights and sampled patches unchanged.The related work also describes generation from semantic codes, self-supervised representations, and CLIP image embeddings.Source: §2 · p. 2
  • Prior register analyses associate register-specialized concepts with illumination, blur, and style; this work studies reconstruction and appearance control using registers as decoder inputs.The encoder's register computation is not altered.Source: §2 · p. 2

3 METHOD

The method retains frozen-encoder patches and registers for decoding, then predicts registers from sampled patches for generation. It compares patch-only and register-input decoding while keeping the generation DiT and sampled patches fixed.

3 METHOD

  • The method extends patch-based RAEs to decode encoder registers alongside spatial patches and to recover registers from sampled patches for generation.The two generation routes use the same pretrained DiT and sampled patches.Source: §3 · p. 2, Figure 1 · p. 3

3.1 ENCODER REPRESENTATIONS AND DECODER INPUTS

  • A frozen DINOv2 or DINOv3 encoder produces N patch features and four register outputs, and both decoder conditions use the same complete patch sequence.Only the patch-and-register decoder additionally receives register outputs.Source: §3.1 · p. 3

3.2 DECODING PATCHES AND REGISTERS

  • The register decoder learns token-wise input projections and prepends projected registers to projected patches for joint decoding.Both variants use a ViT-B transformer backbone and RGB reconstruction, perceptual, Gram, and adversarial image losses; no additional register-prediction loss is used.Source: §3.2 · p. 3

3.3 REGISTER RECOVERY FOR GENERATION

  • For generation, four learned MHAP queries attend to sampled patches to predict registers in encoder-defined order, trained on matched patch–register pairs independently of the decoder and diffusion model.The patch-only reference decodes the same sampled patches; predicted registers are converted back to encoder-feature coordinates before decoding.Source: §3.3 · p. 4

4 EXPERIMENTS

The experiments probe global appearance in register features, measure reconstruction effects, test fixed-decoder interventions, and evaluate register recovery for generation. ImageNet-100 supports probes and paired appearance analysis, while ImageNet-1K supplies the principal reconstruction and generation evaluations.

4.1 REGISTER STATE AND GLOBAL APPEARANCE

  • With full patches retained, adding registers raises mean-color DC R2 from 0.573 to 0.663, while register-only decoding recovers color and texture but loses object and spatial detail.The probes characterize what the ridge readout accesses, not all information available in patches.Source: §4.1 · p. 5, Figure 2 · p. 5

4.2 WHERE REGISTER INPUTS IMPROVE RECONSTRUCTION

  • On ImageNet-100, register inputs raise PSNR by 0.473 dB and reduce DISTS by 0.007; mean RGB MSE falls 8.52%, though centered MSE worsens in 21.63% of image–seed pairs.Of the absolute RGB MSE reduction, 43.21% is in the channel-mean term and 56.79% in the centered term.Source: §4.2 · p. 5

4.3 FIXED-DECODER REGISTER INTERVENTIONS

  • With original patches fixed, transformed registers shift reconstructed mean RGB toward the transformation (mean directional coefficient 0.499), alongside orthogonal changes and a classifier-logit cosine decrease from 0.929 to 0.924.The coefficient measures response along the transformation direction, not the share of reconstruction gain attributable to registers.Source: §4.3 · p. 6
  • Across 20 cross-scene register interpolations, luminance endpoints follow the donor direction in 19/20 cases and normalized-gradient endpoints in 20/20; 14/20 and 17/20 trajectories are monotonic.Receiver patches, decoder weights, and token positions remain fixed while register slots are interpolated.Source: §4.3 · p. 6, Figure 3 · p. 7

4.4 IMAGENET-1K RECONSTRUCTION

  • Register-input decoding reaches mean PSNRs of 19.73 and 20.54 dB for DINOv2 and DINOv3, and lowers mean RGB MSE by 3.28% and 2.96%.Without retraining, seed-0 rFID also falls on NINCO and Places365 for both encoder families.Source: §4.4 · pp. 6–7
  • Register-input ViT-B decoders lower mean rFID by 9.47% with DINOv2 and 5.24% with DINOv3 on ImageNet-1K.The comparison covers the same 50,000 validation images for each encoder.Source: §4.4 · p. 6, Table 2 · p. 7

4.5 REGISTER RECOVERY AND IMAGE GENERATION

  • On identical pretrained-DiT patch samples, MHAP-register decoding reaches mean gFID 1.6414 versus 1.8191 patch-only, a 9.77% reduction.The generator and sampled tensors remain unchanged.Source: Table 3 · p. 8, §4.5 · pp. 8–9
  • On encoded ImageNet-1K images, MHAP-predicted registers yield DISTS 0.1121 versus 0.1044 with native registers, but outperform mean and shuffled replacements.The same decoder and complete patch sequence are held fixed across register sources.Source: §4.5 · p. 8

5 DISCUSSION

The discussion argues that registers carry global appearance information and improve reconstruction and generation when retained alongside patches. The appendix specifies the decoder inputs, reconstruction objective, and training and preprocessing setup.

5 DISCUSSION

  • Register-only reconstructions recover global appearance, including grass or cloud color and texture, but lose substantial object and spatial detail.The discussion reports that appearance probes improve prediction of global image properties such as color histograms compared with mean-pooled patches alone.Source: §5 · p. 9
  • Retaining registers reduces ImageNet-1K reconstruction rFID by 9.5% for DINOv2 and 5.2% for DINOv3, while also improving out-of-distribution reconstruction metrics.On NINCO, rFID decreases by 6.2% and 5.2%; on Places365, it decreases by 43.9% and 28.1% for DINOv2 and DINOv3, respectively, alongside improved PSNR and DISTS.Source: §5 · p. 9
  • Recovering registers from sampled DINOv2 patches reduces generation gFID by 9.8% versus patch-only decoding without changing the diffusion model or samples.Within-class register shuffling worsens gFID with the decoder fixed; the tested MHAP route also outperforms alternatives, including a larger InfoNCE decoder.Source: §5 · p. 9
  • Registers may be dispensable for classification but not for decoding and generation; spherical interpolation between registers is proposed for global scene style transfer.The authors suggest that background information in registers may leave CLS and patch tokens with more disentangled semantic class information.Source: §5 · p. 9

A.1 FEATURE COORDINATES AND RECONSTRUCTION OBJECTIVE

  • DINOv2 and DINOv3 inputs produce 16 × 16 patch grids with 768 channels and four ordered registers, while encoder CLS is excluded.Training crops are 256 × 256; evaluation uses a fixed center crop, and DINOv2 is resized to 224 × 224 before encoding while DINOv3 remains at 256 × 256.Source: §A.1 · p. 13
  • The decoder sequence contains a learned decoder class token, four registers, and 256 patches, but predicts reconstruction pixels only from patch positions.The decoder class token is distinct from the omitted encoder CLS; patch and register features use separate learned projections.Source: §A.1 · p. 13
  • The native-register decoder objective combines RGB, perceptual, Gram, and adversarial terms, with the adversarial term activated after eight completed epochs.There is no register-prediction term in the native-register objective; MHAP is trained separately.Source: §A.1 · p. 14
  • The decoders use AdamW and train for 16 epochs, with evaluation using the final EMA export.Training totals 79,968 steps; clean runs use zero patch noise, register noise, and register dropout.Source: §A.1 · p. 14

A.2 EVALUATION AND UNCERTAINTY

Evaluation pairs outputs by image identity and reports image-level metrics across decoder seeds. The uncertainty intervals reflect seed variation, while metric aggregation and feature spaces differ by measure.

A.2 EVALUATION AND UNCERTAINTY

  • Image-level metrics are averaged across evaluation images within each seed, then summarized across seeds; paired intervals use within-seed model differences.With three seeds, the 95% interval describes training-seed variation, not variation across independently sampled evaluation images.Source: §A.2 · p. 14
  • FID compares image distributions in Inception feature space, PSNR measures paired pixel fidelity, and DISTS measures paired perceptual structure and texture differences.Supplementary FD-DINOv2 distances use DINOv2 features and are distinct from paired feature errors; feature-extraction procedures are fixed within each metric comparison.Source: §A.2 · p. 15
  • PSNR is computed per reconstructed image and averaged, so its mean is not equivalent to applying the logarithm to arithmetic-mean MSE.Relative reductions in mean pixel error are calculated directly from MSE; DISTS differences remain in the metric’s own units.Source: §A.2 · p. 15

A.3 COMPUTE AND INFERENCE ACCOUNTING

The appendix reports the added inference cost of register recovery and register-bank probe findings. The full register bank leads on thumbnail and nonzero low-frequency DCT scores, but not every target.

A.3 COMPUTE AND INFERENCE ACCOUNTING

  • At batch sizes 1 and 8, MHAP-plus-register decoding increases latency by 20.47% and 24.38%, and peak allocated memory by 16.42% and 13.38%, respectively.The measurements use epoch-16 decoders and one A100; they exclude diffusion sampling and characterize one model pair, not variation across trained models.Source: §A.3 · p. 15

B.1 INDIVIDUAL-REGISTER AND REGISTER-BANK PROBES

  • The complete register bank gives the highest thumbnail and nonzero low-frequency DCT scores, while the selected individual register scores higher on color-histogram and DC targets.Adding more register features therefore does not improve every linear prediction target.Source: §B.1 · p. 15, Table 5 · p. 16

B.2 FULL-PATCH APPEARANCE PROBES

Full-patch probes find the clearest register-associated gains in mean color, while register-only reconstructions and alternative decoder interfaces provide additional comparisons. The input study favors registers over CLS for reconstruction FID.

B.2 FULL-PATCH APPEARANCE PROBES

  • In the separate full-patch probe, DC R2 increases by 0.08944 and thumbnail R2 by 0.00582, while aggregate non-DC R2 changes by −0.00019.The probe uses flattened patch features and nested cross-validation; its absolute scores are not directly comparable with the patch-mean experiment.Source: §B.2 · p. 16

B.3 RECONSTRUCTION FROM REGISTERS ALONE

  • Figure 5 presents six register-only reconstruction examples from an ImageNet-100-trained DINOv2 ViT-B decoder using four registers without spatial patches or encoder CLS.Outputs use seed 0 and EMA weights after 15 completed epochs.Source: Figure 5 · p. 17

C.1 ALTERNATIVE REGISTER INTEGRATION

  • Register information can enter the decoder through pooled modulation, cross-attention memory, or joint self-attention over prepended registers and patches.These interfaces differ in whether registers condition patch features through a pooled vector, remain a separate memory, or evolve jointly with patches.Source: §C.1 · p. 18

C.2 CHOICE OF NON-SPATIAL DECODER INPUTS

  • Register inputs yield lower reconstruction FID than CLS inputs in both encoder studies; adding CLS to registers lowers rFID by 0.0181 for DINOv2 and 0.00379 for DINOv3.These correspond to 0.39% and 0.09% of the respective register-input scores; the decoder-capacity study uses the combined CLS-plus-register input.Source: §C.2 · p. 18, Table 7 · p. 18

C.3 DECODER CAPACITY

The capacity sweep finds that the tested CLS-plus-register auxiliary input retains an rFID advantage over patch-only decoding from ViT-B through ViT-XL. The section also compares register supervision and alternative decoder context, with different outcomes across those tests.

C.3 DECODER CAPACITY

  • For ViT-L, PSNR rises from 18.9101 to 19.2504 dB and SSIM from 0.4706 to 0.4790 with the combined CLS-plus-register input.Four-register-only capacity effects and training-seed uncertainty are unavailable for this sweep.Source: §C.3 · p. 18
  • The auxiliary-input decoder lowers rFID by 6.2%, 5.5%, and 5.5% at ViT-B, ViT-L, and ViT-XL, respectively, versus patch-only decoding.The sweep supplies CLS plus four registers alongside all spatial patches; the advantage remains as patch-only capacity increases.Source: §C.3 · p. 18, Table 8 · p. 19

C.4 REGISTER SUPERVISION VERSUS REGISTER INPUTS

  • Contrastive register supervision improves both recorded FIDs over patch-only, whereas register-MSE improves reconstruction FID but raises generation FID; neither matches native-register reconstruction and MHAP generation.Different decoder architectures and objectives mean these tests do not isolate loss choice or establish a parameter-matched comparison.Source: §C.4 · p. 19

C.5 LEARNED DECODER CONTEXT AND PATCH SUMMARIES

  • Learned tokens lower rFID by 0.0292 while raising rFDD by 0.0218; patch summaries lower rFID by 0.0116 and rFDD by 0.4393.These ablations use one seed and five tokens, unlike the principal four-register result with three decoder seeds.Source: §C.5 · p. 20, Table 10 · p. 20

D.1 IMAGENET-100 DECODER-INPUT COMPARISON

  • The ImageNet-100 decoder-input study compares patch-only decoding with patches plus four registers across three decoder-training seeds, retaining all spatial patches and the same base image objective.Source: §D.1 · p. 20

D.2 IMAGENET-1K RECONSTRUCTION BY DECODER SEED

  • For ImageNet-1K, each encoder comparison uses the same 50,000 validation image identities, with means and sample standard deviations computed across three decoder seeds.The aggregate rows reproduce the principal reconstruction comparison in Table 2.Source: §D.2 · p. 20

D.3 IMAGENET-1K REGISTER-DECODER FIDELITY

  • Among the register-input decoders, DINOv3 has lower mean RGB MSE than DINOv2 (0.01134 versus 0.01334), alongside higher PSNR and lower DISTS.The lower DISTS indicates closer agreement under that structure-and-texture metric.Source: §D.3 · p. 20

E.1 IMAGENET-1K-TRAINED DECODERS

ImageNet-1K-trained decoders are evaluated on NINCO and Places365 without additional training. Register-input decoding improves all three reported metrics for both encoder families on both datasets, while the rFID ordering between encoders varies by dataset.

E.1 IMAGENET-1K-TRAINED DECODERS

  • Register-input decoding improves all three metrics for DINOv2 and DINOv3 on NINCO and Places365; rFID decreases 6.18% on NINCO and 43.93% on Places365 for DINOv2, and 5.18% and 28.14%, respectively, for DINOv3.Within each encoder comparison, both routes use a ViT-B decoder backbone and the 16-epoch endpoint; evaluation images, preprocessing, output conversion, and metric implementations are held fixed.Source: §E.1 · p. 21
  • Paired fidelity also improves: PSNR gains range from 0.1686 to 0.4022 dB, while DISTS decreases by 0.00489 to 0.00652 across the encoder–dataset comparisons.The passage reports the DINOv2 and DINOv3 gains separately for NINCO and Places365.Source: §E.1 · p. 21
  • Among register-input decoders, DINOv3 has higher PSNR and lower DISTS on both datasets, but the lower-rFID encoder differs: DINOv3 on NINCO and DINOv2 on Places365.Source: §E.1 · p. 21

E.2 IMAGENET-100-TRAINED PATCH-ONLY DECODERS

  • The ImageNet-100-trained patch-only scores are reference results, not paired counterparts to the ImageNet-1K-trained register-decoder evaluations.The training datasets differ, and no register-decoding comparison is added for those references.Source: §E.2 · p. 21

F.1 IMAGE-SPACE EVALUATION OF RECOVERED REGISTERS

The section tests recovered-register substitutions with fixed patches and decoders, then examines register prediction, generation settings, and further reconstruction analyses. Recovered values outperform mean or shuffled replacements in the reported comparisons, while encoder-produced registers remain better in the image-space test.

F.1 IMAGE-SPACE EVALUATION OF RECOVERED REGISTERS

  • With patches and the decoder fixed, MHAP predictions yield lower pixel and perceptual errors than mean or shuffled register replacements, but encoder registers remain better: prediction substitution raises DISTS by 0.00764.The paired image-bootstrap 95% interval for the DISTS increase is [0.00756, 0.00772], conditional on the fixed models.Source: §F.1 · pp. 21–22

F.2 REGISTER REPLACEMENTS ON GENERATED PATCHES

  • On generated patches, aligned MHAP register predictions yield lower FID than either the per-slot training-set mean or within-class shuffled predictions.The decoder, predictor, and sampled patches are held fixed across the replacements.Source: §F.2 · p. 22

F.3 MHAP PREDICTOR

  • The MHAP predictor uses the full sequence of 256 terminal patch features to predict four registers in encoder-defined order, using four learned queries and two attention-pooling blocks.It does not receive or predict CLS.Source: §F.3 · p. 22

F.4 DINOV2 PRETRAINED-XL EVALUATION

  • The pretrained-XL generation evaluation keeps the same 50,000 stored patch samples across decoding routes, and its released-decoder gFWD value uses a different reference from gFID.The FWD measurements use different representations and reference supports and are not numerically interchangeable.Source: §F.4 · p. 24

F.5 JOINT GENERATION OF PATCHES AND AUXILIARY TOKENS

  • In the separate DiT-S experiment, joint generation has higher gFID than patch generation followed by MHAP recovery at auxiliary-loss weights 0.05 and 0.25 under the recorded common reference.This compares separately trained generators, not auxiliary sources at fixed sampled patches; the experiment uses CLS plus four registers.Source: §F.5 · p. 25

F.6 QUALITATIVE MODEL AND SAMPLE IDENTITIES

  • The reconstruction examples were selected post hoc to illustrate appearance differences and do not estimate average improvement; quantitative comparisons use the complete fixed evaluation sets.Source: §F.6 · p. 25

G.1 DISTS TEXTURE AND STRUCTURE CONTRIBUTIONS

  • Both weighted DISTS components decrease in every decoder seed; texture and structure account for 82.1% and 17.9% of the weighted DISTS reduction, respectively.Those percentages partition the metric change, not preserved texture or the causes of reconstruction improvement.Source: §G.1 · p. 26

G.2 RESIDUAL SPECTRA

  • Register-input decoding reduces residual-gradient MSE by 1.372 × 10^-5 in smooth windows and 9.117 × 10^-4 in detailed windows.Output gradient power also decreases in detailed windows; radial averaging removes orientation information.Source: §G.2 · p. 26

G.3 REGISTER SUBSTITUTIONS

The section quantifies how register substitutions change reconstructed appearance with patches fixed, then tests register interpolation and reports a wavelet-distance comparison.

G.3 REGISTER SUBSTITUTIONS

  • 0.49943 and 0.55917 are the original-patch and transformed-patch coefficients, with a paired interaction of 0.05974.The four-way comparison keeps decoder weights, register positions, normalization, and output procedure fixed.Source: §G.3 · pp. 26–27
  • 0.01552 RGB-unit orthogonal response accompanies a ConvNeXt-Tiny logit-cosine decrease from 0.92937 to 0.92420 under original-patch register substitution.The intervention jointly changes channel-wise color and saturation; independent brightness, color-balance, and contrast control requires separate interventions.Source: §G.3 · pp. 27–28

G.4 REGISTER INTERPOLATION WITH FIXED RECEIVER PATCHES

  • With receiver patches and decoder fixed, interpolation changes only the four register inputs, using a shared interpolation value across slots.The displayed trajectories use the DINOv2 four-register ViT-B decoder trained on ImageNet-1K; interpolation uses the same α for all four slots.Source: §G.4 · p. 28
  • 19/20 luminance endpoints and 20/20 gradient-energy endpoints follow the donor-defined direction; 14/20 and 17/20 trajectories, respectively, are monotonic.These counts describe retained pairs, not independent replicates, and no population-level confidence intervals are assigned.Source: Table 21 · p. 29, §G.4 · p. 29

G.5 WAVELET DISTANCE

  • 0.86126 patch-only FWD versus 0.83224 register-input FWD gives a paired change of −0.02902, with a 95% seed interval that includes zero.The interval is [−0.07072, 0.01267], despite lower FWD for register-input decoding in each seed.Source: §G.5 · p. 29

H ADDITIONAL RECONSTRUCTION AND GENERATION EXAMPLES

The additional galleries show ImageNet-1K reconstructions and generated-image pairs from DINOv2 ViT-B decoders, with maps visualizing pixelwise differences.

Additional reconstruction examples

  • Figure 6 compares originals, patch-only reconstructions, and reconstructions using four encoder registers; signed RGB maps show error reduction and RMS maps show reconstruction change.Red indicates lower error with registers, blue lower error with patches alone; map scales are fixed and values outside them saturate.Source: Figure 6 · p. 30, H ADDITIONAL RECONSTRUCTION AND… · p. 30

Additional generation examples

  • Figure 7 decodes identical stored patch tensors with patch-only or MHAP-predicted-register inputs, and its RMS maps show output-change magnitude, not generation quality.The panels use a fixed display scale, with values above 0.20 saturating.Source: H ADDITIONAL RECONSTRUCTION AND… · p. 30, Figure 7 · p. 31

Related papers

Loading 2610.13299v1…