PaperlayerView exact revision on arXiv

Source-linked AI summary

Implicit Neural Representation Facilitates Unified Universal Vision Encoding

Matthew Gwilliam, Xiao Wang, Xuefeng Hu, Zhenheng Yang

arXiv:2601.14256v1cs.CV

TL;DR

Existing image encoders generally specialize in recognition or generation, while unified representations must also accommodate different computational budgets. HUVR addresses this gap with an INR hyper-network, knowledge distillation, and compressed TinToks; it reports strong recognition and reconstruction results, including gains over DINOv3 and Stable Diffusion VAE baselines.

  • Problem

    Image representation models are typically specialized for recognition or generation, and unified representations must support both task diversity and compressed or non-compressed outputs.

  • Method

    HUVR maps images to INR weights for reconstruction, uses knowledge distillation for semantics, and generates standard representations alongside compressed Tiny Tokens.

  • Results

    HUVR matches or outperforms DINOv3 on reported recognition, segmentation, and reconstruction comparisons, while TinToks improve over DINOv3 PCA and Stable Diffusion VAE baselines.

  • Takeaways & Limitations

    HUVR demonstrates a unified image representation whose compressed embeddings support recognition and reconstruction across multiple visual tasks.

  • Takeaways & Limitations

    HUVR is trained at a smaller scale and scope than several prior methods, and VLM applications would require text-aligned pre-training.

Abstract

from arXiv · show

Models for image representation learning are typically designed for either recognition or generation. Various forms of contrastive learning help models learn to convert images to embeddings that are useful for classification, detection, and segmentation. On the other hand, models can be trained to reconstruct images with pixel-wise, perceptual, and adversarial losses in order to learn a latent space that is useful for image generation. We seek to unify these two directions with a first-of-its-kind model that learns representations which are simultaneously useful for recognition and generation. We train our model as a hyper-network for implicit neural representation, which learns to map images to model weights for fast, accurate reconstruction. We further integrate our INR hyper-network with knowledge distillation to improve its generalization and performance. Beyond the novel training design, the model also learns an unprecedented compressed embedding space with outstanding performance for various visual tasks. The complete model competes with state-of-the-art results for image representation learning, while also enabling generative capabilities with its high-quality tiny embeddings. The code is available at https://github.com/tiktok/huvr.

1. Introduction

HUVR aims to unify recognition and generative representation learning with one INR hyper-network that produces both standard and compressed image representations. Its design combines INR reconstruction, semantic distillation, and Tiny Tokens for broad visual tasks.

  • Motivation: Recognition-focused and generative-focused encoders are typically specialized, motivating a natively unified representation with information spanning high-level semantics through pixel-level reconstruction.
  • Method: HUVR uses an INR hyper-network that maps images to neural network weights for reconstruction while borrowing transformer encoder design and latent dimensions.The resulting INR maps pixel coordinates to color values.
  • Method: HUVR generates both standard-size representations and compressed Tiny Tokens, unifying representations across tasks and computational budgets.The paper frames compression as an additional axis of unified representation learning.
  • Method: Knowledge distillation from a pretrained vision encoder addresses the semantic limitations of INR hyper-networks, while token refactoring supports patch and global representations.
  • Results: +0.4% ImageNet classification, +1.2 mIoU ADE20K semantic segmentation, and +4.84 PSNR reconstruction are reported for HUVR with ViT-B/16 versus DINOv3.
  • Results: At 96x compression, TinToks provide +48% ImageNet classification over a DINOv3 PCA baseline and +1.26 PSNR over Stable Diffusion VAE at equal embedding size.

2. Related Work

Prior work spans implicit neural representations, hyper-networks, and specialized recognition or generative encoders. HUVR is positioned as a unified encoder whose standard and compressed encodings support recognition and reconstruction.

  • Implicit Neural Representation: Implicit neural representations map coordinates to signals and have been widely studied for image and video compression.
  • Hyper-Networks: INR hyper-networks avoid costly per-sample INR training by predicting network weights that reconstruct inputs or generate novel outputs.
  • Image Representation Learning: Unsupervised image representation learning progressed from restoration objectives to contrastive learning, clustering, masked image modeling, and text-image modeling.
  • Unified Representation: HUVR’s standard and compressed encodings are presented as supporting downstream tasks from classification to image generation.
  • Image Representation Learning: Some recent methods jointly target recognition and generation, while others adapt pretrained generative models post-hoc for recognition.

3. Methods

HUVR uses a hyper-network to convert image inputs into instance-specific INR parameters, while restructuring tokens and adding distillation to support both recognition and reconstruction. Its pipeline produces standard, compressed, and INR-modulation encodings.

  • INR Background: An INR maps coordinates to an image, but ordinary INRs require lengthy per-sample optimization to memorize each signal.
  • Hyper-Network: A hyper-network learns a dataset-level mapping from input images to instance-specific INR weights, enabling encoding through one forward pass.
  • Prior Design: Prior hyper-networks use a transformer encoder, shared base INR weights, input weight tokens, and projections that form modulation matrices.
  • HUVR Overview: HUVR removes learnable weight tokens and predicts an INR for each patch, producing standard ViT outputs, TinToks, and INR modulation encodings sequentially.
  • Hyper-Network Design: A global token and patch token are projected into dimensions whose outer product forms the INR modulation matrix, while the global token also supports recognition.
  • Training: The model trains with distillation and visual-quality losses, including pixel mean-squared error between the input image and reconstruction.
  • Training: Distillation is applied to final encoder and decoder outputs rather than directly to compressed tokens, with reconstruction helping TinToks acquire downstream semantics.

4. Results

HUVR is evaluated as a unified representation across recognition, reconstruction, and diffusion, using standard and compressed tokens. Its compressed representations show strong recognition–reconstruction performance, while generation remains promising but below current generative state of the art.

  • 4.1. Settings: HUVR uses unlabeled DataComp and ImageNet22k data with ViT-B/16 and ViT-L/16 models, evaluating feature quality without finetuning.Recognition is measured by linear probing on ImageNet1k, ReaL labels, ObjectNet, and five fine-grained datasets.
  • 4.2. Tiny Tokens are a Unified Representation: Compressed TinToks simultaneously support recognition and reconstruction, with performance gains over PCA-based baselines increasing at higher compression ratios.The evaluation includes linear-probing classification and a decoder-based reconstruction baseline using frozen DINOv3 PCA features.
  • 4.2. Tiny Tokens are a Unified Representation: DiT models trained on HUVR compressed tokens demonstrate generative potential, but the reported results do not surpass Stable Diffusion VAE or current generative state-of-the-art performance.Table 2 compares class-conditional DiT-XL models using original VAE latents and HUVR compressed tokens with FID, Inception Score, Precision, and Recall.
  • 4.2. Tiny Tokens are a Unified Representation: HUVR standard-size embeddings match or beat prior works on ImageNet and perform well on ObjectNet and fine-grained datasets.The authors note that SigLIP 2 uses 25 times more data for the ObjectNet comparison, while DINOv3 uses related mined pretraining data for FGVC.

4.4. HUVR can Perform Dense Recogntion Tasks

HUVR supports dense recognition tasks with both standard and compressed tokens, while its INR design achieves strong reconstruction quality and can improve with training and teacher selection.

  • 4.4. HUVR can Perform Dense Recogntion Tasks: HUVR achieves strong semantic segmentation and depth estimation results with both standard tokens and TinToks.Evaluation uses ADE20K semantic segmentation and NYUv2 depth estimation after additional mixed-resolution training.
  • 4.4. HUVR can Perform Dense Recogntion Tasks: TinToks achieve the overall best performance for dense tasks despite fitting patch reconstruction and segmentation information into highly compressed representations.Dense-task performance can improve at the expense of reconstruction performance.
  • 4.4. HUVR can Perform Dense Recogntion Tasks: HUVR’s INR hyper-network achieves state-of-the-art image reconstruction results compared with prior methods.The comparison covers ImageNette, LSUN, and CelebA in terms of PSNR.
  • 4.4. HUVR can Perform Dense Recogntion Tasks: HUVR achieves the best PSNR with equal or shorter training time while using equal or fewer encoder, shared, and predicted-latent parameters.The authors use fewer epochs than prior works because the network generally requires more time per epoch.
  • 4.4. HUVR can Perform Dense Recogntion Tasks: The patch-wise design is the major driver of reconstruction performance, while the global token also improves reconstruction.Compression is unnecessary for reconstruction but essential for good TinToks, and the decoder is essential for well-behaved distillation.
  • 4.4. HUVR can Perform Dense Recogntion Tasks: Recognition and reconstruction improve together with training time, although reconstruction saturates more quickly than classification.Teacher selection also changes the balance across classification and reconstruction metrics.

4.7. Some Design Decisions Involve Trade-offs

HUVR’s design choices involve explicit trade-offs among standard classification, TinTok classification, segmentation, reconstruction, speed, and computational cost.

  • 4.7. Some Design Decisions Involve Trade-offs: Distillation teacher choice creates task trade-offs: RADIOv3 gives the best overall segmentation but the weakest ImageNet classification.The authors suggest that an optimal method might distill from a mixture of teachers, but leave that engineering effort out of scope.
  • 4.7. Some Design Decisions Involve Trade-offs: Distilling to the global token helps classification, while distilling to patch tokens helps segmentation.Distilling to all tokens gives the best d = 768 classification but substantially worsens reconstruction.
  • 4.7. Some Design Decisions Involve Trade-offs: The best reconstruction and standard classification block selection is the worst for TinTok classification, whereas (11, 4) improves TinTok classification.The final choice, (12, 4), is presented as a middle ground and a simpler design.
  • 4.7. Some Design Decisions Involve Trade-offs: Increasing the INR hidden dimension can hurt results, while increasing decoder token dimension improves reconstruction at the expense of speed and TinTok classification.The evaluated hidden-dimension change is from 256 to 512.
  • 4.7. Some Design Decisions Involve Trade-offs: Removing decoder attention improves TinTok classification without affecting standard-token classification or segmentation, but significantly harms reconstruction.The authors use this setting for TinTok experiments and counterbalance the reconstruction loss with more training time.
  • 4.7. Some Design Decisions Involve Trade-offs: HUVR and TinyTokens lead many benchmarks and uniquely provide native reconstruction, but other methods remain superior on some datasets and metrics.Potential VLM applications would require text-aligned pre-training.

5. Conclusion

HUVR is presented as a unified universal image representation model whose compressed TinToks support recognition and reconstruction across multiple visual tasks.

  • 5. Conclusion: HUVR combines recognition and reconstruction through compressed TinToks across classification, segmentation, and generation tasks.The paper evaluates different datasets and embedding sizes.
  • 5. Conclusion: The authors position HUVR as a basis for further work on unified representation learning, compressed representations, and implicit neural representation.

6. Implicit Neural Representation and Hypernetwork Tutorial

Implicit neural representations store images in neural-network weights by mapping coordinates to signal values. Hyper-networks avoid fitting a separate INR for each image by learning to generate instance-specific weights from image inputs, while modulating shared base weights.

  • Image INR Basics: An INR maps coordinate inputs X to an image or other signal I through a neural network fθ.For images, coordinates are typically transformed with sinusoidal or Fourier positional encodings before entering the network.
  • Image INR Basics: Internal learning overfits an INR to one image so its weights memorize that image for reconstruction.The training and testing distinction disappears because the target is image-specific memorization.
  • Image INR Basics: Image-specific INRs can support generative operations such as super-resolution by evaluating the learned function at interpolated coordinates.The supplied passage also notes novel-view synthesis for scene-based INRs.
  • Hyper-networks: Hyper-networks pretrain a mapping h from input images to corresponding instance-specific INR weights, reducing encoding each image to one forward pass.This replaces separate per-sample INR fitting with a learned dataset-level mapping.
  • Hyper-networks: A hyper-network modulates shared base INR weights rather than predicting the entire INR, with HUVR modulating only the second layer in the described formulation.The remaining layers are reused directly from the learned base INR, while the second layer becomes W2 ⊙ M2.

7. Ablations

The ablations examine teacher size, distillation weighting, and the reconstruction objective. Larger teachers can become superior with sufficient training, while removing reconstruction prevents reconstruction and substantially harms TinTok classification.

  • Distillation Teacher Size Crossover Point: Larger DINOv3 teachers initially underperform smaller teachers but become superior after sufficient training, with the crossover occurring later for standard-size tokens than TinToks.The reported crossover can require up to 40 epochs for classification and 10 epochs for segmentation with standard tokens.
  • Distillation Loss Weights: Distillation-weight ablations evaluate main-token classification, tiny-token classification, main-token segmentation, and reconstruction PSNR under changes to decoder, global-token, and patch-token weights.The experiment trains for five ImageNet22k epochs and compares the main setting with independently increased weights.
  • Reconstruction Objective: Removing INR prediction and image reconstruction makes the model unable to reconstruct images, preventing its embeddings and TinToks from supporting image compression or generation.The reconstruction objective is therefore necessary for the stated unified representation in this ablation.
  • Experimental Settings: The ablation settings use encoder architectures corresponding to the ViT-B/16 and ViT-L/16 configurations described for the model.The supplied table reference identifies these as the encoder architecture settings.
  • Reconstruction Objective: Without the image reconstruction objective, TinToks classification is significantly worse, indicating synergy between reconstruction and recognition training.This comparison uses the standard system versus a system without INR prediction and its associated loss.

8. Detailed Settings

The detailed settings describe model sizes, decoder and INR components, training conventions, and fairness controls. Efficiency choices include strided coordinates and a single-layer decoder, while comparisons account for encoder, base-INR, and unique-parameter sizes.

  • Architectures: The ViT-B and ViT-L encoders contain 85.9M and 304M parameters, with decoder configurations varying by experiment.The appendix reports decoder settings separately and characterizes the displayed sizes as worst-case sizes.
  • Architectures: The base patchwise INR architecture is specified separately, including a learnable base-weight configuration for fθb.The supplied table caption identifies Table 16 as the source of these INR settings.
  • Efficiency: A single-layer decoder can achieve good reconstruction performance, motivating future work to reduce parameter count and training and inference costs.The passage presents efficiency improvement as a future hypothesis rather than a demonstrated result.
  • INR Implementation: Strided coordinates with stride 4 reduce coordinate-processing computation by 16×, followed by convolution and PixelShuffle upsampling to recover the full image.The model therefore does not use a pure MLP INR operating on every coordinate at inference.
  • Training: Training uses a base learning rate of 0.0005 rescaled by batch size, AdamW, cosine annealing after five warm-up epochs, gradient clipping at norm 0.01, and standard data augmentation.The standard setting trains for 50 epochs or the equivalent iteration count on ImageNet22k.
  • Fair Comparisons: Fair comparisons control encoder size, learnable base-INR size, and the smallest modulating weight-token parameter count.When tokens are downsampled before modulation, unique parameters are counted after downsampling rather than at the encoder dimension.

9. Diffusion with HUVR

HUVR diffusion experiments provide an initial proof of concept for generation, but the generated samples remain substantially below current generative state of the art. The authors attribute this mainly to an unoptimized diffusion model and process for HUVR’s token structure.

  • Generation Results: The generated results are significantly worse than current generative state-of-the-art methods.The authors frame the visual diffusion examples and VAE comparisons as evidence of potential rather than competitive generative quality.
  • Limitations and Future Work: The diffusion model and process were not optimized for HUVR’s architecture, which treats global and patch tokens differently from the DiT implementation.The HUVR setup uses a global token to combine with patch tokens for modulating patch INRs, whereas DiT handles them equivalently.
  • Generation Results: HUVR diffusion samples are intended to demonstrate generation potential, but they contain many degradations and artifacts.The experiment uses a DiT-XL trained on HUVR embeddings with TinTok dt = 256.
  • Experimental Setup: The proof-of-concept uses 4500k training steps, compared with 400k steps in the referenced HUVR table and 7000k steps for DiT-XL/2’s final model.The caption also identifies the experiment’s TinTok embedding dimension as 256.
Loading 2601.14256v1…