PaperlayerView exact revision on arXiv

Source-linked AI summary

Brain-Inspired Stochastic Joint Embedding Representation Learning

Makoto Yamada, Kian Ming A. Chai, Ayoub Rhim, Satoki Ishikawa, Mohammad Sabokrou, Yao-Hung Hubert Tsai

arXiv:2505.11129v2cs.CVcs.AIcs.LG

TL;DR

Existing SSL methods have not fully incorporated biological visual processing or learning from continuous visual streams without strong augmentation. PhiNet v2 addresses this gap with a Transformer-based, variationally grounded architecture, achieving competitive or superior benchmark performance while avoiding additional components used by RSP.

  • Problem

    Existing vision SSL methods have largely overlooked brain-inspired sequential processing, while PhiNet v1 was not designed for sequential learning and relied on strong augmentation.

  • Method

    PhiNet v2 uses Transformer-based modules, sequential visual input, representation-level alignment, and a variationally motivated stochastic design.

  • Results

    PhiNet v2 achieves competitive or superior performance against strong methods including DINO [Caron et al., 2021], CropMAE [Eymaël et al., 2024], and RSP [Jang et al., 2024], including 60.1 J&Fm on DAVIS versus RSP’s 57.7 without MAE.

  • Takeaways & Limitations

    PhiNet v2 supports brain-inspired sequential video representation learning without strong augmentation or an auxiliary MAE module, simplifying the architecture and hyperparameter tuning.

  • Takeaways & Limitations

    PhiNet v2’s predecessor PhiNet v1 did not use actual temporal information and relied on augmentation and stop-gradient operations to simulate temporal differences.

Abstract

from arXiv · show

Representation learning is one of the key research topics in machine learning, and the framework of self-supervised learning (SSL) has revolutionized computer vision. However, these approaches have not yet fully leveraged insights from biological visual processing systems. In this paper, we introduce PhiNet v2, a novel architecture that processes temporal visual input (i.e., sequences of images) without relying on strong data augmentation, enabling it to learn robust visual representations in a manner similar to human visual processing. Our learning objective is derived from variational inference. Through extensive experiments, we demonstrate that PhiNet v2 achieves competitive performance compared to state-of-the-art vision representation models, including RSP and CropMAE, while retaining the ability to learn effectively from sequential input without strong data augmentation. This work represents a step toward more biologically plausible computer vision systems that process visual information in a manner more aligned with human cognitive processes.

1 Introduction

PhiNet v2 extends brain-inspired representation learning to sequential visual input with a Transformer-based architecture and reduced reliance on strong augmentation. It combines probabilistic learning with architectural changes intended to improve stability and representation quality.

  • Motivation: Strong data augmentation limits PhiNet v1’s generalizability for sequential learning, motivating a model that can learn from continuous visual streams directly.PhiNet v1 was not designed for sequential learning, and effective augmentations are harder to define for sequential input.
  • PhiNet v2: PhiNet v2 processes sequential visual input with Transformer-based modules, replacing PhiNet v1’s ResNet encoder and predictor.The design also incorporates an uncertainty model to capture stochasticity in visual input.
  • Contributions: PhiNet v2 formulates a variational learning objective and incorporates brain-inspired principles from biological visual processing.Its architecture draws on the original PhiNet framework while adding Transformer modules and stochasticity modeling.
  • Results: Ablation studies indicate that each PhiNet v2 architectural component contributes to stable and robust training.This supports the importance of the complete design rather than an isolated module.
  • Results: PhiNet v2 achieves competitive or superior performance against strong pretraining methods including DINO [Caron et al., 2021], SiamMAE [Gupta et al., 2023], CropMAE [Eymaël et al., 2024], and RSP [Jang et al., 2024].The introduction presents this comparison as an outcome across standard computer vision tasks.

2 Related work

Related work progresses from contrastive and non-contrastive image SSL to Transformer-based and sequential video representation learning. PhiNet v1 differs from reconstruction-focused methods by predicting latent representations, but remains limited by simulated rather than actual temporal information.

  • Contrastive learning: Contrastive SSL learns representations by bringing augmented views together and separating dissimilar examples, with SimCLR [Chen et al., 2020a] and MoCo [He et al., 2020] as representative methods.MoCo uses momentum encoding and a queue-based dictionary, while MoCo v2 [Chen et al., 2020b] adds stronger augmentations and architectural changes.
  • Non-contrastive learning: Non-contrastive methods such as BYOL, SimSiam [Chen and He, 2021], and Barlow Twins [Zbontar et al., 2021] avoid negative samples through asymmetric architectures or redundancy reduction.These approaches are typically implemented with convolutional backbones such as ResNet.
  • Transformer-based SSL: Transformer-based SSL includes DINO [Caron et al., 2021], MAE [He et al., 2022], and I-JEPA [Assran et al., 2023], which use self-distillation, masked reconstruction, or contextual latent prediction.These methods extend SSL beyond earlier convolutional approaches.
  • Sequential learning: Sequential visual representation learning includes VideoMAE [Tong et al., 2022], SiamMAE [Gupta et al., 2023], CropMAE [Eymaël et al., 2024], RSP [Jang et al., 2024], and V-JEPA [Bardes et al., 2024].Most listed methods use pixel-level prediction, while V-JEPA predicts in representation space.
  • Brain-inspired representation learning: PhiNet v1 predicts latent representations rather than reconstructing images, but it uses a ResNet encoder and simulates temporal differences with augmentation and stop-gradient operations.Its performance with modern Transformer architectures remained unclear.

3 Background

This section grounds PhiNet in hippocampal circuitry, temporal prediction, and complementary learning systems, then identifies PhiNet v1's architecture, strengths, and limitations. PhiNet v1 applies these brain-inspired ideas to image-based self-supervised learning but does not use actual temporal information.

  • 3.1 Neuroscience findings: The hippocampus comprises CA1, CA3, and dentate gyrus, with sensory signals reaching it through entorhinal-cortex pathways.The dentate gyrus is assumed to act as a delay operator in PhiNet models.
  • 3.1 Neuroscience findings: The temporal prediction hypothesis assigns CA3 prediction of upcoming inputs and CA1 comparison of predictions with future inputs to generate learning errors.Experimental recordings are described as consistent with CA3's predictive role and synchronized DG–CA1 activity.
  • 3.1 Neuroscience findings: Complementary Learning Systems theory separates rapid, sparse hippocampal learning from gradual neocortical integration of distributed representations.This organization supports quickly storing new memories without overwriting existing knowledge, followed by later consolidation.
  • 3.2 PhiNet v1: PhiNet v1 combines a Siamese ResNet encoder, two predictors, and an EMA-updated slow encoder to implement temporal prediction and CLS-inspired learning.It is trained on images with heavy cropping, flipping, blurring, and color-jitter augmentation.
  • 3.2 PhiNet v1: PhiNet v1 is robust to weight-decay choices and outperforms existing representation-learning methods in continual-learning scenarios [Ishikawa et al., 2025].Its limitations are that it simulates temporal differences through augmentation and stop-gradient operations and leaves Transformer performance unclear.
  • 3.3 PhiNet v2: PhiNet v2 combines PhiNet v1 and RSP architectures, while the RSP MAE module is not required by PhiNet v2.The figure compares the architectural designs of RSP and PhiNet v2.

4 PhiNet v2

PhiNet v2 extends brain-inspired representation learning to sequential visual input with Transformer modules and a variational objective. Its design combines future-representation prediction, stochastic latent variables, and slow-learning dynamics to address collapse and learning stability.

  • Architecture and comparison with RSP: PhiNet v2 predicts future representations from current video frames and aligns them with long-term target representations, replacing RSP’s pixel prediction with representation-level learning.The model uses Transformer-based encoder and predictor modules and does not require RSP’s auxiliary masked autoencoder.
  • Sequential input and brain-inspired components: The model is trained on related video-frame sequences, with ViT encoders producing tokenized representations for current and future frames.The future representation includes injected noise, while the predictor forecasts the future from the current representation.
  • Brain-inspired interpretation: PhiNet v2’s brain-inspired pathway maps encoding to entorhinal cortex, prediction to CA3, discrepancy computation to CA1, and EMA-based slow learning to neocortex.The CA1-related pathway uses a probability divergence for Sim-1, while the neocortical pathway aligns representations through Sim-2.
  • Variational objective: The variational formulation maximizes a lower bound on encoded video likelihood using skipped Markov components, latent-variable stochasticity, and future prediction from the current frame.The latent-variable objective includes a KL divergence between an approximate posterior and a conditional prior, while the mixing distribution q(ϵ) is fixed rather than optimized.
  • Learning considerations: Training must prevent representational collapse and slow learning because gradients from multiple encoder paths can negate one another.PhiNet v2 addresses these issues with KL balancing, path suppression, stop-gradient updates to the long-term encoder, EMA updates, and variance scaling tied to encoder output size and latent dimensionality.

5 Experiments

PhiNet v2 is evaluated against established video representation methods and through robustness and ablation studies. It performs competitively while avoiding some auxiliary components and retaining robustness under noisy inputs.

  • 5 Experiments: PhiNet v2 achieves performance comparable to RSP and CropMAE across video label-propagation benchmarks.Table 1 covers DAVIS, VIP, and JHMDB using ViT-S/16 representations pretrained on Kinetics-400 for 400 epochs.
  • 5.1 Results: 60.1 J&Fm on DAVIS lets PhiNet v2 outperform RSP’s 57.7 without an auxiliary MAE module.This also removes the need to tune additional MAE-related hyperparameters.
  • 5.1 Results: PhiNet v2 is more robust than RSP under noisy DAVIS inputs and compares favorably on the other evaluated cases.The robustness evaluation adds Gaussian noise to input images; Table 2 reports multiple noise levels across DAVIS, VIP, and JHMDB.
  • 5.2 Ablation Study: The proposed component combination is important for superior performance, with symmetric loss improving final J&Fm by 0.4 to 0.9.The ablations also show that the predictor slightly improves performance, SG-prior is beneficial for final performance, and β = 0.01 provides a good trade-off.
  • 5.2 Ablation Study: Removing EMA causes representational collapse and degraded performance, whereas EMA stabilizes training and enables effective representation learning.The ablation identifies EMA as an essential stabilization mechanism for PhiNet v2.
  • 5.2 Ablation Study: Performance remains stable across batch sizes, including smaller batches that enable training on limited hardware such as V100 GPUs.The evaluated batch sizes are 192, 384, 768, and 1536.

6 Conclusion

PhiNet v2 combines brain-inspired modeling with latent representation alignment and variational inference. Across multiple benchmarks, it performs competitively without auxiliary components, supporting its practicality for video representation learning.

  • 6 Conclusion: PhiNet v2 aligns latent representations and achieves comparable or superior results to RSP without an auxiliary MAE module.This simplifies the architecture and removes extensive additional hyperparameter tuning.
  • 6 Conclusion: Across multiple benchmarks, PhiNet v2 performs competitively with state-of-the-art methods for video representation learning.The authors frame this as evidence supporting neuroscience-inspired approaches for scalable and generalizable video representation learning.

Broder Impact

PhiNet v2 connects hippocampal circuitry, temporal prediction, and Complementary Learning Systems theory while comparing competitively with JEPA- and DINO-based approaches.

  • Broder Impact: PhiNet v2 offers a coherent connection between hippocampal modeling and Complementary Learning Systems theory.Its architecture is inspired by hippocampal biological circuitry and the temporal prediction hypothesis.
  • Broder Impact: PhiNet v2 demonstrates competitive performance relative to JEPA- and DINO-based approaches that use large-scale training.The passage places this comparison within the paper’s neuroscience-inspired modeling perspective.

A.1 Probabilistic derivation of PhiNet v2

The probabilistic derivation models encoded video likelihood with skipped temporal dependencies and a stochastic latent variable. PhiNet v2 learns the encoder by maximizing a variational lower bound whose terms correspond to its brain-inspired losses.

  • A.1 Probabilistic derivation of PhiNet v2: The encoder is learned by maximizing a lower bound on the likelihood of encoded video sequences.The objective is designed to support variational learning while remaining brain-inspired and simple.
  • A.1 Probabilistic derivation of PhiNet v2: The temporal likelihood uses a uniform mixture of skipped Markov models, with each component conditioning the current representation on an earlier frame.This models sequential dependencies through multiple temporal offsets.
  • A.1 Probabilistic derivation of PhiNet v2: A latent variable r introduces stochastic dependence into the conditional encoded-frame likelihood for video that is both variable and structured.The derivation applies an evidence lower bound and KL divergence between approximate posterior and conditional prior.
  • A.1 Probabilistic derivation of PhiNet v2: The combined KL divergences form the Sim-1 loss, while the conditional likelihood of the future representation forms the basis of Sim-2.The latent distributions are conditioned on encoder variables and use amortized inference.
  • A.1 Probabilistic derivation of PhiNet v2: The mixing distribution q(ϵ) is fixed rather than optimized, preserving biological plausibility without substantially complicating the model.The authors present the resulting exposition as a probabilistic derivation not previously described in this manner.

A.2 Implementation of PhiNet v2

PhiNet v2 uses a Vision Transformer encoder and specialized predictors to process video representations. Its implementation combines cross-attention, linear prediction, and [CLS]-token-based distribution networks.

  • EC: PhiNet v2 uses a Vision Transformer backbone to encode non-overlapping patches from each video frame with positional embeddings and a [CLS] token.The resulting tokens are processed by the ViT to obtain representations.
  • CA3: CA3 primarily uses a linear predictor that maps representations through a learned d × d transformation.The authors report that linear predictors outperform more complex alternatives, which may otherwise overfit.
  • CA1: The CA1 predictor is ViT-based with cross-attention, using predicted representations and latent inputs as keys and values.Queries are learned in the first transformer block and derive from preceding outputs in later blocks.
  • Distribution networks: The distribution networks for p and q use single-hidden-layer ReLU networks whose inputs are restricted to [CLS] tokens.

B Limitations

The paper identifies biological fidelity, patch-based visual processing, and linear CA3 modeling as limitations of PhiNet v2. Its pseudocode implements stochastic posterior sampling, prior estimation, and separate hippocampal and neocortical losses.

  • Limitations: PhiNet v2 remains an engineering-driven approach because its alignment with actual human neural mechanisms is unclear.The paper specifically notes that its Vision Transformer patches may not faithfully approximate human vision and that its linear CA3 differs from biological recurrence.
  • Algorithm 1: The pseudocode defines the forward pass over source and target images before extracting encoder embeddings and an EMA target representation.
  • Algorithm 1: The forward pass samples a posterior distribution from current and target representations, then estimates a prior from the current representation alone.The posterior is formed from CA3 output and target prediction features, while the prior uses detached current-image features.
  • Algorithm 1: PhiNet v2 optimizes a hippocampal KL loss alongside a neocortical prediction loss.The KL term compares posterior and prior logits, while the prediction term uses mean squared error between predicted and target latent representations.

C Extended Related Work

PhiNet v2 connects hierarchical predictive coding with hippocampal temporal prediction in a sequential learning framework. The related evaluation materials cover video label propagation, robustness to noise, and implementation settings across several benchmarks.

  • Evaluation scope: Table 5 evaluates video label propagation across DAVIS segmentation, VIP part segmentation, and JHMDB pose tracking using Kinetics-400 pretraining for 400 epochs.The comparison includes fairness caveats because some VIP and JHMDB evaluations use different code, while RSP is retrained in the authors’ environment.
  • Noise robustness: Table 6 compares video segmentation with and without noise ϵ on DAVIS using ViT-small models pretrained on Kinetics-400 for 400 epochs.The reported setting uses β = 0.01 and batch size 768.
  • Extended related work: PhiNet v2 integrates hierarchical predictive coding and hippocampal temporal prediction to process video streams without heavy data augmentation.This unified sequential framework contrasts with prior models that typically emphasize cortical predictive coding or image-based CLS applications.
  • Additional analyses: The study also reports ablations involving batch sizes, regularization parameters, EMA parameters, training statistics, and qualitative propagation results.These materials include Tables 7–9, Figure 3, and qualitative results on DAVIS, VIP, and occluded video instance segmentation datasets.
Loading 2505.11129v2…