PaperlayerView exact revision on arXiv

Source-linked AI summary

Sparse-View Interpretable 3D Animal Behavior Representations for Neural Encoding and Decoding

Xinming Dai, Qihang Jin, Tianshu Tan, Baiyuan Chen, Hanrui Lyu, Lenny Aharon, Kyle Daruwalla, Xun Helen Hou, Matthew R. Whiteway, Liam Paninski, Yizi Zhang

arXiv:2609.36217v2cs.CV

TL;DR

Pose estimates omit behavioral complexity, while nonlinear video embeddings can be difficult to interpret, and sparse camera views challenge 3D reconstruction. SABLE uses geometric priors to learn 3D behavior representations from video. It reconstructs animal behavior from two sparse views and matches or exceeds prior SOTA in neural encoding and decoding, with zero-shot generalization to unseen animals.

  • Problem

    Pose estimates miss behavioral complexity, nonlinear video embeddings can be hard to interpret, and sparse minimally overlapping views challenge 3D animal reconstruction.

    Source: §1 · pp. 1–2

  • Method

    SABLE is a self-supervised framework that uses monocular-depth and pose-derived geometric priors to learn explicit 3D behavior representations from sparse multi-view video.

    Source: §1 · p. 2

  • Results

    SABLE reconstructs 3D behavior from two sparse IBL views without 3D labels and matches or exceeds prior SOTA in neural encoding and decoding across IBL and Cheese3D.

    Source: §1 · p. 2, §5.1 · p. 6

  • Takeaways & Limitations

    Once pretrained across animals, SABLE generalizes zero-shot to unseen animals and supports decoding 3D behavior from neural recordings.

    Source: §1 · pp. 2–3

    SABLE has so far been applied only to head-fixed mouse datasets; extension to freely moving animals and other species remains future work.

    Source: §6 · pp. 9–10

Abstract

from arXiv · show

A deeper understanding of brain function requires a precise, structured characterization of behavior. Yet, extracting behavioral representations from video in a form suitable for scientific analysis remains a fundamental challenge. Many prior studies represent behavior via pose estimation or nonlinear video embeddings. However, pose tracking discards rich information beyond predefined keypoints, while nonlinear video embeddings lack interpretability. We address this limitation with SABLE (Sparse-view Animal Behavior Latent Embeddings), a self-supervised framework that leverages a geometric inductive bias to learn behavior representations. By augmenting a multi-view transformer with priors from monocular depth and pose estimation, SABLE reconstructs 3D animal behavior from extremely sparse views while learning explicit 3D latent structure. Without ground-truth 3D labels, it reliably recovers 3D behavior from two-view videos, whereas state-of-the-art (SOTA) methods fail or yield degenerate solutions. Across the International Brain Lab and Cheese3D datasets, we demonstrate that SABLE learns 3D representations that match or exceed prior SOTA performance in neural encoding and decoding. Once pretrained across animals, SABLE serves as an off-the-shelf model that generalizes zero-shot to unseen animals without animal-specific calibration or retraining. Our method establishes 3D-aware video embeddings that capture complex behavior, opening new avenues for studying brain-behavior relationships.

1 INTRODUCTION

Behavior representations from video must preserve rich movement information while remaining interpretable, but pose estimates omit complexity and nonlinear embeddings can be hard to interpret. SABLE uses geometric priors to learn 3D behavior from sparse views, supporting neural analysis and generalization to unseen animals.

1 INTRODUCTION

  • Pose estimates miss behavioral complexity, nonlinear video embeddings can be difficult to interpret, and 2D projections obscure movements in three dimensions.The paper motivates dense, interpretable 3D video representations to address these limitations.Source: §1 · p. 1
  • In two-camera IBL data, existing 3D foundation models fail to recover coherent animal structure, whereas SABLE learns camera parameters and reconstructs it.Without the geometric loss, SABLE’s reconstruction degenerates.Source: Figure 1 · p. 2
  • SABLE reconstructs high-fidelity 3D point clouds from two sparse IBL views without 3D labels, while existing 3D models fail in this regime.Its geometric inductive biases help avoid degenerate reconstructions.Source: §1 · p. 2
  • SABLE combines masked modeling with monocular-depth and pose-derived geometric priors to learn explicit 3D behavioral representations from sparse multi-view video.It fine-tunes a multi-view transformer on minimally overlapping two-view videos.Source: §1 · p. 2
  • SABLE matches or exceeds prior SOTA in neural encoding and decoding across IBL and Cheese3D, and supports decoding 3D behavior from neural recordings.The paper reports that a pretrained model generalizes zero-shot to unseen animals.Source: §1 · pp. 2–3

2 RELATED WORK

Prior work offers tools for keypoint tracking, 3D reconstruction, and neural analysis, but sparse-view animal videos remain a difficult setting. Existing approaches often provide only sparse keypoints, require dense camera coverage, or produce unstable geometry on minimally overlapping views.

Sparse keypoint representations for 3D animal behavior analysis

  • Supervised multi-view methods such as DANNCE and Lightning Pose 3D integrate across views but output sparse keypoints rather than dense animal geometry or appearance.Earlier pipelines commonly estimate 2D keypoints per view and triangulate them into 3D.Source: §2 · p. 3

Large-scale foundation models for 3D reconstruction

  • Sparse, minimally overlapping animal views are out of distribution for 3D foundation models, often leading to unstable cameras, degenerate geometry, or 2D collapse.This motivates domain-adapted geometric priors for animal behavior reconstruction.Source: §2 · p. 3

Learning behavior representations for neural analysis

  • Existing behavior representations for neural analysis are limited to 2D or dense-view settings; Pose Splatter and BEAST3D require at least three to four cameras.BEAST emphasizes 2D semantic information, while MoReMouse’s synthetic-data reliance limits applicability to real-world sparse-view recordings.Source: §2 · p. 3

3 DATASET

The study evaluates SABLE on head-fixed mouse behavior and neural recordings from IBL, with Cheese3D used to assess generalizability. IBL recordings pair synchronized two-camera videos with neural activity across held-out sessions and animals.

3 DATASET

  • IBL records head-fixed mice discriminating left-versus-right visual stimuli and reporting choices by turning a response wheel; Cheese3D provides an additional head-fixed mouse dataset.Cheese3D is used to show generalizability beyond IBL.Source: §3 · p. 3
  • IBL neural data comprise 2,619 units from six probe insertions across three held-out mice, sampled from ten sessions across ten animals.Seven animals were used for pretraining and three held out for generalization testing.Source: §3 · p. 3
  • Synchronized left- and right-camera videos capture wheel-turning behavior with limited overlap, primarily showing the nose, forepaws, and wheel.Video is aligned at 60 Hz; training frames are selected to capture diverse behaviors.Source: §3 · p. 4

4 METHODS

SABLE uses geometric priors to learn 3D behavioral representations from sparse views through self-supervised reconstruction. Its architecture combines masked multi-view modeling and Gaussian rendering with reconstruction, perceptual, and geometric losses.

4 METHODS

  • Monocular-depth pseudo-point clouds are aligned across views using pose keypoints, supplying geometric regularization that discourages trivial 2D shortcuts; multi-animal pretraining supports zero-shot transfer to unseen animals.SABLE requires no dataset-specific adaptation for that transfer.Source: §4 · p. 4, Figure 2 · p. 4

4.1 MODEL ARCHITECTURE

  • Two E-RayZer-style transformers estimate camera parameters and fuse ray and image tokens; patch-level masking preserves cross-view context, and embeddings parameterize pixel-aligned Gaussians rendered into views.The transformers are initialized from E-RayZer (Zhao et al., 2026).Source: §4.1 · p. 5

4.2 TRAINING OBJECTIVES

  • Training combines masked pixel reconstruction, VGG-19-feature perceptual loss, and normalized predicted-to-pseudo point-cloud geometric consistency in a weighted objective.The weights apply to the reconstruction, perceptual, and geometric loss terms.Source: §4.2 · p. 6

5 RESULTS

SABLE reconstructs coherent 3D mouse structure from sparse views where existing 3D models fail, and its representations support neural decoding and encoding. It also outperforms video-embedding baselines in decoding and neural prediction.

5.1 3D ANIMAL BEHAVIOR RECONSTRUCTION

  • SABLE reconstructs coherent 3D mouse structure from two minimally overlapping IBL views, while existing large-scale 3D models fail to recover coherent geometry.DA3 misaligns views, VGGT collapses to a 2D solution, and E-RayZer produces flattened point clouds even after fine-tuning; SABLE uses a pseudo-point-cloud geometry loss.Source: §5.1 · p. 6

5.2 NEURAL DECODING

  • SABLE surpasses BEAST and ResNet AE in video-frame decoding quality on both PSNR and SSIM.Neural activity is used to predict video embeddings, which are decoded into images and 3D point clouds; the decoded point clouds capture fine-grained paw movements.Source: Figure 3 · p. 7, §5.2 · p. 8

5.1 3D ANIMAL BEHAVIOR RECONSTRUCTION

  • SABLE’s interpretable 3D representations support analysis of animal pose and movement dynamics alongside downstream neural encoding and decoding.The neural decoding evaluation uses IBL, which provides at least 708 neurons per session, compared with 8 in Cheese3D.Source: §5.1 · p. 7, §5.2 · p. 7

5.3 NEURAL ENCODING

  • SABLE achieves the best neural prediction performance among the evaluated video representations and improves prediction quality for most neurons across held-out sessions.PCA outperforms keypoints, and ResNet AE outperforms PCA; the authors suggest explicit 3D structure adds information beyond conventional 2D video embeddings.Source: §5.3 · p. 9, Figure 4 · p. 9

6 DISCUSSION

SABLE’s experiments are limited to head-fixed mouse datasets, leaving freely moving animals and other species for future work. The authors also identify further use of synthesized views as a possible route to improved neural encoding.

6 DISCUSSION

  • SABLE has been applied only to head-fixed mouse datasets; extending it to freely moving animals and other species remains future work.The authors also suggest using synthesized views to extract additional behavioral keypoints and potentially improve neural encoding.Source: §6 · pp. 9–10

AI USE STATEMENT

The appendix describes procedures for selecting training frames, estimating pose and depth, and comparing sparse-view reconstruction on Cheese3D; the AI-use statement reports software debugging and writing assistance.

AI USE STATEMENT

  • The authors used generative AI tools for software debugging, grammar checking, and writing improvements, and reviewed the assisted content.The authors take responsibility for the final work.Source: AI USE STATEMENT · p. 10

A.1.1 PCA + k-MEANS FRAME SELECTION

  • A PCA-and-k-means frame-selection procedure selects 1,000 synchronized frame pairs per session to provide a compact, behaviorally diverse training set.It filters for high-motion frames, clusters them, and expands anchor indices by one frame on either side; evaluation frames are aligned with neural data.Source: §A.1.1 · p. 13

A.1.2 LIGHTNING POSE KEYPOINTS

  • Three Lightning Pose 3D models are trained per session on 200 manually labeled frames each, then combined through uncertainty filtering and pose-diversity distillation.The retained pseudo-labeled frames augment the original labels to retrain one model for full-video keypoint trajectories.Source: §A.1.2 · pp. 13–14
  • The nonlinear multi-view Ensemble Kalman Smoother uses temporal continuity and multi-view geometry to downweight inconsistent, overconfident predictions.Variance inflation increases observation noise for views that disagree with the remaining views.Source: §A.1.2 · pp. 13–14

A.1.3 VIDEO DEPTH EXTRACTION

  • SABLE uses frozen Video Depth Anything estimates to provide temporally consistent relative-depth maps aligned with the input RGB images.Relative rather than metric depth is used because absolute scene scale is unnecessary for the reconstruction objective.Source: §A.1.3 · p. 14

A.2 CHEESE3D: NOVEL-VIEW SYNTHESIS AND NEURAL ENCODING

  • On Cheese3D’s selected sparse top-right/top-left view pair, SABLE recovers coherent mouse 3D with consistent cross-view geometry, foreground-background depth separation, and no degenerate solution, unlike existing 3D foundation models.Source: Figure 5 · p. 14

A.2.1 DATASET

Cheese3D provides a six-view mouse dataset used to test sparse-view reconstruction and neural encoding. SABLE outperforms E-RayZer on held-out-view synthesis and ResNet AE and BEAST in neural prediction quality.

A.2.1 DATASET

  • Cheese3D contains six synchronized views of head-fixed mice recorded at 100 Hz; videos are downsampled to 60 Hz, with ten sessions used for training and one for neural evaluation.The ten training sessions provide approximately 5,500 synchronized frame pairs.Source: §A.2.1 · pp. 14–15

A.2.5 NOVEL VIEW SYNTHESIS

  • With limited-overlap top-left and top-right inputs, SABLE reconstructs coherent mouse structure, while existing 3D foundation models produce inconsistent or degenerate reconstructions.Because Cheese3D lacks ground-truth 3D labels, reconstruction is also assessed by rendering the held-out top-center view and comparing it with the real image.Source: §A.2.1 · p. 15, §A.2.5 · p. 15

A.2.2 LIGHTNING POSE KEYPOINTS

  • An ensemble of three single-view LP3D models trained on 665 labeled instances supplies keypoints, retaining frames with median likelihood above 0.6 in at least two views.Triangulated keypoints are projected back into all six views to produce geometrically consistent estimates.Source: §A.2.2 · p. 15

A.2.3 KEYPOINTS + k-MEANS

  • Pose-diverse frames are selected by clustering flattened 3D keypoints, then removing the 25% with the largest reprojection errors; the final sets contain 450 training and 150 test instances.Approximately 50–55 representative frames per session are selected before reprojection-error filtering.Source: §A.2.3 · p. 15

A.2.4 VIDEO DEPTH EXTRACTION

  • Video Depth Anything independently produces temporally consistent relative-depth maps from Cheese3D RGB videos resized to 320 × 320 pixels without cropping.Source: §A.2.4 · p. 15

A.2.5 NOVEL VIEW SYNTHESIS; A.2.6 NEURAL ENCODING

  • SABLE outperforms E-RayZer on held-out-view PSNR and SSIM, and ResNet AE and BEAST on Cheese3D neural prediction quality.The held-out top-center view is rendered from the top-left and top-right inputs; neural encoding is evaluated on a held-out session.Source: §A.2.5 · p. 16, Table 1 · p. 16, §A.2.6 · p. 17

B METRICS AND BASELINES

The section defines neural encoding and image-reconstruction metrics, adds a temporally shuffled embedding baseline, and tests SABLE with different numbers of camera views. SABLE remains usable from one view after fine-tuning and from three Cheese3D views.

B.1 BITS PER SPIKE

  • Bits per spike measures predictive log-likelihood improvement over a mean-rate null model, normalized by observed spikes and expressed in base-2 units.A value of zero corresponds to the mean-rate baseline.Source: §B.1 · p. 17

B.2 RANDOM-EMBEDDING BASELINE FOR NEURAL ENCODING

  • Randomly permuted SABLE embeddings achieve near-chance encoding performance at 0 bps, while the models’ positive BPS values indicate behavior-related predictive information.The permutation destroys temporal structure while preserving each frame’s embedding content.Source: §B.2 · p. 17

B.3 PEAK SIGNAL-TO-NOISE RATIO (PSNR)

  • PSNR measures pixel-level reconstruction fidelity; with images normalized to [0, 1], higher PSNR corresponds to lower mean-squared error.It primarily reflects pixel-wise accuracy.Source: §B.3 · pp. 17–18

B.4 STRUCTURAL SIMILARITY INDEX MEASURE (SSIM)

  • SSIM compares local luminance, contrast, and structural statistics, and is designed to better align with perceptual image quality than PSNR.The reported calculation averages 11×11 Gaussian windows over spatial locations and channels.Source: §B.4 · p. 18

B.5 FOREGROUND-ONLY PSNR AND SSIM

  • Foreground-only PSNR and SSIM reduce confounding from decoder differences in static background regions by evaluating masked mouse frames.SAM3 masks are applied to both ground-truth and reconstructed frames, leaving uniformly black backgrounds.Source: §B.5 · p. 18

C GENERALIZATION ACROSS VARYING NUMBERS OF CAMERA VIEWS

  • Fine-tuned SABLE reconstructs consistent 3D point clouds from one IBL input view, and reconstructs Cheese3D point clouds using three of six views.The single-view setting uses cross-view prediction fine-tuning; the Cheese3D three-view setting retains sparse inputs despite the available six views.Source: C GENERALIZATION ACROSS VARYING… · p. 18

D.1 MODEL SIZE

The ablations test whether SABLE’s downstream advantage can be explained by larger baseline models or by loss-weight choices. Neither scaling the baselines nor removing geometric regularization yields a better overall result while preserving coherent 3D geometry.

D.1 MODEL SIZE

  • Increasing ResNet-AE or BEAST model size does not improve performance, and SABLE outperforms their best variants across encoding and decoding metrics.This indicates the reported advantage is not simply explained by scaling the backbone.Source: §D.1 · p. 19, Table 3 · p. 19

D.2 LOSS WEIGHTING

  • Among loss-weight combinations that preserve coherent 3D geometry, the chosen weighting achieves the highest reconstructed-view PSNR.Removing Lgeom collapses the point clouds, while increasing Lrecon introduces noticeable geometric distortions.Source: §D.2 · p. 19, Table 4 · p. 20

E.1 SABLE

The appendix describes the SABLE model, the ResNet autoencoder and BEAST representation baselines, and the temporal convolutional network used for neural encoding and decoding.

E.1 SABLE

  • SABLE builds on the E-RayZer backbone and uses patch-level masking across both views, with loss weights λrecon = 1, λpercep = 0.3, and λgeom = 1.Source: §E.1 · p. 20, Table 5 · p. 20

E.2 RESNET AUTOENCODER

  • The ResNet autoencoder encodes each frame into a low-dimensional latent vector through a ResNet encoder and MLP bottleneck, then reconstructs the input with a symmetric decoder.Source: §E.2 · p. 20, Table 6 · p. 20

E.3 BEAST

  • BEAST uses a ViT-MAE with a ViT-B ImageNet-pretrained backbone, masked image reconstruction, and temporal contrastive learning.It applies 75% random patch masking and an InfoNCE loss for nearby-frame consistency.Source: §E.3 · pp. 20–21, Table 7 · p. 21

E.4 TCN

  • The TCN processes behavioral features for encoding or neural activity for decoding through temporal convolutional layers, then predicts neural activity or behavioral features with a linear readout.Multiple receptive-field scales capture temporal dynamics across different timescales.Source: §E.4 · p. 21, Table 8 · p. 21

F TRAINING DETAILS

This section describes training data, objectives, and optimization for SABLE and three comparison models. SABLE uses synchronized two-camera views and reconstruction targets; the baselines use dataset-matched frame pairs or validation-selected configurations.

F.1 SABLE

  • SABLE is further pretrained on approximately 7,000 IBL frame pairs from 7 sessions and 5,500 Cheese3D frame pairs from 10 sessions, using synchronized camera views and reconstruction targets.Its Gaussian splatting decoder reconstructs 3D structure from multi-view transformer embeddings while jointly inferring camera geometry.Source: §F.1 · p. 21
  • SABLE combines image reconstruction, perceptual, and geometric consistency losses, training for 100 epochs with AdamW on a single A40 or A100 GPU.Training takes approximately 17 hours per session.Source: §F.1 · p. 22

F.2 RESNET AUTOENCODER

  • The ResNet autoencoder trains on the same IBL and Cheese3D frame pairs for 160 and 400 epochs, respectively, taking approximately 0.5 hours per session on one A100.It uses Adam, weight decay 0.05, and the BEAST repository’s default image augmentation pipeline.Source: §F.2 · p. 22

F.3 BEAST

  • The BEAST ViT-MAE backbone trains on the same dataset frame pairs, using a 0.75 mask ratio and running for 160 IBL or 400 Cheese3D epochs.Training takes approximately 1 hour per session on one A100.Source: §F.3 · p. 22

F.4 TCN

  • TCN configurations are selected from 30 random Ray Tune trials using validation BPS for encoding or R2 for decoding, with test-set results reported afterward.Each TCN trains for 300 epochs with MSE loss and a temporal-filter smoothing penalty of 0.5; the full search takes approximately 0.5 hours on two GPUs.Source: §F.4 · p. 22
Loading 2609.36217v2…