Source-linked AI summary
Hyper-RED: Scalable Event Pre-training via Semantic Hypergraph Distillation
Meisen Wang, Zhiqiang Tian, Wei Bao, Chengjie Wang, Shaoyi Du, Siqi Li
TL;DR
Sparse event data and limited annotations make scalable event representation learning challenging, while rigid image–event matching can struggle across the modality gap. Hyper-RED transfers high-order semantic structures through hypergraphs, and reports state-of-the-art overall performance across three tasks and five datasets.
Problem
Event streams are sparse and asynchronous, annotated event datasets are limited, and rigid image–event correspondence is fragile across modalities.
Source: §1 · pp. 1–2Method
Results
Across three tasks and five datasets, Hyper-RED achieves state-of-the-art overall performance, scales from ViT-S to ViT-L, and transfers strongly across linear probing, few-shot fine-tuning, and full supervision.
Source: §6 · pp. 7–8Takeaways & Limitations
The results establish semantic hypergraphs as structured knowledge carriers for scalable and annotation-efficient event representation learning.
Source: §6 · p. 8
Abstract
from arXiv · showhide
Event cameras have shown great potential for robust visual perception, yet scaling event representation learning remains challenging due to the scarcity of large-scale annotated event data. Pretrained image models provide scalable semantic supervision, but existing image-to-event methods rely on rigid pixel-wise or token-wise alignment that overlooks modality discrepancies in texture, density, and appearance, potentially causing semantic collapse and limiting transferability. To address this issue, we propose Hyper-RED, a simple, painless, and scalable image-to-event pretraining framework that transfers high-order semantic structures from images to events. Hyper-RED uses hypergraphs to model and align high-order semantic associations among multiple image and event tokens, enabling cross-modal knowledge transfer while accommodating modality-specific differences rather than enforcing rigid one-to-one correspondence. Specifically, given a paired event--image sample, Hyper-RED leverages DINOv3 to extract spatial token representations and constructs image, event, and cross-modal semantic hypergraphs, where each hyperedge connects multiple semantically correlated tokens. We further introduce a hypergraph relational distillation loss that imposes complementary intra- and cross-modal constraints, enabling the event encoder to inherit image-derived semantic organization while preserving local relational consistency and event-specific characteristics. Experiments on three tasks across five event datasets demonstrate consistent scaling from ViT-S to ViT-L and state-of-the-art performance (Fig.1). The code is available at: https://github.com/meisenwang/Hyper--RED.
1 Introduction
Event data are sparse and annotations scarce, while rigid image–event alignment can fail to preserve semantic organization across modalities. Hyper-RED uses semantic hypergraphs to transfer high-order structure, with reported state-of-the-art overall performance across three tasks and five datasets.
1 Introduction
- Event cameras offer high temporal resolution, high dynamic range, low latency, and low power, but sparse asynchronous streams and limited annotations make scalable learning challenging.Pretrained image models provide semantic representations as a scalable alternative to annotation-intensive event pre-training.Source: §1 · p. 1
- Across three tasks and five datasets, Hyper-RED achieves state-of-the-art overall performance and scales consistently from ViT-S to ViT-L.The overall comparison covers ten metrics across datasets using the same backbone model.Source: Figure 1 · p. 1, §1 · p. 2
- Rigid alignment can make event-encoder similarity patterns more diffuse and reduce class separability, while pairwise graph relations may lose high-order semantic associations.The paper frames this as a bottleneck to scalable image-to-event pre-training.Source: §1 · p. 2
- Hyper-RED constructs image, event, and cross-modal semantic hypergraphs and applies complementary intra-modal and cross-modal distillation constraints.Hyperedges group semantically correlated tokens, transferring region-level organization without rigid one-to-one correspondence.Source: §1 · p. 2
2 Related Work
Prior event pre-training uses event-only self-supervision or transfers knowledge from pretrained image models, but existing transfer objectives often remain point-wise or pairwise. Hypergraphs model group-wise relations, though their role as cross-modal knowledge carriers remains underexplored.
2.1 Event-Based Representation Pre-training
- Event-only methods reconstruct masked inputs or model long sequences, while image-to-event methods transfer pretrained image knowledge through feature distillation, prompt fusion, masked modeling, generative pre-training, and graph alignment.Event-only pre-training avoids paired data and manual annotations, whereas cross-modal transfer draws on image-model semantic priors; existing objectives still provide insufficient region-level organization.Source: §2.1 · pp. 2–3
- Existing image-to-event distillation relies on point-wise imitation, task-specific proxy supervision, or pairwise relations that are sensitive to the modality gap.The section identifies robust alignment beyond point-wise and pairwise relations as a need for scalable pre-training.Source: §2.1 · p. 3
2.2 Hypergraph Representation Learning
- Hypergraphs connect multiple nodes in one hyperedge, and prior work uses them for representation learning, dynamic structure learning, message passing, and multilateral metric-learning objectives.The cited work primarily applies hypergraphs within a single modality, motivating their investigation as structured carriers for image-to-event distillation.Source: §2.2 · p. 3
3 Preliminary
The preliminary setup pairs an event representation with a spatially and temporally aligned image of the same scene. A trainable event encoder and frozen image teacher produce spatial-token features, while hyperedges represent semantic groups among tokens.
3 Preliminary
- An event representation and its spatially and temporally aligned image describe the same scene but differ in sensing characteristics and available visual cues.Images contain dense appearance and texture cues; events capture sparse brightness changes caused by motion or illumination variation.Source: §3 · p. 3
- A trainable event encoder and frozen image teacher extract spatial-token features from an aligned pair, with corresponding tokens sharing a spatial layout.Conventional distillation aligns corresponding tokens directly, whereas Hyper-RED transfers organization among multiple tokens.Source: §3 · p. 3
- In the hypergraph formulation, each node is a spatial token, each hyperedge is a semantic group, and incidence weights quantify connected tokens’ relative contributions.A hyperedge connects multiple nodes to capture group-wise dependencies beyond ordinary pairwise graph edges.Source: §3 · p. 3
4 Methodology
Hyper-RED transfers image-derived semantic organization to event representations by grouping related tokens into teacher-guided hypergraphs. Its complementary intra- and cross-modal losses align group structure while supporting ViT-S/B/L encoders without scale-specific modifications.
4 Methodology
- Hyper-RED transfers teacher-defined semantic organization through image, event, and cross-modal hypergraphs, using complementary structure losses across ViT-S/B/L encoders.The formulation operates on normalized token affinities and requires no scale-specific modifications.Source: §4 · p. 3
4.1 Semantic Hypergraph Construction
- Image-token anchors and their most-correlated neighbors define shared semantic groups, letting each branch compare relation strengths within the same topology.The teacher-selected anchor and member indices provide the common topology.Source: §4.1 · p. 4
4.2 High-Order Structure Distillation
- The intra-modal loss aligns event-token organization with teacher memberships and prototypes, while the cross-modal loss anchors event groups to image semantics.Together, the objectives provide high-order supervision beyond token-wise correspondence and pairwise relations.Source: §4.2 · pp. 4–5
5 Experiments
Experiments evaluate Hyper-RED on object recognition, depth estimation, and semantic segmentation, including capacity scaling and annotation-efficient transfer. Results show strong performance across tasks and supervision settings, while ablations support the complementary roles of its high-order objectives.
5.1 Experimental Setup
- Pretraining uses eight real-world and five simulated event sources; matched DINOv3 ViT-S/B/L architectures initialize a frozen image teacher and trainable event encoder.Evaluation uses shared task-specific protocols for full supervision, linear probing, and few-shot fine-tuning with 1%, 5%, 10%, or 20% of annotations.Source: §5.1 · p. 5
5.2 Depth Estimation
- On MVSEC-Depth, ViT-L/16 beats ScaleEvent by 0.182 in δ1, 0.113 in δ2, and 0.049 in δ3, while reducing AbsRel by 0.116, RMSE by 2.171, and RMSElog by 0.141.On DSEC-Depth, it raises δ1 from 0.896 to 0.904 and reduces AbsRel from 0.101 to 0.096 and RMSE from 3.694 to 3.624.Source: §5.2 · p. 6, Table 3 · p. 6
5.3 Object Recognition
- ViT-L/16 reaches 98.25% top-1 and 99.89% top-5 accuracy on N-Caltech101, establishing state-of-the-art performance.ViT-S/16 achieves 94.81% top-1 accuracy, exceeding matched-scale DINOv2 by 2.87 and ECDP by 7.15 percentage points; ViT-B/16 reaches 97.62% top-1 and 99.68% top-5 accuracy.Source: Table 2 · p. 5, §5.3 · p. 6
5.4 Semantic Segmentation
- On DDD17-Seg, ViT-L/16 reaches 67.16% mIoU, 2.08 points above ScaleEvent; on DSEC-Semantic, it records 93.86% Acc and 69.04% mIoU, 0.61 points below ScaleEvent.Under the same ViT-L/16 backbone, Hyper-RED improves Acc over ScaleEvent by 0.46 and 0.76 percentage points on DDD17-Seg and DSEC-Semantic, respectively; Figure 4 shows qualitative predictions on DSEC.Source: §5.4 · p. 6, Table 4 · p. 6, Figure 4 · p. 7
5.6 Ablation Studies
- Combining the high-order intra- and cross-modal losses achieves the best ablation results across all metrics, improving MVSEC-Depth δ1 by 0.068 and reducing RMSE by 0.855 versus Exp. (c).The combination also raises mIoU by 1.34 points on DDD17-Seg and 0.31 points on DSEC-Semantic; the losses have complementary roles in event-token organization and anchoring event groups to image semantics.Source: Table 6 · p. 7, §5.6 · p. 7
5.5 Scalability Analysis
- Across LP, few-shot, and full supervision, Hyper-RED outperforms ScaleEvent on MVSEC-Depth and DDD17-Seg, and ranks first on DSEC-Depth at the listed settings.Performance also improves monotonically from ViT-S to ViT-L on recognition, segmentation, and depth; the distillation formulation requires no scale-specific modifications.Source: §5.5 · pp. 6–7, Table 3 · p. 6, Table 5 · p. 7
6 Conclusion
Hyper-RED transfers semantic structure from a frozen pretrained image teacher to event representations using teacher-guided hypergraphs and complementary high-order relational distillation. Across three downstream tasks and five event datasets, it achieves state-of-the-art overall performance and strong transferability across supervision settings.
6 Conclusion
- Hyper-RED transfers high-order semantic structures by organizing correlated tokens into teacher-guided hypergraphs and aligning their memberships and prototypes with intra- and cross-modal distillation.The framework aims to preserve relational consistency and group-level semantics while retaining event-specific characteristics.Source: §6 · p. 7
- Hyper-RED scales from ViT-S to ViT-L without scale-specific modifications to its distillation formulation.The experiments cover three downstream tasks across five event-based datasets.Source: §6 · p. 7
- Experiments demonstrate state-of-the-art overall performance and strong transferability under linear probing, few-shot fine-tuning, and full supervision.The paper presents semantic hypergraphs as structured knowledge carriers for scalable and annotation-efficient event representation learning.Source: §6 · p. 8