Source-linked AI summary
HoloLens 2 Research Mode as a Tool for Computer Vision Research
Dorin Ungureanu, Federica Bogo, Silvano Galliani, Pooja Sama, Xin Duan, Casey Meekhof, Jan Stühmer, Thomas J. Cashman, Bugra Tekin, Johannes L. Schönberger, Pawel Olszta, Marc Pollefeys
TL;DR
Computer vision and robotics researchers need accessible sensor data and tools for developing mixed reality applications on HoloLens 2. This report introduces Research Mode, its APIs, and an accompanying repository for processing and correlating raw sensor streams with tracking data. The examples show its use for 3D reconstruction and SLAM, while its intended research-only scope and uncertain future support define important boundaries.
Problem
Research on HoloLens 2 requires access to its sensor streams and ways to combine them with the platform’s tracking capabilities, but the first-generation Research Mode is incompatible with the new hardware.
Method
The report presents a second-generation Research Mode API and open-source tools for accessing, recording, processing, and spatially correlating HoloLens 2 sensor data with eye, hand, and head tracking.
Results
Research Mode supports applications including TSDF volume integration and SLAM using HoloLens sensor streams, with TSDF reconstruction demonstrated from almost 700 frames.
Takeaways & Limitations
The API and repository provide a research platform for computer vision and robotics experiments on HoloLens 2 and invite further community contributions.
Takeaways & Limitations
Research Mode is intended for academic and industrial research rather than end-user deployment, and Microsoft does not assure support in future hardware or OS updates.
Abstract
from arXiv · showhide
Mixed reality headsets, such as the Microsoft HoloLens 2, are powerful sensing devices with integrated compute capabilities, which makes it an ideal platform for computer vision research. In this technical report, we present HoloLens 2 Research Mode, an API and a set of tools enabling access to the raw sensor streams. We provide an overview of the API and explain how it can be used to build mixed reality applications based on processing sensor data. We also show how to combine the Research Mode sensor data with the built-in eye and hand tracking capabilities provided by HoloLens 2. By releasing the Research Mode API and a set of open-source tools, we aim to foster further research in the fields of computer vision as well as robotics and encourage contributions from the research community.
1. Introduction
HoloLens 2 extends the research potential of mixed reality with new sensing and tracking capabilities, but requires a new Research Mode because it is incompatible with the first-generation version. The report introduces that API and its research scope.
- Motivation: Mixed reality headsets are increasingly used in applications including assisted surgery, remote collaboration, task guidance, and digital-twin overlays.The report frames access to research tools as important because mixed reality application development often requires fundamental research and novel sensor combinations.
- Prior Research Mode: The first-generation Research Mode enabled on-device computer vision research by exposing raw image streams, including depth and infrared.Its public repository also included auxiliary tools and sample applications for computer vision and robotics research.
- HoloLens 2: HoloLens 2 adds a dedicated DNN core, articulated hand tracking, and eye-gaze tracking, but its new hardware makes it incompatible with the previous Research Mode.These changes motivate a second-generation API rather than a direct continuation of the earlier version.
- Contribution: The report introduces the second-generation Research Mode, released in 2020, with C++ APIs and tools for accessing HoloLens 2 sensor streams.It also discusses differences from the previous version and presents applications built on the API.
- Scope: Research Mode targets academic and industrial researchers in computer vision and robotics rather than end-user deployments.Microsoft does not assure continued support across future hardware or operating-system updates.
2. HoloLens 2
HoloLens 2 combines multiple cameras, depth sensing, and inertial sensing with on-device processing. Research Mode exposes these input streams through interfaces that provide frames, metadata, and sensor relationships for research applications.
- Device Architecture: HoloLens 2 includes a depth camera, an RGB camera, four grayscale cameras, an IMU, and a five-channel microphone array.Its custom HPU runs on-device computer-vision algorithms and hosts the DNN core, while the application CPU remains available.
- Input Streams: Research Mode exposes four 30 fps visible-light tracking cameras used for real-time visual-inertial SLAM.These are grayscale cameras accessed as input streams.
- Depth Sensing: The depth camera provides AHAT near-depth sensing at 45 fps and Long Throw far-depth sensing at 1–5 fps.AHAT supports hand tracking within 1 meter using aliased depth, while Long Throw supports on-device spatial mapping.
- Depth Sensing: The IR stream offers two depth modes computed from the same modulated infrared signal and is unaffected by ambient visible light.These modes are described as Active Brightness images for depth computation.
- API Access: Research Mode provides sensor interfaces for retrieving frames and metadata such as resolution and timestamps, and for mapping sensors relative to the device and world.The API also exposes accelerometer, gyroscope, and magnetometer data through the IMU.
3. Research Mode API
Research Mode exposes HoloLens 2 sensor streams through C++ interfaces for enumerating sensors, retrieving frames, and accessing sensor-specific data. It also supports coordinate transforms and world localization through the rigNode and HoloLens Perception APIs.
- 3.1. Main Sensor Loop: The main sensor loop creates a ResearchModeDevice, enumerates sensor descriptors, opens streams, and retrieves frames for processing.Per-sensor loops should run on separate threads because GetNextBuffer is blocking and must share a thread with OpenStream.
- 3.2. Sensors and Sensor Frames: The common sensor interface supports stream control, sensor identification, frame retrieval, and specialized camera or IMU interfaces obtained through QueryInterface.Camera sensors expose projection methods, whereas IMU sensors do not.
- 3.2. Sensors and Sensor Frames: Camera frames expose resolution, exposure, gain, timestamps, and sensor-specific buffers, including grayscale, depth, sigma, and active brightness data.Long Throw frames include a sigma buffer for invalidation, while AHAT frames embed invalidation information in the depth channel.
- 3.2. Sensors and Sensor Frames: IMU frames store batches of accelerometer, gyroscope, or magnetometer samples, with accelerometer and gyroscope frames also storing temperature.
- 3.3. Sensor Coordinate Frames: Each sensor provides an extrinsics transform to the rigNode, while camera mapping methods convert between camera-space 3D coordinates and image-space 2D points.The rigNode is the device-defined coordinate frame, and world localization uses its GUID with HoloLens Perception APIs.
4. The HoloLens2ForCV Repository
The HoloLens2ForCV repository provides sample apps for visualizing, recording, correlating, and processing HoloLens 2 sensor streams. These tools support prototyping computer vision workflows including marker triangulation, 3D reconstruction, and SLAM.
- Sample Apps: The repository provides UWP apps for live stream visualization, on-device processing, and recording Research Mode data for offline postprocessing.The initial release contains four apps and welcomes future community contributions.
- Sensor Visualization and Processing: SensorVisualization displays depth, VLC, and optionally IMU streams, while CameraWithCVAndCalibration detects arUco markers in two frontal VLC cameras and triangulates them.CalibrationVisualization visualizes VLC and depth coordinate frames.
- Stream Recorder: StreamRecorder captures depth, VLC, RGB, head, hand, and eye-tracking streams for later synchronization and analysis.The associated scripts place streams in a common coordinate frame, enabling 3D scene reconstruction and analysis of eye gaze, head pose, and hand pose.
- Computer Vision Examples: TSDF Volume Integration combines Long Throw depth, RGB frames, and head poses from StreamRecorder with an off-the-shelf library to reconstruct scenes.The example reconstruction uses almost 700 frames.
- Computer Vision Examples: The repository also demonstrates SLAM by feeding Long Throw depth and RGB frames to BAD SLAM and comparing estimated camera poses with HoloLens head poses.This comparison provides an evaluation reference for the SLAM camera trajectories.
5. Conclusion
Research Mode for HoloLens 2 provides an API and functionalities for accessing raw sensor streams on device, alongside public sample applications. These resources are intended to support computer vision and robotics research and encourage community contributions.
- Research Mode for HoloLens 2 enables on-device access to raw sensor streams through an API and related functionalities.
- The accompanying public repository provides sample apps and examples for building computer vision applications with HoloLens.
- The authors aim for these tools to facilitate further computer vision and robotics research and encourage contributions from the research community.