Source-linked AI summary
Rate-Utility Optimized Streaming of Volumetric Media for Augmented Reality
Jounsup Park, Philip A. Chou, Jenq-Neng Hwang
TL;DR
Frequent interaction and changing viewpoints make streaming volumetric media for AR/VR a distinct adaptation challenge. The paper combines 3D tiling, a window-based buffer, and rate-utility optimization; simulations report higher throughput and robustness, with no stalls in a variable-network test.
Problem
Volumetric streaming must adapt to frequent user interaction and changing viewpoints while managing bandwidth and keeping latency low.
Source: §I · pp. 1–2Method
The system uses 3D tiles, a window-based buffer, and a greedy optimal algorithm to allocate rates by tile utility.
Source: §I · p. 3, §VII · p. 15Results
In variable network conditions, the proposed algorithm achieved the highest average bitrate (8.6 Mbps compared to 6.91 Mbps and 5.26 Mbps) and had no stall events.
Source: A. Network-adaptivity evaluation · p. 14Takeaways & Limitations
Simulations report that volumetric streaming handles user interaction with sub-second latency.
Source: §VII · p. 16
Abstract
from arXiv · showhide
Volumetric media, popularly known as holograms, need to be delivered to users using both on-demand and live streaming, for new augmented reality (AR) and virtual reality (VR) experiences. As in video streaming, hologram streaming must support network adaptivity and fast startup, but must also moderate large bandwidths, multiple simultaneously streaming objects, and frequent user interaction, which requires low delay. In this paper, we introduce the first system to our knowledge designed specifically for streaming volumetric media. The system reduces bandwidth by introducing 3D tiles, and culling them or reducing their level of detail depending on their relation to the user's view frustum and distance to the user. Our system reduces latency by introducing a window-based buffer, which in contrast to a queue-based buffer allows insertions near the head of the buffer rather than only at the tail of the buffer, to respond quickly to user interaction. To allocate bits between different tiles across multiple objects, we introduce a simple greedy yet provably optimal algorithm for rate-utility optimization. We introduce utility measures based not only on the underlying quality of the representation, but on the level of detail relative to the user's viewpoint and device resolution. Simulation results show that the proposed algorithm provides superior quality compared to existing video-streaming approaches adapted to hologram streaming, in terms of utility and user experience over variable, throughput-constrained networks.
I. INTRODUCTION
Hologram streaming combines the bandwidth demands of immersive media with frequent changes in a user’s position and view, making low-latency adaptation important. The paper introduces 3D tiles, a window buffer, and utility-based rate allocation for this setting, against a background of video-streaming methods that do not account well for frequent interaction.
I. INTRODUCTION
- Frequent head turns and free movement among holograms change users’ views, positions, and distances to content, raising user-adaptation demands beyond ordinary video.Immersive media also includes spherical video, whose large pixel count can be managed by streaming higher resolution near the gaze and lower resolution elsewhere.Source: §I · pp. 1–2
- 3D tiles, distance- and viewpoint-aware utility, and a greedy optimal rate-utility algorithm adapt hologram streaming to visible content and multiple simultaneous objects.The system also uses a window buffer to request updated content near playback rather than only at the end of a queue.Source: §I · pp. 2–3
- The window-based system reports higher throughput and robustness than prior algorithms, and lower latency and higher quality than alternative designs.The paper attributes the latter comparison to using a window rather than a queue and a utility-maximization algorithm suited to the window.Source: §I · p. 3
II. RELATED WORKS
- Throughput-based and buffer-based streaming algorithms are described as ill-suited to frequent interaction because buffer duration directly affects interaction delay.Throughput-based methods select bitrate from estimated network throughput; buffer-based methods use buffer occupancy and can build long buffers for stability.Source: A. Network-adaptive video streaming · p. 4
B. Tile-based streaming of spherical video
Tile-based streaming allocates different representations across spatial regions to manage limited bandwidth and user-dependent viewing. Volumetric media add sparse, changing, viewpoint-dependent tiles and interactive viewpoint changes, motivating a window buffer that can request upgrades before playback.
B. Tile-based streaming of spherical video
- Spherical-video tile streaming partitions content into independently encoded tiles and allocates their bitrates under an overall constraint, favoring tiles more likely to be viewed.Existing approaches may retain a minimum bitrate for every tile to protect quality after an unexpected head turn.Source: B. Tile-based streaming of spherical… · p. 5
C. Volumetric media communication
- No published work known to the authors addressed on-demand or live volumetric-media streaming; a nearby free-viewpoint-video work described a streamable format but no streaming experiments or interaction-focused algorithms.The paper frames interactive volumetric streaming as the unresolved problem.Source: C. Volumetric media communication · p. 5
- Volumetric tiles are sparse and content-dependent, while visibility also depends on viewpoint because of occlusions.The set of occupied tiles may be large and changing, unlike spherical-video tiles.Source: C. Volumetric media communication · p. 5
III. WINDOW-BASED CLIENT BUFFER MANAGER
- A long queue can delay a fidelity change after a head turn, so the buffer treats content as a moving window and permits requests for any content within it.The client can request an upgraded representation for content already buffered at lower resolution, including content near playback.Source: §III · p. 6
- The window grows from 1 to 5 seconds during the first 4 seconds of playback, then remains at 5 seconds.The stated purpose is to start with a small window for quick playback and grow it to improve robustness to network dropouts.Source: §III · p. 6
- Reliable HTTP/TCP delivery of large data chunks makes the paper’s window-based rate-utility optimization simpler than prior packet-level decisions and reducible to a greedy algorithm.The authors say round-trip time is negligible relative to transmission time and model returned data as certain.Source: §III · pp. 6–7
B. Multiple objects
The framework supports simultaneous hologram objects with distinct media timelines that share a user-time start and window. Its DASH-consistent file structure organizes object metadata, segment representations, and tile-addressable data.
B. Multiple objects
- Simultaneously played objects may have different seek times and playback speeds, but share a user-time start and window function.Their windows can cover different media times while coinciding in user time.Source: B. Multiple objects · p. 7
C. File format
- The proposed file structure is consistent with DASH but includes extensions or modifications for tiled holograms.The authors note that other file structures are possible.Source: C. File format · p. 7
- Each object’s manifest lists its properties and representations, while segments partition its media into equal-duration intervals and provide multiple quality or resolution representations.A segment index accompanies each segment to locate representations, GOFs, and tiles.Source: C. File format · p. 7
- Segments contain GOFs made of occupied tiles, which are ordered and indexed by Morton codes.The file structure allows independently addressable tile content within temporal partitions.Source: C. File format · p. 8
Segment Index
The segment index lets the client locate representations, GOFs, and individual tiles for selective download. Its visibility and bitrate trade-offs depend on tile granularity and how much index data the client fetches.
Segment Index
- The segment index maps representations, GOFs, and tiles to locations and sizes, enabling the client to identify tile representations to request.Its fields include GOF offsets and header sizes, plus per-tile byte counts.Source: Segment Index · p. 8
- About 120 Kbps is the estimated segment-index bitrate for 100 tiles, four representations, 32-bit Morton codes and byte offsets, and 4-frame GOFs.The passage estimates this as about 1% of a typical total bitrate; the client may fetch only relevant index portions.Source: Segment Index · p. 8
- Visibility testing can use hundreds of tiles per GOF, while omitting visibility testing permits a single tile for the whole cube.The authors also identify many frames per GOF, few tiles, and few representations as ways to keep index bitrate low.Source: Segment Index · p. 8
- Nearby tiles can be grouped into one byte range within a multipart request because tiles are stored in Morton order.Source: Segment Index · p. 8
D. Client buffer manager interfaces
The client buffer manager connects the server and playback engine, receiving viewing and device information to select content. During operation it estimates throughput, allocates a request bit budget, and updates tile priorities as the view changes.
D. Client buffer manager interfaces
- The buffer manager communicates with the server through HTTP requests, preferably HTTP/2, and fetches initial low-bitrate content before notifying the playback engine to start.It downloads manifests and initial segments and indexes, then buffers a default period of media time.Source: D. Client buffer manager interfaces · pp. 8–9
- The playback engine supplies object URLs, seek and playback settings, initial viewpoints, and device resolution so the buffer manager can assess level of detail.It also notifies the manager when the user’s frustum changes or content is removed from the buffer.Source: D. Client buffer manager interfaces · pp. 8–9
E. Client buffer manager operation
- At request opportunities, the manager smooths its throughput estimate and sets the current request bit budget to estimated throughput multiplied by request duration.The experiments use a 0.5-second request duration and a smoothing weight of 0.75.Source: E. Client buffer manager operation · p. 9
- The manager chooses tile representations within the window to maximize expected utility under the request bit budget.Only changed representations count toward the requested bits; requests for tiles in the same segment and representation can be combined.Source: E. Client buffer manager operation · p. 9
- Updated viewpoints can raise a tile’s utility and priority, prompting a higher-quality request; newly visible tiles can be fetched within T seconds.The window allows requests beyond its leading edge, including content already buffered at a lower resolution.Source: E. Client buffer manager operation · pp. 9–10
IV. UTILTY MEASURES
Tile utility combines representation bandwidth with viewpoint-dependent visibility and distinguishable detail. The model limits detail by tile distance, representation resolution, and display resolution.
IV. UTILTY MEASURES
- The utility model accounts for changing viewpoint, including zero utility outside the view and potentially saturated utility at close range.The section also notes that future viewpoints and multiple viewpoints can complicate expected utility.Source: §IV · p. 10
- Expected tile utility weights bandwidth-dependent quality by distinguishable voxels and the probability that the tile will be visible.Visibility depends on the user's frustum and whether the tile faces the user.Source: §IV · p. 10
- Per-voxel utility increases with representation bandwidth, using a normalized affine function of log bandwidth.The normalization coefficients remain constant during playback.Source: A. Utility per distinguishable voxel · p. 10
B. Distinguishable voxels in a tile
- Distinguishable voxel count approximates the tile's covered area, using the lesser of representation voxels and display pixels per degree.Tile angular width depends on its real-world width and distance from the viewpoint.Source: B. Distinguishable voxels in a tile · pp. 10–11
C. Probability that a tile will be visible
The model weights tile utility by the probability that it will be visible when it exits the buffer window. Its simple prediction rule makes visibility less certain farther into the window.
C. Probability that a tile will be visible
- Visibility probability represents whether a tile will be visible when it reaches the window's trailing edge for display.The uncertainty arises from user behavior during the interim.Source: C. Probability that a tile will be… · p. 11
- User prediction remains an open research area for AR streaming, and the paper's simple uncertainty model could be improved with machine learning.The model represents increasing uncertainty about where the user will look farther into the future.Source: C. Probability that a tile will be… · p. 11
- The model predicts visibility from the current view, with error rising linearly from 0.1 for early-window tiles to 0.4 for late-window tiles.The rule treats a tile's current visibility as a prediction of its later visibility.Source: C. Probability that a tile will be… · p. 11
V. UTILITY MAXIMIZATION
The paper formulates tile selection as rate-utility maximization and solves it with a greedy, provably optimal algorithm. A Lagrange parameter orders representation choices by their efficiency under the rate constraint.
V. UTILITY MAXIMIZATION
- The utility-maximization algorithm is greedy yet provably optimal for the constrained rate-utility problem.It selects representations across tiles in the window while respecting the rate budget.Source: §V · p. 11
- For each tile, optimal choices lie on the upper convex hull of rate-utility points, whose vertices are traversed as λ decreases.The Lagrange parameter makes the maximization separable across tiles.Source: §V · pp. 11–12
- The algorithm lowers λ until the aggregate rate would exceed the budget, then requests each tile's selected representation.As λ decreases, each tile's selected representation and the total rate increase.Source: §V · p. 12
- When a tile already has a buffered representation, the modified algorithm treats reacquiring it as costing zero bits while retaining its utility.This raises the hull's initial point, making upgrades harder unless the bit budget or another representation's utility rises.Source: §V · pp. 12–13
VI. EXPERIMENTAL RESULTS
The evaluation tests network adaptivity with and without user interaction. WBA performs better than TBA and BBA on reported network metrics, while tile size and camera movement shape user-adaptive quality.
A. Network-adaptivity evaluation
- WBA has better average throughput than TBA and BBA under stable network conditions while considering estimated throughput and buffer occupancy together.Its buffer-occupancy pattern resembles BBA's, whereas TBA's occupancy grows continuously.Source: A. Network-adaptivity evaluation · pp. 13–14, Figure 10 · p. 14
- WBA achieves the highest average bitrate under variable network conditions: 8.6 Mbps, compared with 6.91 Mbps and 5.26 Mbps, with no stall events.TBA and BBA also avoid underflow; BBA requires much larger buffer occupancy than WBA to prevent rebuffering.Source: A. Network-adaptivity evaluation · p. 14, Figure 11 · p. 14
B. User-adaptivity evaluation
- Smaller tiles improve utility under both stable and variable network conditions by allowing more bits to be allocated only to visible tiles.The evaluation varies tile size through dyadic divisions of a cube.Source: B. User-adaptivity evaluation · p. 14
- With smaller tiles, visible regions retain a higher average representation level, and faster camera movement hardly degrades that level.Smaller tiles reduce redundant data streaming; the evaluation uses slow and fast camera movements.Source: B. User-adaptivity evaluation · pp. 14–15, Figure 13 · p. 15
- Across five competing objects, the algorithm allocates more bits to visible tiles while also requesting some lower-quality non-visible tiles for possible user changes.On replay, users receive more tiles at better quality.Source: B. User-adaptivity evaluation · p. 15
VII. CONCLUSION
The conclusion presents a volumetric-streaming system designed to balance network and user adaptivity. Its window-based buffer, 3D tiling, and rate-utility allocation support the reported network and latency results.
VII. CONCLUSION
- Window-based buffering allows requests for data anywhere inside the window, supporting quick adaptation to unexpected user interactions while remaining robust to throughput variation.This replaces queue buffering, which inserts content only at the queue's end.Source: §VII · p. 15
- A greedy algorithm, described as provably optimal, maximizes expected utility for each request under a rate constraint.The paper contrasts serial HTTP chunk requests with earlier packet-level requests whose download time was dominated by round-trip time.Source: §VII · p. 15
- 3D tiling concentrates network resources on areas of interest, while compact indexes address the larger, content-dependent set of occupied tiles and viewpoint-dependent occlusion.The paper distinguishes these challenges from 2D tiling for spherical video.Source: §VII · p. 15
- Tile utility accounts for representation bitrate and spatial resolution, view and facing direction, user distance, display resolution, and time until display.Source: §VII · p. 15
- The solution is consistent with modern HTTP streaming and requires only minor modifications, including manifest and tile-index information.The paper also lists multi-object allocation, sub-second response, fast start, trick modes, and quality improvement on replay among its supported features.Source: §VII · p. 16
- Even without tiles or rate-utility optimization, the algorithm has higher throughput and fewer rebuffering events than existing algorithms in stable and variable networks, and handles interaction with sub-second latency.These results are from simulations.Source: §VII · p. 16
APPENDIX: PSEUDOCODE
The pseudocode initializes playback and a request window, estimates network throughput, computes tile utilities, and greedily allocates bits before issuing requests and updating playback.
APPENDIX: PSEUDOCODE
- The client initializes playback and network-rate estimates, establishes a target window duration, and retrieves manifests and segment indexes covering that window.Source: APPENDIX: PSEUDOCODE · p. 17
- For tiles in the window, the procedure obtains the user's view and computes tile utility and bitrate across representations.The pseudocode uses view frusta, tile position and normal, representation bandwidth, and display characteristics in these calculations.Source: APPENDIX: PSEUDOCODE · pp. 17–18
- The allocator repeatedly selects the tile with the greatest marginal utility per bit while downloaded bits remain within the smoothed-rate budget.Source: APPENDIX: PSEUDOCODE · p. 18
- The client assembles byte-range requests for selected tiles, updates its throughput estimate, advances playback, and signals when playback can safely begin.The startup condition is playback time passing 1.5 seconds.Source: APPENDIX: PSEUDOCODE · p. 18