PaperlayerView exact revision on arXiv

Source-linked AI summary

Taming the latency in multi-user VR 360$^\circ$: A QoE-aware deep learning-aided multicast framework

Cristina Perfecto, Mohammed S. Elbamby, Javier Del Ser, Mehdi Bennis

arXiv:1811.07388v2cs.ITmath.IT

TL;DR

Wireless multi-user VR must deliver HD video with low latency, yet FoV processing and fluctuating rates make timely delivery challenging. The paper predicts and clusters users’ FoVs, then proactively schedules mmWave multicast with latency and queue-stability constraints. In the reported comparison, multicasting increases HD rate by 40–50% and reduces latency by 33–70%.

  • Problem

    Wireless 360° VR must deliver high-rate HD content within a tight motion-to-photon delay budget, while processing and fluctuating rates make timely FoV-tile delivery challenging.

    Source: §I · pp. 1–2

  • Method

    The framework predicts future FoVs, clusters users by FoV and location correlations, and schedules proactive mmWave multicast through latency- and stability-aware admission and matching.

    Source: A. Related Work · p. 3, C. User-SBS Chunk Scheduling · p. 9

  • Results

    40–50% increase in HD rate and 33–70% latency reduction are reported for multicasting compared with the other schemes.

    Source: 2) Impact of the Requested Video Quality · p. 13

  • Takeaways & Limitations

    The approach uses shared FoV content and predicted demand to improve delivery efficiency while maintaining strict latency bounds.

    Source: §I · p. 2, A. Related Work · p. 3

Abstract

from arXiv · show

Immersive virtual reality (VR) applications require ultra-high data rate and low-latency for smooth operation. Hence in this paper, aiming to improve VR experience in multi-user VR wireless video streaming, a deep-learning aided scheme for maximizing the quality of the delivered video chunks with low-latency is proposed. Therein the correlations in the predicted field of view (FoV) and locations of viewers watching 360$^\circ$ HD VR videos are capitalized on to realize a proactive FoV-centric millimeter wave (mmWave) physical-layer multicast transmission. The problem is cast as a frame quality maximization problem subject to tight latency constraints and network stability. The problem is then decoupled into an HD frame request admission and scheduling subproblems and a matching theory game is formulated to solve the scheduling subproblem by associating requests from clusters of users to mmWave small cell base stations (SBSs) for their unicast/multicast transmission. Furthermore, for realistic modeling and simulation purposes, a real VR head-tracking dataset and a deep recurrent neural network (DRNN) based on gated recurrent units (GRUs) are leveraged. Extensive simulation results show how the content-reuse for clusters of users with highly overlapping FoVs brought in by multicasting reduces the VR frame delay in 12\%. This reduction is further boosted by proactiveness that cuts by half the average delays of both reactive unicast and multicast baselines while preserving HD delivery rates above 98\%. Finally, enforcing tight latency bounds shortens the delay-tail as evinced by 13\% lower delays in the 99th percentile.

I. INTRODUCTION

Wireless 360° VR must deliver high-definition frames within a tight motion-to-photon delay budget, while growing multi-user demand makes that difficult. The paper combines FoV prediction, clustering, proactive mmWave multicast, and latency-aware scheduling, and evaluates the approach in a VR theater model.

I. INTRODUCTION

  • HD content must be consistently streamed with end-to-end latency, or motion-to-photon delay, below 15–20 milliseconds; otherwise, delayed or low-quality VR content might cause VR sickness (cybersickness) and ruin the experience.Wired connections limit user movement, motivating wireless solutions that provide convenience and high quality.Source: §I · p. 1
  • Proactive FoV-tile delivery can help meet fluctuating rate demands: in the example, predicting 24 of 32 tiles lets an SBS send only 8 remaining tiles in the next period.The example contrasts a 70 Mb first-period budget with 10 Mb in the second period, which causes an outage without proactivity.Source: §I · p. 2

A. Related Work

  • Prior work leaves a gap in cellular VR: the authors say no cited approach combines prediction-related information with proactive mmWave multicast under high-reliability, low-latency constraints.They propose leveraging FoV and location correlations to schedule multicast for users with overlapping FoVs, while optimizing HD-frame admission and scheduling subject to latency and queue-stability constraints.Source: A. Related Work · p. 3
  • The proposed evaluation uses a real 360° pose dataset and a GRU-based DRNN to predict FoVs, then clusters users by FoV overlap and proximity.Simulations examine prediction horizon, frame size, clustering, and network size; the authors report more HD-quality frames than considered baselines while maintaining tight transmission-delay bounds.Source: A. Related Work · p. 3

A. Deployment Scenario

  • The modeled VR theater caches HD frames at edge SBSs and pre-caches SD versions on users’ HMDs for use when HD delivery misses its deadline.Video frames are projected into a regular EQR tile grid, and each user’s FoV is represented as a subset of those tiles.Source: A. Deployment Scenario · pp. 3–4

B. FoV and Spatial Inter-user Correlation

Users report pose and video information so the edge can predict future FoVs and cluster viewers using FoV and spatial correlations. The system then schedules real-time and predicted chunks over a modeled mmWave channel.

B. FoV and Spatial Inter-user Correlation

  • Users report their video index and 6DoF pose periodically; FoV prediction uses past 3DoF head orientations, since physical location does not affect viewing direction in the centered 360° display model.The controller receives current FoVs and predicted clusters, while the manuscript assumes sufficient uplink resources for timely pose and channel-state reports.Source: B. FoV and Spatial Inter-user… · pp. 4–5
  • Predicted FoVs and reported poses feed clustering of users watching the same video according to FoV and spatial correlation; scheduling covers both current and predicted chunks before their deadlines.The channel model neglects multipath and adopts the 3GPP indoor open-office scenario, with human blockage also accounted for.Source: B. FoV and Spatial Inter-user… · pp. 4–5, C. mmWave Channel and Communication… · p. 5

C. mmWave Channel and Communication Model

  • SBSs use multiple beams while each HMD has one RF chain, and a user’s achievable multicast rate depends on the composition of its cluster.The model also accounts for beam misalignment, interference, and Gaussian noise in the SINR description.Source: C. mmWave Channel and Communication… · pp. 5–6

III. PROBLEM FORMULATION

The formulation maximizes delivered frame quality while enforcing transmission-latency constraints and stable queues. It separates proactive predicted-FoV requests from real-time requests and uses queue-aware control to select admissions and schedules.

III. PROBLEM FORMULATION

  • The network-wide objective maximizes FoV content quality while maintaining latency constraints and transmission-queue stability.The formulation includes proactive and real-time requests and multicast and unicast transmission capabilities.Source: §III · p. 6
  • Proactive cluster-level FoV requests can reuse shared chunks, while real-time requests cover tiles missed by imperfect DRNN predictions.Missed-tile requests need expediting to meet MTP latency constraints.Source: §III · p. 7
  • The available scheduling time is the MTP deadline minus the elapsed time since the chunk was requested.Transmission delay exceeding this time is constrained to occur with low probability, bounded by the predefined reliability metric.Source: §III · p. 7
  • HD frame admission and chunk scheduling are jointly chosen subject to latency constraints, with the admission and scheduling variables collected separately.The decision variables are frame-admission indicators and binary chunk-scheduling indicators.Source: §III · p. 7
  • Lyapunov virtual queues represent the time-average utility and delay constraints, and a drift-plus-penalty controller trades queue backlog against utility.The control action minimizes a bound using current traffic and virtual queues; the subproblems are separable and do not require traffic statistics.Source: §III · p. 8

A. Auxiliary Variable Selection

The auxiliary-variable subproblem is solved independently per user. The chosen utility function determines the optimization objective, with a logarithmic option supporting proportional fairness.

A. Auxiliary Variable Selection

B. HD Streaming Admission

The HD streaming admission subproblem maximizes admission rate through a convex optimization. Its optimal control admits or discards each whole frame according to physical and virtual queue state.

B. HD Streaming Admission

C. User-SBS Chunk Scheduling

The chunk-scheduling subproblem assigns user or cluster requests to mmWave small-cell base stations through a matching algorithm. The matching formulation uses player preferences and quotas to characterize assignments.

C. User-SBS Chunk Scheduling

  • The combinatorial chunk-scheduling problem uses matching to associate cluster or individual user requests with mmWave SBSs.The scheduling subproblem maximizes term #3.Source: C. User-SBS Chunk Scheduling · p. 9

IV. A MATCHING THEORY APPROACH TO HD CHUNK

A. Matching Theory Preliminaries

  • A matching game pairs clusters and SBSs using their preference profiles, subject to each player's quota.The paper defines the two player sets as clusters and base stations.Source: A. Matching Theory Preliminaries · p. 9

B. Matching Utility Formulation

The scheduling subproblem is modeled as a matching game between SBSs and user clusters, with utilities reflecting request priority, chunk relevance, and transmission opportunity. To address interference-driven externalities, the method estimates interference and uses deferred acceptance under a stated stability condition.

B. Matching Utility Formulation

  • The scheduling game matches SBSs to user clusters so they can allocate mmWave resources while meeting VR QoE constraints.Matching pairs are updated each timeslot using channel and queue state.Source: B. Matching Utility Formulation · p. 9
  • SBS utility reflects both the priority of a requested chunk’s frame and the chunk’s relevance to the cluster FoV.Cluster-level multicast transmits each chunk in the cluster FoV once, and serving more relevant chunks first contributes to system welfare.Source: B. Matching Utility Formulation · p. 9
  • The cluster-side utility depends on the quality of the transmission opportunity through the offered rate.The passage states that the indicator for a frame meeting the MTP delay bound can be nonzero only for the currently playing frame.Source: B. Matching Utility Formulation · p. 9

C. Stability of the Matching

  • The matching is called stable when no unmatched SBS–cluster pair would both prefer each other to their current matches.The stability definition describes such a pair as blocking the matching.Source: C. Stability of the Matching · pp. 9–10
  • Canonical deferred acceptance is guaranteed to produce a two-sided stable matching, but this game has interference-driven externalities that prevent direct application.The stated assumption is that each SBS matches to one cluster using orthogonal, non-overlapping multicast beams, leaving inter-SBS interference as the impairment to consider.Source: C. Stability of the Matching · p. 10
  • The proposed distributed, computationally efficient approach replaces instantaneous service rates in utilities with estimates to avoid requiring system-wide matching-state knowledge.Interference estimates blend the previous time instant’s measured interference with a moving average over a window of samples.Source: C. Stability of the Matching · p. 10
  • The algorithm learns interference and updates queues, then creates cluster-level request pools and sorts requests by urgency before matching.It proceeds through interference learning and candidate chunk selection, matching-game construction, and deferred acceptance.Source: C. Stability of the Matching · p. 10
  • In deferred acceptance, SBSs propose to their best-ranked eligible clusters, and clusters accept or reject proposals according to their preferences.The algorithm removes matched players from the unmatched sets and continues while eligible proposals remain.Source: C. Stability of the Matching · p. 10

C ; end end end end Phase IV - Stable matching

With estimated utilities, the scheduling game can be solved by deferred acceptance, which converges to a stable matching once interference learning converges.

C ; end end end end Phase IV - Stable matching

V. DRNN FOV PREDICTION AND FOV+LOCATION AWARE USER CLUSTERING

The paper predicts users’ future tiled FoVs from pose sequences with a GRU-based deep recurrent model. Its architecture processes pose inputs through stacked GRU layers and outputs thresholded tile predictions for a frame TH frames ahead.

A. Sequential Deep Learning Model Operation

  • Tiled-FoV prediction maps a sequence of past and current pose vectors to multi-label tile predictions for a future frame.Each output label corresponds to a video tile, with its value estimating the likelihood that the tile belongs to the user’s future FoV.Source: A. Sequential Deep Learning Model… · pp. 10–11
  • The model uses GRUs whose update and reset gates regulate how previous hidden states contribute to the next state.The paper motivates GRUs by their handling of long-term dependencies and states that they are faster to train than LSTMs and perform better on small datasets.Source: A. Sequential Deep Learning Model… · p. 11

B. DRNN architecture

  • The DRNN takes a sequence of 3DoF pose vectors and processes it through two GRU layers, each with 512 hidden-state units.The second layer’s output passes through a serial-to-parallel layer and a dense layer connecting to the tile outputs.Source: B. DRNN architecture · p. 11
  • The model predicts a user’s FoV tiles TH frames ahead, as shown in the edge-controller block diagram.A sigmoid maps the dense output to tile probabilities, which are binarized using a threshold chosen to balance accuracy and recall.Source: Figure 5 · p. 11, B. DRNN architecture · p. 11

C. DRNN Training

The DRNN is trained with supervised binary cross-entropy and is paired with a clustering scheme that uses predicted FoVs and user locations. Hierarchical agglomerative clustering with average linkage supports exploration of cluster-count trade-offs.

C. DRNN Training

  • Supervised DRNN training minimizes binary cross-entropy over training instances using model logits and labels.Source: C. DRNN Training · p. 11
  • Training uses backpropagation through time and Adam for 20 epochs, with batches of 512 and learning rate 0.01.Adam parameters are β1 = 0.9 and β2 = 0.999, with no decay.Source: C. DRNN Training · p. 12

D. Proposed FoV and Location Aware User Clustering

  • Users watching the same video are clustered for the predicted frame using their spatial and content correlations.The FoV-related distance between users is scaled by relative physical distance, whose minimum is set by theater dimensions and seat arrangements.Source: D. Proposed FoV and Location Aware… · p. 12
  • Average-linkage hierarchical agglomerative clustering operates on the distance metric and allows cluster count to vary.The paper uses this flexibility to investigate trade-offs between communication resource utilization and performance as the number of clusters changes.Source: D. Proposed FoV and Location Aware… · p. 12

VI. SIMULATION AND PERFORMANCE EVALUATION

The evaluation uses real 360° video head-tracking data to train and assess GRU-based FoV prediction, then tests proactive multicast scheduling across theater sizes and network loads. It measures delay, HD delivery, and FoV overlap to examine latency–quality trade-offs.

A. 360◦Video Head-tracking Dataset and DRNN FoV Prediction Accuracy Results

  • The test set contains 15 users from a 50-user head-tracking dataset, with 35 users reserved for training and FoVs mapped to 200 tiles per video frame.The videos are 60 seconds long, 4K, and encoded at 30 fps; the considered FoV is 100° × 100°.Source: A. 360◦Video Head-tracking Dataset and… · p. 12
  • Prediction accuracy falls as the horizon extends, while adding GRU layers is counterproductive because it overfits and increases model complexity.Prediction accuracy is assessed with the Jaccard similarity index; the supplied passages do not give its numerical values.Source: A. 360◦Video Head-tracking Dataset and… · p. 12

B. Deployment Details and Reference Baselines

  • The simulations cover small theaters with 10 users per video and big theaters with 15, across scenarios ranging from 1 to 5 and 1 to 10 videos, respectively.The deployment places small-cell base stations at the ceiling’s four upper corners.Source: B. Deployment Details and Reference… · pp. 12–13
  • The four schemes compare reactive unicast, reactive multicast, proactive multicast, and proactive multicast with the latency constraint.The study tracks average and 99th-percentile delay, HD successful-delivery rate, and Jaccard similarity; the latter helps characterize multicast overhead and FoV correlation within clusters.Source: B. Deployment Details and Reference… · p. 13, C. Discussion · p. 13

C. Discussion

  • The 99th-percentile delay tests whether latency stays below the MTP threshold with the desired probability, while HD delivery rate captures its trade-off with quality.Jaccard similarity reflects the relationship between cluster FoV overlap, extra chunks sent, and missed tiles.Source: C. Discussion · p. 13

1) Impact of the FoV Prediction Horizon:

The evaluation finds that multicast and proactive scheduling improve HD delivery and delay across tested video-quality, clustering, and network-size settings. Prediction horizon and Lyapunov trade-offs affect performance, while the latency-constrained scheme also limits worst-case delay.

1) Impact of the FoV Prediction Horizon:

  • Longer prediction horizons first reduce frame delay by allowing earlier scheduling, then increase it as prediction accuracy falls and more missed chunks require real-time transmission.The increased real-time transmission also lowers utilization of the predicted FoV and reduces Jaccard similarity; high HD quality is maintained even at long horizons.Source: 1) Impact of the FoV Prediction Horizon · p. 13, Figure 6 · p. 14

2) Impact of the Requested Video Quality:

  • 40–50% higher HD rate and 33–70% lower latency: multicasting outperforms unicast as chunk size varies, while proactive schemes reduce average delay without sacrificing HD quality.The MPROAC+ latency constraint also bounds worst delays compared with MPROAC.Source: 2) Impact of the Requested Video Quality · p. 13, Figure 7 · p. 14
  • 17–37% lower 99th-percentile latency: MPROAC+ improves on MPROAC and MREAC, respectively, as the Lyapunov parameter trades HD delivery rate against queue stability.Higher V∆ prioritizes HD delivery; lower values prioritize traffic and virtual-queue stability, at the expense of HD delivery rate.Source: Figure 7 · p. 14, 2) Impact of the Requested Video Quality · p. 14

4) Impact of the Network Size:

  • Close to 100% HD streaming rate with lower frame delay is achieved in scenarios 1, 2, 4, and 5; in congested scenarios, MPROAC+ achieves higher HD delivery and lower delay than the baselines.Multicast improves performance over unicast in congested scenarios, where requested data limits advance scheduling of predicted frames.Source: Figure 9 · p. 15, 4) Impact of the Network Size · p. 15

3) Impact of the Number of Clusters:

  • Fewer clusters improve content reuse, reducing delay and increasing HD delivery, while more clusters increase Jaccard similarity by sending fewer unnecessary chunks.The reported comparison is between multicast schemes and the unicast UREAC scheme.Source: Figure 8 · p. 14, 3) Impact of the Number of Clusters · p. 15

VII. CONCLUSIONS

  • The proposed framework combines FoV prediction, user clustering, proactive mmWave multicast, Lyapunov optimization, and matching theory to maximize HD frame admission under latency constraints.The conclusion reports considerable gains over reactive baselines and notable outperformance of unicast.Source: §VII · p. 15

APPENDIX A PROOF OF LEMMA 1

The proof bounds the one-slot Lyapunov drift using queue updates and quadratic-term bounds, then rearranges the result to obtain equation (20).

APPENDIX A PROOF OF LEMMA 1

  • The proof derives upper bounds on terms in the one-slot Lyapunov drift after squaring the physical and virtual queues and applying an inequality for nonnegative x.The queues referenced are those in equations (11), (17), and (18).Source: Appendix A · p. 15
  • Substituting the queue update for q_u(t + 1) into the drift bound uses the non-empty-queue guarantee associated with I{d_t^2MTP} = 1.Source: Appendix A · p. 15
  • Quadratic terms are upper bounded for queue stability, collected into a constant, and the penalty term VΔE is subtracted before rearrangement yields equation (20).Source: Appendix A · pp. 15–16
Loading 1811.07388v2…