Source-linked AI summary
Beyond Mean Frametime: Time-Series Signatures for XR Timing Analysis
Marvin Thäns, Marc Erich Latoschik
TL;DR
XR timing summaries often discard temporal ordering, limiting analysis of how timing unfolds over time. This paper represents timing traces as compact, interpretable time-series signatures and finds stronger HMD-group separation than distribution-only summaries, with separation reduced by shuffling temporal order.
Problem
Distribution-only summaries discard temporal organization that can distinguish XR timing traces with different pacing patterns.
Method
The method represents each XR timing trace as a compact multivariate signature of distributional and order-dependent time-series features for comparison.
Results
Structure-aware signatures reveal stronger systematic multivariate differences between HMD-labelled groups than distribution-only summaries, while temporal-order shuffling reduces separation.
Takeaways & Limitations
Timing signatures provide a structure-aware reporting layer that complements conventional summaries by capturing temporal organization in XR timing traces.
Takeaways & Limitations
The evaluation uses engine-level application frametime rather than end-to-end motion-to-photon latency, and observed HMD-associated differences are not causal effects of the headset.
Abstract
from arXiv · showhide
XR systems expose timing quantities, such as motion-to-photon latency, frametime, or component-level runtime timings, that can be observed repeatedly as temporally ordered timing traces. Conventional reporting with means, standard deviations, percentiles, or histograms is useful, but it discards temporal ordering. We propose a general structure-aware methodology for analyzing and reporting XR timing traces. Each trace is represented by a compact, interpretable time-series signature, and collections of signatures can be visualized and compared statistically. We evaluate the method using engine-level application frametime traces from a large-scale in-the-wild VR dataset and compare timing signatures across HMD-labelled groups. Across multiple sampling and content-control conditions, structure-aware signatures reveal substantially stronger systematic multivariate differences between HMD-labelled groups than distribution-only summaries. A within-trace temporal-order shuffle control reduces this separation, particularly under content matching, providing direct evidence that original temporal ordering contributes information to the timing signatures. The strongest individual feature contributions vary across sampling and content-control conditions, indicating that no single timing characteristic dominates across analysis settings. Although demonstrated on application frametime, the representation operates on timing traces and therefore provides a basis for future application to other XR timing quantities, including instrumented motion-to-photon measurements.
1 Introduction
XR timing traces contain temporal organization that scalar and distributional summaries discard, motivating a compact, interpretable time-series signature. The paper proposes and validates a structure-aware methodology for representing, visualizing, and comparing timing traces across conditions and systems.
- Motivation: Temporal ordering distinguishes variability, dependence, periodicity, transients, and burstiness that identical distributional summaries cannot reveal.Figure 1 shows traces with identical values and histograms but markedly different temporal organization and pacing patterns.
- Motivation: Timing instability and added delay can impair XR performance, presence, and cybersickness, making temporal structure relevant beyond average timing.Controlled work further shows that jitter can impair interaction beyond mean delay.
- Contribution: The methodology provides scalable visualization and comparison of timing signatures across traces, conditions, systems, and studies.Raw traces retain temporal information but are difficult to inspect at scale, whereas scalar summaries are compact but discard ordering.
- Contribution: The paper addresses a downstream measurement gap by representing each timing trace as a compact multivariate signature of interpretable time-series features.The contribution is representational rather than instrumental: it assumes that a timing trace has already been obtained.
2 Related Work
Prior XR timing work measures latency, jitter, and runtime timings but commonly reports temporally ordered traces through distribution-only summaries. Time-series features, visualizations, and multivariate signatures provide a compact, interpretable layer for structure-aware comparison at scale.
- Positioning: This paper uses multivariate timing signatures as a structure-aware reporting methodology, without claiming that any feature family directly identifies a runtime mechanism.The representation bridges XR timing measurement and downstream comparison while retaining interpretable descriptive feature families.
- Measurement: XR timing research measures MTP latency, jitter, and stage-specific runtimes using dedicated, synchronized, and integrated instrumentation frameworks.These approaches span tracking, simulation, rendering, compositing, and display update while targeting temporal resolution and reproducibility.
- Reporting: Means, standard deviations, worst cases, percentiles, histograms, and outlier counts support comparison but can miss informative temporal patterns.Recursive modified-z-score outlier counts extend scalar reporting by highlighting extreme deviations, while remaining distribution-oriented.
- Time-Series Analysis: Interpretable time-series feature collections and compact sets such as hctsa and catch22 characterize temporal organization beyond distributional shape [10, 11, 17].These methods support systematic comparison across datasets and conditions while emphasizing efficiency, low redundancy, and broad discriminative utility.
- Time-Series Analysis: Structure-aware analysis captures dependence, recurrence, oscillations, and local predictability in addition to threshold-based extreme-value deviations [1, 4, 5, 22].This perspective treats departures from expected temporal structure as informative, extending pointwise outlier reporting.
- Visualization and Representation: Trace plots preserve ordering, distribution plots summarize variability, and aggregate matrix views enable scalable inspection of large time-series collections [21].Feature-based trace×feature matrices can be examined with heatmaps, clustered views, and feature-family groupings.
3 Dataset and evaluation conditions
The evaluation uses large-scale in-the-wild Beat Saber engine-level frametime traces, treating HMD labels as descriptive groupings rather than causal system identifiers. It compares Full, content-matched Restricted, and balanced conditions to address heterogeneous content and unequal group sizes.
- Dataset: The study analyzes millions of Beat Saber per-frame engine-timestamp recordings across heterogeneous gameplay configurations, computing a time-series signature for each frametime trace.Frametime reflects application-side frame pacing, not full tracking-to-display motion-to-photon latency.
- HMD-group comparison: The primary comparison is descriptive across HMD-labelled groups, not causal attribution to any single pipeline component.The HMD label identifies the reported headset category but does not control the complete system configuration.
- Dataset conditions: The Full condition contains N = 5,337,700 traces after restricting analysis to the 10 most-represented HMD groups and removing playthroughs covering less than 15% of a song.The raw dataset contains N = 7,949,374 traces spanning 32 HMD labels.
- Dataset conditions: The Restricted condition fixes song, difficulty, environment, gameplay mode, modifier, and speed, yielding N = 6,740 traces for more direct HMD-associated comparisons.These controls address content factors that can confound HMD-group comparisons.
- Dataset conditions: Balanced subsets equalize HMD-group contributions, producing N = 165,140 traces for Full (bal.) and N = 180 for Restricted (bal.) as robustness checks against imbalance and large-N effects.Each group is subsampled to the smallest group size within its condition.
4 Methods · 4.1 Timing-trace representation · 4.2 Preprocessing and resampling
The method represents each timing trace as an interpretable feature vector combining distributional and temporal-order properties. For frametime evaluation, traces are resampled to a common 1 ms grid while retaining observable pacing structure and recognizing potential confounds from raw dataset factors.
- 4.1 Timing-trace representation: Each trace is converted into a timing signature φ(x) that summarizes distributional characteristics, temporal dependence, periodicity, scaling, and frame-to-frame variability.Features are computed independently for each trace.
- 4.1 Timing-trace representation: All features are computed independently per trace, enabling collections of trace-level signatures to be analyzed without collapsing observations into a single aggregate trace.
- 4.1 Timing-trace representation: The evaluation applies the representation to per-frame application frametimes derived from engine-internal timestamps.The framework could also represent instrumented motion-to-photon latency or component-level runtime timings when suitable traces are available.
- 4.2 Preprocessing and resampling: Frametime series are resampled onto a common 1 ms grid using zero-order hold because recordings have differing sampling densities across HMD groups.This preprocessing is specific to the frametime evaluation and the catch22/catch24 implementation, which expects evenly sampled input.
- 4.2 Preprocessing and resampling: Raw-dataset factors can systematically affect frametime characteristics and therefore confound comparisons between HMD groups if they are not controlled or matched.
- 4.2 Preprocessing and resampling: The preprocessing retains pacing- and cadence-related structure because delivered frame pacing is part of the observable timing trace.
4.3 catch22-family feature extraction · 4.4 Feature standardization for visualization
The timing-signature representation uses catch22-family features spanning distributional, dependence, periodicity, and spectral characteristics, with expanded feature sets benchmarking standard summaries. Features are z-standardized across the analyzed dataset for visualization, while catch22 internally standardizes most inputs to emphasize relative structure.
- 4.3 catch22-family feature extraction: The representation instantiates 22 canonical catch22 characteristics derived from hctsa for compact, interpretable timing-trace analysis.Features are computed with pycatch22 version 0.4.5.
- 4.3 catch22-family feature extraction: The catch22 feature families capture distributional shape, linear and nonlinear temporal dependence, periodicity, and spectral structure.These families include DN, CO/IN, and PD/SP characteristics.
- 4.3 catch22-family feature extraction: Catch22 treats traces as evenly sampled values without timestamps and internally z-standardizes all features except mean and standard deviation.Consequently, most features emphasize relative distributional shape and temporal dependence rather than absolute scale or offset.
- 4.3 catch22-family feature extraction: Absolute frametime level and dispersion are primarily represented by DN Mean and DN Spread Std in the catch24 representation.These features are added because catch22 internally standardizes most inputs.
- 4.3 catch22-family feature extraction: The benchmark compares catch22, catch24, catch24+2, mean/SD-only, and outlier-only feature sets against common reporting practice.Catch24 adds mean and standard deviation, while catch24+2 further adds two outlier measures used in comparable latency analyses.
- 4.4 Feature standardization for visualization: For visualization, every feature is z-score standardized across the analyzed dataset to place HMD-labelled groups on a common scale.This standardization is applied across the analyzed dataset rather than separately within each HMD-labelled group.
4.5 MANOVA formulation
The study uses MANOVA to test whether multivariate timing-signature means differ across HMD-labelled groups, emphasizing Pillai’s trace and permutation testing for robustness. Pairwise and exploratory feature-level comparisons complement the global analysis.
- Global MANOVA: MANOVA tests whether multivariate timing-signature vectors differ across HMD-labelled groups, rather than predicting outcomes or ranking systems.The model is run separately for catch22, catch24, catch24+2, mean/SD-only, and outlier-only feature sets.
- Robustness testing: Pillai’s trace is emphasized alongside Wilks’ Λ, with 7,000 within-condition label permutations providing a non-parametric robustness check.This approach addresses potential violations of multivariate normality, covariance homogeneity, and equal group sizes.
- Pairwise comparisons: Pairwise HMD-group differences are tested with pooled-covariance Hotelling’s T 2, converted to F statistics and corrected using Benjamini-Hochberg.
- Feature-level comparisons: Exploratory one-vs-rest t-tests report Cohen’s d, η2, and median one-vs-rest η2 feature contributions descriptively after multiple-comparison correction.
4.6 Robustness analyses · 4.7 Temporal-order shuffle control
The study evaluates robustness across dataset conditions that vary content variability and HMD-group sample-size balance, then uses within-trace shuffling to isolate sensitivity to original temporal ordering. Shuffled traces preserve observed frametimes and length while undergoing the same preprocessing, feature extraction, and multivariate analyses as original traces.
- 4.6 Robustness analyses: Robustness analyses compare all MANOVA results across Full, Full (bal.), Restricted, and Restricted (bal.) dataset conditions.Balanced conditions randomly subsample each HMD group to match the smallest group, addressing content variability and sample-size imbalance.
- 4.7 Temporal-order shuffle control: The temporal-order control randomly permutes per-frame frametimes within each selected trace using a fixed reproducible seed.This manipulation tests whether extracted signatures depend on each trace’s original sequential organization.
- 4.7 Temporal-order shuffle control: Within-trace shuffling preserves trace length and the set of observed frametime values while destroying their original sequential organization.Thus, the control changes temporal arrangement without changing the raw value inventory or number of frames.
- 4.7 Temporal-order shuffle control: Shuffled traces use the same 1 ms ZOH preprocessing and catch24/catch24+2 feature-extraction pipeline as the original traces.Applying the identical pipeline makes differences attributable to temporal arrangement rather than processing changes.
- 4.7 Temporal-order shuffle control: The shuffle control is evaluated on Full (bal.) and Restricted using the same MANOVA and pairwise Hotelling’s T 2 procedures as corresponding original conditions.The analysis therefore directly compares multivariate results before and after within-trace temporal reordering.
- 4.7 Temporal-order shuffle control: Within-trace shuffling tests sensitivity to original sequential organization, whereas HMD-label permutation tests the null hypothesis of no group association.These controls address distinct questions: temporal structure in signatures versus the existence of HMD-group associations.
4.8 Feature heatmaps
Feature heatmaps visualize standardized features by HMD-labelled group, making the feature groups contributing to HMD-associated differences more interpretable.
- Feature heatmaps: Heatmaps display standardized features for each HMD-labelled group to highlight feature groups contributing most to HMD-associated differences.This visualization supports interpretation of the statistical comparisons.
- Feature heatmaps: Features are organized into descriptive families, including distribution shape, autocorrelation, spectral/periodic structure, symbolic dynamics, scaling/forecasting, and incremental differences.Examples include DS 1-2, LAC 1-2, NLA 1-3, SS 1-2, PER 1, SYM 1-4, SAS 1-2, SF 12, and ID 1.
- Feature heatmaps: Analysis code, example data, and aggregated results supporting the statistical comparisons, tables, and figures are publicly available.The materials are available at https://go.uniwue.de/timing-signatures.
5 Results
Structure-aware timing signatures produce stronger multivariate separation between HMD-labelled groups than distribution-only summaries across analysis conditions. Heatmap and shuffle-control results show that group differences reflect temporal frametime structure, while the strongest individual features vary by condition.
- Group-level signatures: Group heatmaps reveal distinct temporal frametime patterns: HTC Vive Pro 2 and HTC Vive Cosmos broadly co-vary, whereas Meta Oculus Rift S shows an opposing pattern.ID 1, measuring pNN40 successive differences, is elevated for Valve Index and HTC Vive Pro 2 but reduced for Meta Oculus Rift S.
- Robustness checks: Permutation tests corroborate MANOVA separation across conditions and feature sets, with all combinations significant at p < 0.001 except two specified Restricted balanced cases.catch22 in Restricted (bal.) has p = 0.0011, while the outlier-only baseline there is not significant at p = 0.173.
- Multivariate separation: Across analysis conditions, structure-aware signatures yield stronger HMD-group separation than distribution-only baselines, with Table 3 summarizing global and pairwise comparisons.Mean/SD-only provides almost no separation in Full, while catch24 and catch24+2 remain strongly informative in Restricted after content variability is removed.
- Temporal-order control: Within-trace shuffling reduces separation most under content matching, supporting original temporal ordering as an information source rather than the sole source of group differences.For catch24 in Restricted, Pillai’s trace falls from 0.671 to 0.375, significant pairs from 41/45 to 29/45, and median pairwise η2 from 0.328 to 0.124.
- Feature contributions: The strongest feature contributors vary by condition—NLA 2, ID 1, SF 1, and RD 1—indicating complementary characteristics rather than one dominant timing feature.Forecasting and spectral features are prominent under content matching, while raw mean is strongest only in Restricted (bal.).
6 Discussion
The discussion supports temporal structure in XR timing traces beyond distribution-only summaries, with HMD-associated frametime signatures remaining separable under balancing and content matching. It frames the method as a comparative, diagnostic reporting workflow while limiting conclusions to application frametime and noncausal HMD-associated differences.
- Empirical interpretation: HMD-associated frametime signatures remain separable under balanced sampling and content matching, whereas conventional distribution-only summaries are substantially weaker in several conditions.This supports the methodological premise that timing traces contain systematic temporal structure beyond means, standard deviations, and outlier summaries.
- Feature interpretation: Feature rankings and heatmaps show condition-dependent contributions from nonlinear dependence, forecasting, spectral, scaling, symbolic, raw-distribution, and incremental features.The variation cautions against treating any individual descriptor as universally dominant.
- Reporting workflow: Group-level timing-signature heatmaps provide a scalable middle ground between inspecting raw traces and using scalar summaries that discard temporal ordering.They show which cohorts share multivariate structure and which feature families drive differences, supporting comparative and diagnostic reporting rather than a universal quality score.
- Temporal-order control: In Restricted, shuffling reduces global separation, median pairwise effect sizes, and significant HMD-group comparisons from 41/45 to 29/45 for both catch24 and catch24+2.In Full (bal.), significance remains saturated at 45/45, so effect-size changes are more informative than pair-count changes.
- Scope and limitations: The study evaluates engine-level application frametime, so observed differences are HMD-associated timing differences rather than causal headset effects or direct end-to-end motion-to-photon latency.Dataset heterogeneity and repeated trials mean p-values primarily evidence systematic differences in this dataset, not population-level causal effects.
- Methodological limitations and future work: Resampling, shuffle design, and global multivariate tests constrain interpretation of cadence-sensitive features, ordering-removal invariance, and per-feature causal attribution.Future work should validate the pipeline on independent datasets, instrumented motion-to-photon traces, and mixed-reality or augmented-reality workloads.
7 Conclusion
The paper presents a structure-aware representation and reporting layer for XR timing traces, complementing distribution-only summaries with compact time-series signatures. Validation on real-world VR frametime traces shows systematic multivariate group differences, while the method’s current scope excludes causal attribution and absolute quality scoring.
- 7 Conclusion: The methodology separates the observed timing quantity from its representation, encoding repeated observations as time series and compact time-series features as timing signatures.The signatures capture selected temporal properties while complementing means, standard deviations, percentiles, and histograms.
- 7 Conclusion: Structure-aware signatures reveal systematic multivariate differences between HMD-labelled groups that are less visible in distribution-only summaries and remain observable under content matching and balanced sampling.The evaluation uses engine-level VR frametime traces, with catch22/catch24 features as the concrete signature implementation.
- 7 Conclusion: The paper does not establish causal hardware or runtime mechanisms or validate the representation on motion-to-photon, AR, or MR traces.Future work should apply the pipeline to instrumented motion-to-photon measurements and connect signature dimensions to perceptual and behavioral outcomes.
- 7 Conclusion: The contribution is a structure-aware representation and reporting layer between XR timing measurement and later system, experimental, or perceptual analysis, not a new latency metric or HMD classifier.The method supports comparison and reporting rather than an absolute quality score.
Data Availability Statement
The study uses an extended BOXRR dataset release that was not publicly available at submission, and raw timing traces were not redistributed.
- Data Availability Statement: The extended BOXRR release was being prepared as a separate data publication, with future access subject to participant privacy, consent, and data-shar.The article does not redistribute raw timing traces; access is planned through the forthcoming dataset release.