PaperlayerView exact revision on arXiv

Source-linked AI summary

Eye Tracking Signal Quality Evaluation: Meta Quest Pro vs HTC VIVE Pro Eye vs HTC VIVE Focus Vision

Aleksandr Chernysh, Mehedi Hasan Raju, Kamrul Hasan, Md Arif Shahriar, Proloy Karmakar, Isayas B. Adhanom, Oleg Komogortsev

arXiv:2610.32915v2cs.HC

TL;DR

Because gaze signal quality limits VR applications, this study compares three headset trackers and examines what their device clocks reveal beyond mean error. The Focus Vision remained least accurate after recalibration, while the Pro Eye's error rose in the later block.

  • Problem

    Because gaze signal quality limits VR applications, prior headset evaluations differed in protocol, processing, and participants, and the Focus Vision tracker had no published evaluation identified by the authors.

    Source: §1 · pp. 1–2

  • Method

    Thirty participants completed the same protocol and analysis on the Meta Quest Pro, HTC VIVE Pro Eye, and HTC VIVE Focus Vision, with gaze streams examined on each device clock.

    Source: §1 · p. 2, §3.2 · p. 3

  • Results

    The Focus Vision was least accurate in the first block and remained less accurate than uncorrected Quest Pro after recalibration; Pro Eye error rose by HL 0.70° in the second block.

    Source: §7 · p. 12

  • Takeaways & Limitations

    Evaluations should add device-clock checks for repeated timestamps, data-loss composition, and validity flags during instructed eye closures.

    Source: §7 · p. 12

    The study tested one unit per headset once per participant, so it does not establish variation across units.

    Source: §6 · p. 12

Abstract

from arXiv · show

Gaze research in virtual reality needs eye trackers of known error. Thirty participants completed the same protocol on the Meta Quest Pro (QP), HTC VIVE Pro Eye (PE) and HTC VIVE Focus Vision (FV); to our knowledge, this is the first evaluation of the FV eye tracker. In a random-saccade task, median participant mean error was 1.04{\deg} on QP, 1.50{\deg} on PE and 2.41{\deg} on FV, which remained least accurate after linear recalibration. In a later block with a darker background, PE error rose by 0.70{\deg}. FV samples were positively correlated: dispersion was 9.8 times the sample-to-sample root mean square. The device clocks showed losses that mean error hides. QP paused for up to 6.04 s, and FV skipped single samples. During instructed eye closures, QP flagged every sample valid while gaze moved 9.74{\deg}. Evaluations should report repeated timestamps, loss composition on the device clock and validity during eye closures.

1 Introduction

Gaze supports several VR applications, but practical signal quality must be measured because manufacturer accuracy specifications may not be reached. The study compares three headsets under a shared protocol and asks what their device clocks reveal beyond mean error.

1 Introduction

  • Prior headset evaluations used differing protocols, processing, and participants, and no published evaluation of the HTC VIVE Focus Vision tracker was found.The study therefore pairs the three headsets under the same protocol and analysis.Source: §1 · pp. 1–2
  • Mean error cannot test whether each row is a new eye measurement or whether validity flags match eye state, so the authors propose three device-clock checks.The reported diagnostics include stale QP rows, interruptions, skipped FV samples, and valid flags during instructed eye closures.Source: §1 · p. 2
  • The study asks how headset accuracy and precision compare across the field of view, and how accuracy changes in a later darker-background block and with raised eyebrows.It also asks what device clocks reveal about data availability, repeated, missing, and invalid samples, including samples during eye closure.Source: §1 · p. 2
  • Thirty participants completed the same protocol and analysis on three headsets, including the first evaluation of the Focus Vision tracker the authors found.The comparison also covers accuracy across the field of view and precision.Source: §1 · p. 2

2 Related Work

Prior VR eye-tracking evaluations used differing protocols and participants, and the HTC VIVE Focus Vision’s integrated tracker had not been evaluated. This study uses a paired three-headset protocol and defines measures for gaze error, temporal structure, data loss, and recalibration.

2 Related Work

  • Earlier evaluations differed in protocol, processing, or participants; prior comparisons included the Quest Pro and Pro Eye, while the Focus Vision’s integrated tracker had no published evaluation.Prior work also found Pro Eye accuracy and precision worsened with eccentricity and, in a 30-participant comparison, the Quest Pro more accurate and Focus 3 add-on more precise.Source: §2 · p. 2

3.1 Participants

  • Thirty university students participated; the sample was 26 men and 4 women, aged 18–47 years.Participants completed the protocol on all three headsets.Source: §3.1 · p. 3

3.2 Apparatus

  • The three custom Unity applications used the same scene, tasks, stimuli, and per-sample log, but each read gaze through a different vendor interface.The analyzed stream therefore combined device, runtime, and logger outputs; the cyclopean ray was valid only when both eyes were valid.Source: §3.2 · p. 3

3.3 Tasks and Procedure

  • Participants completed five tasks per headset in fixed order, including random-saccade blocks with gray and darker backgrounds and instructed eye closures.The six headset orders were counterbalanced; the darker-background block always followed the gray block and a reading task.Source: §3.3 · pp. 3–4

3.4 Processing and Measures

  • The analysis defined window error from usable gaze samples and summarized participant mean error across non-missing random-saccade windows by the participant median.It also measured temporal structure with dispersion-to-sample-to-sample RMS ratios, and separated expected-sample losses into invalid, frozen, and dropped categories on each device clock.Source: §3.4 · pp. 4–5
  • The 30 participants completed all headset conditions, and the paired comparisons used rank tests with Holm correction within contrast families.The validation gate required mean error at most 3° and at least 80% valid samples per target; failed gates led to repeat validation or recalibration.Source: §3.1 · p. 3, §3.3 · p. 4, §3.4 · p. 5

4 Results

Across the two random-saccade blocks, accuracy, temporal structure, and data availability differed among QP, PE, and FV. Error summaries alone missed important distinctions, including QP’s valid flags during eye closure and the different composition of data loss.

4.1 Spatial Accuracy

  • 1.04° on QP, 2.41° on FV, and 1.50° on PE were the RAN127 median participant mean errors; QP–PE was inconclusive, while both beat FV.In RAN63, the medians were 1.07°, 2.26°, and 2.28°, respectively, and QP was more accurate than both other headsets.Source: Table 2 · p. 6, §4.1 · p. 6

4.2 Accuracy Across the Field of View

  • Error rose from center to outer ring on all headsets: QP 0.71° to 1.18°, FV 1.52° to 2.86°, and PE 0.55° to 2.05°.RAN127 U95|E95 was 6.71–8.37° across headsets; the authors note that target sizes still require confirmation in an interaction test.Source: §4.2 · pp. 6–7, Figure 3 · p. 7

4.3 Systematic Error and Recalibration

  • Linear recalibration reduced participant mean error on all headsets, most on FV, from 2.41° to 1.48°; transferred to RAN63, QP’s change was not significant.The within-RAN127 reductions were 0.82° on FV, 0.29° on PE, and 0.15° on QP; FV remained less accurate before recalibration.Source: §4.3 · p. 7

4.4 Change in the Second Block

  • From RAN127 to darker-background RAN63, PE error rose 0.70°; QP and FV changes were not significant, and the cause remains open.Pupil diameter grew on both HTC headsets, but its change did not correlate with error change across participants.Source: §4.4 · pp. 7–8

4.5 Precision and Temporal Structure

  • FV had the lowest RAN127 sample-to-sample RMS (0.04°), but its median STD/RMS ratio was 9.8 across blocks, versus 1.3 on QP and 2.4 on PE.FV increments were positively correlated; results were consistent with temporal filtering or nonlinear interpolation of a lower-rate stream, but no filter was tested directly.Source: §4.5 · p. 8, Figure 4 · p. 8

4.6 Data Loss and the Device Clock

  • QP had 233 intervals over 0.1 s between unique device timestamps, including a 6.04-s interruption; loss was frozen on QP, dropped on FV, and invalid on PE.FV’s 5.9% RAN127 sample loss largely reflected gaps of one skipped sample, while FV had the fewest empty 250-ms windows because QP and PE could lose consecutive samples.Source: §4.6 · pp. 8–9, Table 3 · p. 9, Figure 5 · p. 9

4.7 Eyebrow Raising and Eye Closure

  • During eye closure, QP flagged 100.0% of rows valid while gaze moved a median 9.74°; FV and PE mostly showed sustained validity loss.Raised eyebrows increased median error on FV and PE, but the data did not rank the headsets by eyebrow sensitivity.Source: §4.7 · pp. 9–10, Table 4 · p. 10, Figure 6 · p. 10

5 Discussion

FV was least accurate in the first block, and device-clock diagnostics exposed data-quality problems hidden by mean error. Recalibration improved FV accuracy but did not bring it to QP’s level; the discussion also compares prior measurements and summarizes headset trade-offs.

5.1 Main findings

  • 2.41° FV median participant mean error exceeded QP at 1.04° and PE at 1.50°; QP and PE did not differ reliably.These are first-block results; the discussion also reports QP interruptions and eye closures that its validity flag did not mark.Source: §5.1 · p. 10

5.2 What the device clocks add

  • QP’s validity flag did not mark instructed 2-s closures or blink screens, and gaze during closures had 13 times higher STD than with open eyes.The authors recommend checking repeated timestamps, device-clock loss composition, and validity during instructed closures.Source: §5.2 · p. 10

5.3 FV accuracy and signal

  • 38.5% linear recalibration reduced FV median error from 2.41° to 1.48° in the spatial holdout, but corrected FV remained less accurate than uncorrected QP at 1.04°.FV precision should be reported with both RMS and STD; the reported 48-ms longer optimal shift does not isolate signal delay.Source: §5.3 · pp. 10–11

5.4 Comparison with prior work

  • 17.06° versus 6.71° U95|E95 values distinguish an earlier QP report from this study; possible explanations for the difference remain untested.The studies used the same grid and analysis window, but differed in sample, session content, validation gate, and potentially gaze-estimation pipeline.Source: §5.4 · p. 11

5.5 Choosing a Headset

  • 6.71–8.37° first-block and 10.26–10.54° second-block U95|E95 error budgets grew toward the periphery on all three headsets.Only the PE block change differed from the other headsets; QP also dropped out for up to several seconds and did not flag instructed closures.Source: §5.5 · p. 11, Table 5 · p. 11

6 Limitations

The study’s design and measurement choices constrain interpretation of its results, including block comparisons, skipped FV samples, and generalization beyond the participant sample.

6 Limitations

  • Fixed block order confounds the RAN63–RAN127 difference with time in session and the preceding background.The PE increase may reflect pupil-size dependence or block order, which the study could not separate.Source: §6 · p. 12
  • Skipped FV samples cannot be assigned to the device or interface from the logged output; resolving whether they existed would require polling more than twice per frame.The study evaluated each interface’s output as logged by its application.Source: §6 · p. 12
  • The 30 participants were university students aged 18–47, including 26 men, limiting generalization to other populations..Source: §6 · p. 12

7 Conclusion

The conclusion identifies FV as least accurate in the first block and recommends reporting device-clock diagnostics alongside accuracy summaries.

7 Conclusion

  • 38.5% recalibration reduced FV median error from 2.41° to 1.48° in a spatial holdout, but corrected FV remained less accurate than uncorrected QP.The conclusion also reports a 0.70° PE error rise in the second block, notes that its cause is unidentified, and recommends reporting FV RMS with STD.Source: §7 · p. 12

Privacy and Ethics Statement

The paper reports ethics approval and informed consent, provides access to recordings and code, and describes the supplement’s scope and terminology.

Privacy and Ethics Statement

  • The study had Texas State University IRB approval (protocol 10770), and all participants gave written informed consent.Recordings are stored under pseudonymous codes on access-restricted storage; the paper notes that eye movements can identify people and reveal attributes such as attention, fatigue, or health.Source: Privacy and Ethics Statement · p. 12

Data and Code

  • Raw and processed recordings, documentation, acquisition applications, analysis code, and an independent reimplementation are available in the Texas State University digital repository.The paper specifies licenses for the recordings and code.Source: Data and Code · p. 12

SUPPLEMENTARY MATERIAL FOR “EYE TRACKING SIGNAL QUALITY EVALUATION: META QUEST PRO VS HTC VIVE PRO EYE VS HTC VIVE FOCUS VISION”

  • The supplement provides details referenced by the main paper and defines QP, FV, PE, RAN127, RAN63, and U|E.It states that its tables and figures are generated from this study and that participants are identified by pseudonymous codes.Source: SUPPLEMENTARY MATERIAL FOR “EYE… · p. 15

S1 Protocol, Apparatus and Geometry

The supplement documents the first located evaluation of the VIVE Focus Vision eye tracker and details the protocol, device geometry, timing, and validation procedures used for headset comparisons.

S1 Protocol, Apparatus and Geometry

S2 Timing

The supplement documents device-clock timing and robustness checks for spatial accuracy, percentile definitions, and off-target windows.

S2 Timing

  • The FV logger polls twice per rendered frame and writes a row only when the pupil timestamp changes, so logged cadence does not establish that every timestamp has a new gaze estimate.Table S6 summarizes device-clock intervals and long intervals; PE’s 1 ms clock resolution makes its intervals 8 or 9 ms.Source: S2 Timing · p. 17, Table S6 · p. 17, Table S7 · p. 17

S3 Spatial Accuracy

  • Leaving out windows with less than 80% of expected samples changes which headset contrasts cross the 0.05 threshold; the PE–FV block-change contrast lies on both sides of it.Table S8 gives the primary paired contrasts, while Tables S9 and S10 report the sample-threshold sensitivity checks.Source: S3 Spatial Accuracy · p. 17, Table S8 · p. 18, Table S9 · p. 18

S3.1 User and Error Percentiles

S3.2 Off-Target Windows

  • Windows exceeding the median error of spatial neighbors by more than 5° are flagged, and Table S13 compares medians and U95|E95 with and without them.Flagged windows remain in the main analysis.Source: S3.2 Off-Target Windows · p. 19, Table S13 · p. 19

S4 Accuracy Across the Field of View

The supplement examines error across target positions and reports how geometric conventions affect linearity, offsets, and recalibration analyses.

S4 Accuracy Across the Field of View

S5 Linearity, Offsets and Recalibration

  • Using head-origin angles removes headset differences in horizontal RAN127 linearity slopes, but FV’s vertical compression remains.Changing the geometric convention also moves two centroid-error contrasts across the significance threshold; Tables S19 and S20 report signed offsets and linear recalibration, respectively.Source: S5 Linearity, Offsets and Recalibration · p. 21, Table S16 · p. 21, Table S18 · p. 22
  • Figure S4 and Table S19 report signed gaze-minus-target offsets, while Table S20 evaluates linear user-supplied recalibration as an in-sample upper bound.Table S16 compares linearity slopes under eye-origin and head-origin target angles.Source: Table S16 · p. 21, Table S19 · p. 22, Table S20 · p. 22

S6 Difference Between the Two Blocks

The supplement compares the two random-saccade blocks and investigates temporal structure in gaze signals, including repeated values and correlated increments.

S6 Difference Between the Two Blocks

S7 Precision and Temporal Structure

S8 Data Loss

The supplement defines device-clock loss categories and reports how repeated timestamps, long application frames, and intervals without valid samples affect the data accounting.

S8 Data Loss

  • Expected samples equal device-clock span times timestamp cadence; lost samples are classified as invalid, frozen, or dropped.Frozen samples occupy missing slots in intervals containing repeated timestamps; dropped samples are the remaining gaps.Source: S8 Data Loss · p. 27
  • Table S31 reports total loss by task; Tables S32 and S33 report QP intervals over 0.1 s and short windows without valid samples.S31 gives median and 75th-percentile loss; S32 reports interval counts and durations, and S33 covers three window lengths.Source: Table S31 · p. 27, Table S32 · p. 27, Table S33 · p. 27
  • Keeping repeated QP rows increases the RAN63 tail because 13 formerly empty windows contain only frozen gaze; timestamp-zero rows lack geometry and a validity flag.Table S34 compares QP accuracy with repeated-timestamp rows removed or retained.Source: S8 Data Loss · p. 27, Table S34 · p. 28
  • Table S35 relates long application frames to FV two-tick intervals and QP interruptions; its long-frame threshold is more than 1.5 times the recording’s median frame.The caption describes FV’s poll pattern as writing the first poll’s row, returning the same sample on the second poll, and receiving a sample two ticks later at the next frame’s first poll.Source: S8 Data Loss · p. 27, Table S35 · p. 28
  • The QP interruption count is 230 between fresh task-row timestamps, versus 233 when three intervals beginning at previously timestamped task-start rows are included.The main-paper count includes those three task-start intervals.Source: S8 Data Loss · p. 27

S9 Eyebrow Raising and Eye Closure

The supplement revisits eyebrow-raising and eye-closure analyses, examines per-eye rays and reliability, and documents the independent reproduction and its limits.

S9 Eyebrow Raising and Eye Closure

  • The preprint exclusion mask removes participant–headset records with the largest raised errors and lowers raised medians on all headsets.The supplement compares that mask with the main paper’s no-exclusion rule.Source: S9 Eyebrow Raising and Eye Closure · p. 28, Table S36 · p. 28
  • QP’s validity flag stayed set during instructed closures and blink screens, while its confidence did not mark instructed closures; no independent blink record was logged.The QP eye-closure table reports that all rows carried gaze geometry and a valid flag; the blink screens had no fixation target.Source: S9 Eyebrow Raising and Eye Closure · p. 28, Table S38 · p. 29, Table S39 · p. 29
  • The closure-onset rules differ by headset: FV uses its first invalid row, PE uses both openness values below half, and onset is not detectable on QP.The PE onset differs from the main paper’s loss-latency rule, which uses the first of three consecutive invalid rows and is 0.68 s.Source: Table S37 · p. 29

S10 Per-Eye Rays and Reliability

  • On QP, left and right gaze rays have nearly the same vergence angle across participants and within windows, so they are not independent measurements.Table S40 also reports per-eye accuracy and split-half reliability of participant mean error.Source: S10 Per-Eye Rays and Reliability · p. 29, Table S40 · p. 30

S11 Sources and Reproduction

  • The independent reproduction recomputed the main analyses from raw recordings and reproduced their values, but did not recompute analyses A1–A12, exact Hodges–Lehmann intervals, split-half reliability, or per-eye accuracy.It corrected three points: spectral-ratio frequency bands, eye-closure counts and onset rule, and effective-rate and expected-sample definitions.Source: S11 Sources and Reproduction · p. 30
Loading 2610.32915v2…