Multimodality in VR: A survey

1 INTRODUCTION

• VR expands media interaction through wider views, spatial sound, and camera control, with potential to increase realism, immersion, and engagement.These features also support applications such as education, training, rehabilitation, neuroscience, and virtual cinematography.Source: §1 · p. 1
• Sensory cues must be synchronized within context-sensitive integration windows to be perceived as belonging to one event.Synchronization sensitivity varies by context, task, and individual.Source: §1 · p. 2

1.1 The five senses

• Sight, hearing, smell, taste, and touch each contribute distinct information, and the senses influence one another.Hearing can provide information about unseen locations, while smell and taste are often linked to emotions or memories.Source: §1.1 · pp. 2–3

1.2 Proprioception

• Proprioception conveys static position and dynamic motion information and contributes to presence; conflicts with visual or vestibular cues may cause discomfort.The survey describes proprioception as awareness of the spatial and mechanical status of the musculoskeletal framework.Source: §1.2 · p. 3

1.3 Reproducing sensory modalities in VR

• Consumer VR commonly provides visual and auditory feedback, while haptic, olfactory, gustatory, and proprioceptive support varies across devices and techniques.The survey reviews haptic options including wearables, controllers, ultrasound, and electrical muscle stimulation, as well as hardware for smell and taste.Source: §1.3 · pp. 3–4

1.4 Related surveys

• Unlike prior reviews focused on individual senses or particular VR topics, this survey synthesizes sensory interactions and their applications across disciplines.Its focus includes visual, auditory, haptic, and proprioceptive feedback.Source: §1 · p. 2, §1.4 · p. 5

1.5 The challenges of multimodality

• Empirical knowledge and multimodal VR data remain scarce, while many modalities and their interactions are unexplored.The survey focuses on visual, auditory, haptic, and proprioceptive modalities; consumer devices often limit haptic and proprioceptive feedback, and olfactory and gustatory feedback are usually absent.Source: §1.5 · p. 5
• Designing multimodal experiences is challenging because each added modality increases problem dimensionality and may distract or overload users.Such overload may hamper the experience and diminish user satisfaction.Source: §1.5 · pp. 5–6

Scope

• The survey reviews significant work on multimodality's effects in VR and how sensory combinations affect perception, creation, and interaction with virtual experiences.Source: §2 · p. 6

Organization

• Its user-focused scope covers realism, immersion and presence, attention during exploration, and performance in task completion.Source: §2 · p. 6
• Further coverage addresses perceptual illusions, navigation, applications, and future research directions.Applications are reviewed in Section 8, followed by discussion and future avenues in Section 9.Source: §2 · p. 6

3.1 Perception of the environment

• Spatialized or otherwise appropriate auditory information can improve environmental realism and presence, but may increase cognitive load or reduce comfort when integration is poor.Validation studies favored added auditory information, while a survey reported that it may also cause discomfort and lower experience quality.Source: §3.1 · p. 7, Figure 3 · p. 8
• Expected movement and spatial coherence matter: visual-vestibular conflicts can cause motion sickness and diminish perceived realism.Users generally expect virtual movement to match real movement; rotating the environment beyond a noticeable threshold can create a mismatch.Source: §3.1 · p. 7
• Synchronous visual-tactile and visual-motor cues can induce body distortion, while physical touch, smell, or taste can raise perceived realism and preference.Multisensory systems have also been reported to outperform traditional audiovisual systems in presence and user preference, though the limits of integration remain open.Source: §3.1 · pp. 7–8

3.2 Perception of the self

• Embodiment comprises self-location, agency, and body ownership; combining sensory information can strengthen presence and users’ reported involvement.The sense of embodiment concerns owning, controlling, and being inside a body.Source: §3.2 · pp. 8–9
• Walking-related auditory, visual, and vibrotactile cues can enhance self-motion and presence, while incongruent anchor cues diminish perceived realism.Providing dynamic tactile signals remains complex, and how to use them to increase presence is an open research question.Source: §3.2 · pp. 9–10
• Multisensory overload and cues that violate interpersonal space can diminish presence and comfort, making a balance of cues and user capacity important.Establishing guidelines for this balance remains an open avenue.Source: §3.2 · p. 10

• Most VR saliency work uses head position and gaze to estimate attention from a largely unimodal visual perspective, despite evidence that multiple senses can influence attention.The figure contrasts an input panorama with an estimated saliency map.Source: §4 · p. 11, Figure 5 · p. 11
• VR multimodal saliency prediction remains in an early phase, with few studies and open questions about competing audiovisual cues and other senses.The survey calls for varied multimodal-stimulus datasets to support new attentional models.Source: §4 · p. 11
• Because users control the camera and may attend to different regions, multimodal cues can help guide attention toward creators’ intended content.The survey notes that this variability makes assumptions about user behavior and attention difficult.Source: §4 · pp. 11–12
• Sound and movement can direct attention, offering subtler guidance than visual manipulations that may disrupt presence, immersion, or suspension of disbelief.Visual arrows, rear-region insets, and marked points of interest are examples of guidance techniques described as often intrusive.Source: §4 · p. 12, Figure 6 · p. 12
• Auditory cues can redirect attention after cinematic cuts, but distractor audio may feel unnatural and too many sound sources can clutter a scene.Directional sound can help users converge on the main action even when misaligned with the region of interest.Source: §4 · p. 13

• Audiovisual, tactile-visual, and olfactory cues have improved detection, search times, confidence, or performance in studies, though many used simple laboratory stimuli.The survey says applicability and limitations in more complex scenarios remain to be studied.Source: §5 · p. 13
• When multiple senses are involved, users start cognitive processing faster and can attend to more cues and details, yielding a richer, more complete, coherent experience.Source: §5 · p. 14
• Multimodality can benefit spatial reasoning, navigation, search, and direct interaction, including auditory guidance when relevant objects lie outside users’ field of view.Examples include auditory-only navigation, auditory-haptic spatial reasoning, and auditory assistance for touching virtual objects outside the field of view.Source: §5 · p. 14
• Some multimodal combinations can suppress or diminish abilities, while enabling olfactory or haptic modalities on consumer devices remains a future challenge.The survey recommends care when designing multimodal experiences.Source: §5 · p. 14

• Multimodal manipulation can guide users, reduce required physical space, or help reduce simulator sickness, but forced guidance may diminish immersion or break suspension of disbelief.The survey notes that multimodal cues may improve the effectiveness of these techniques.Source: §6 · p. 15
• Tactile funnelling can convey contact using a small fixed set of haptic devices, and multisensory cues can help simulate depth or alter perceived object weight.The perceived contact point can be shifted by adjusting the relative intensities of adjacent tactile devices.Source: §6 · p. 15
• Proprioceptive and haptic cues can support arm or whole-body ownership illusions, while altered virtual limbs can extend users’ reach without losing body ownership.Source: §6 · p. 15
• Auditory stimuli can degrade visual performance in a specific VR spatiotemporal arrangement, while broader crossmodal interactions may either enhance or diminish perception.The survey calls for deeper study of facilitatory and inhibitory interactions in VR and of whether traditional illusions carry over to virtual environments.Source: §6 · pp. 16–17, Figure 8 · p. 16

7 MULTIMODALITY IN NAVIGATION

• Constrained physical space limits 1:1 movement mapping, and redirected walking techniques manipulate users or environments to enable exploration of larger virtual worlds.Most earlier techniques described here rely on visual cues rather than other sensory modalities.Source: §7 · p. 17
• Auditory cues can redirect perceived rotation: users could be turned about 20% more or 12% less than their physical rotation, while acoustic redirected walking showed similar detection thresholds of ±20%.Auditory distractors were also reported safe for complex navigation; behavior differed between auditory-only and audio-visual conditions.Source: §7 · p. 17
• Proprioceptive, visual, and haptic cues have been used to manipulate jumping and distance perception, while foot-applied haptics can influence audiovisual self-motion illusions.Movement-directional wind, footstep vibrations, and footstep sounds were associated with improved experience and realism during non-fatiguing walking.Source: §7 · p. 18
• When users cannot move, externally generated visual motion can produce vection, and auditory cues can increase its strength compared with visual cues alone.Moving sounds enhance circular vection; real and virtual foreground objects can serve as visual, auditory, and haptic reference points for postural responses.Source: §7 · p. 18
• Some findings come from rigidly controlled laboratory setups, and the effects of senses beyond auditory and haptic cues in navigation remain unexplored.The survey identifies effectiveness in more complex viewing scenarios as a future research avenue.Source: §7 · p. 18

8.1 Medicine

• Medical VR supports image visualization, planning, teaching, and simulation; multimodal feedback in surgery training has been reported to improve learning effects.Medical systems also combine speech and gestures for interacting with patient information.Source: §8.1 · p. 19
• Multimodal VR exposure therapy can provide controlled exposure to fears, and the illusion of physically touching a virtual spider was reported to increase treatment effectiveness.Exposure in multiple contexts was suggested to improve generalizability; large VR exposures may hinder distinguishing real from virtual worlds.Source: §8.1 · pp. 19–20
• Rehabilitation studies report reduced phantom limb pain with multimodal feedback, improved daily activities and real-life transfer with a VR neglect treatment, and benefits from spatialized sound for postural control.The neglect treatment combined contralesional-arm sensorimotor activation with visual scanning training.Source: §8.1 · p. 20

8.2 Education and training

• Educational VR studies report student motivation and immersion, richer perceptual experiences from virtual manipulatives, and learning benefits from multimodal chemistry experiments.A speech-based multimodal interface also yielded accuracy benefits over tactile diagrams for visually impaired and blind users.Source: §8.2 · p. 21
• In a multisensory educational game, olfactory stimuli made the experience more enjoyable and improved users’ performance and overall learning.The survey also notes the use of immersive serious games for learning processes and transfer.Source: §8.2 · pp. 21–22
• Multimodal virtual training spans manufacturing, air-traffic-control warning detection, and other complex skills; reviewed work reports learning transfer and potential replacement of some real training.Virtual training is described as less expensive and time-consuming, while missing useful modalities such as haptics can reduce effectiveness relative to hands-on experience.Source: §8.2 · p. 22

8.3 Entertainment

• In entertainment, multimodality is used to pursue realism and engagement across games, cinematic VR, and narrative experiences.Olfactory cues remain in an early exploratory phase in videogames, while multisensory walking cues have been reported to enhance vection and presence.Source: §8.3 · pp. 23–24
• For VR narratives, directional sound can influence how users explore environments, and studies have explored guiding attention through light, movement, sound, olfactory, and haptic stimuli.The survey notes that sound design warrants special attention because users cannot see the whole virtual scenario at once.Source: §8.3 · p. 24

9 CONCLUSIONS

• Multimodal feedback can be both a liability when mishandled and a strength that boosts realism, directs attention, or improves user performance.The survey frames compelling user experience as central to VR’s potential.Source: §9 · pp. 24, 26
• Multimodal setups increase cost and complexity; consumer haptics remain rudimentary, while taste and smell are largely untapped in hardware and research.Multimodal input also raises bandwidth and computational demands, though it can help maintain realism and immersion with lower-quality visuals.Source: §9 · p. 26
• Most studies of sensory interactions use constrained laboratory experiments, so whether their insights generalize to consumer free-viewing scenarios remains an open question.Studies of three or more modalities are rarer, and the expanding parameter space can become intractable.Source: §9 · p. 26
• Future work needs public, carefully crafted datasets, benchmarks, and reproducible experimental setups; VR itself offers reproducibility advantages over physical setups.Data-driven approaches offer a tool for addressing multimodal interaction, but the survey calls for shared resources to support them.Source: §9 · pp. 26–27
• Understanding sensory interactions can help practitioners create believable experiences with limited resources and inform algorithm, content, and hardware design.The survey presents these as routes toward better virtual experiences.Source: §9 · p. 27