Source-linked AI summary
All Reality: Virtual, Augmented, Mixed (X), Mediated (X,Y), and Multimediated Reality
Steve Mann, Tom Furness, Yu Yuan, Jay Iorio, Zixin Wang
TL;DR
The paper addresses the limits of one-dimensional reality–virtuality taxonomies and proposes a broader framework that also represents modification and otherwise invisible phenomena. It develops Mediated and Multimediated Reality, reports a sound-speed demonstration, and concludes that Multimediated Reality subsumes the other reality categories.
Problem
Existing reality taxonomies do not capture technologies that modify reality or sensory technologies outside the one-dimensional reality–virtuality mixer.
Method
The paper constructs taxonomies for Virtual, Augmented, Mixed, X-, Mediated, and Multimediated Reality using virtuality, mediality, and multisensory dimensions.
Results
Multimediated Reality is presented as a proper superset of mediated, mixed, augmented, and virtual reality, and a demonstration calculates sound speed as 350 m/s.
Takeaways & Limitations
Multimediated Reality can make real but otherwise invisible phenomena available to human senses through cross-sensory mappings.
Takeaways & Limitations
The framework prefers undigital sensing and effectors, while current digital cameras still exhibit tonal and level quantization.
Abstract
from arXiv · showhide
The contributions of this paper are: (1) a taxonomy of the "Realities" (Virtual, Augmented, Mixed, Mediated, etc.), and (2) some new kinds of "reality" that come from nature itself, i.e. that expand our notion beyond synthetic realities to include also phenomenological realities. VR (Virtual Reality) replaces the real world with a simulated experience (virtual world). AR (Augmented Reality) allows a virtual world to be experienced while also experiencing the real world at the same time. Mixed Reality provides blends that interpolate between real and virtual worlds in various proportions, along a "Virtuality" axis, and extrapolate to an "X-axis". Mediated Reality goes a step further by mixing/blending and also modifying reality. This modifying of reality introduces a second axis. Mediated Reality is useful as a seeing aid (e.g. modifying reality to make it easier to understand), and for psychology experiments like Stratton's 1896 upside-down eyeglasses experiment. We propose Multimediated Reality as a multidimensional multisensory mediated reality that includes not just interactive multimedia-based reality for our five senses, but also includes additional senses (like sensory sonar, sensory radar, etc.), as well as our human actions/actuators. These extra senses are mapped to our human senses using synthetic synesthesia. This allows us to directly experience real (but otherwise invisible) phenomena, such as wave propagation and wave interference patterns, so that we can see radio waves and sound waves and how they interact with objects and each other. Multimediated reality is multidimensional, multimodal, multisensory, and multiscale. It is also multidisciplinary, in that we must consider not just the user, but also how the technology affects others, e.g. how its physical appearance affects social situations.
1 HISTORICAL BACKGROUND & CONTEXT
The paper traces competing definitions of virtual, augmented, mixed, and X-reality, then argues that modifying reality requires a second axis beyond the reality–virtuality continuum.
- Virtual, Augmented, Mixed, and X-Reality: VR replaces the real world with a computer-generated or recorded virtual experience, whereas AR overlays computer-generated content onto the real-world experience.
- Virtual, Augmented, Mixed, and X-Reality: Mixed reality places experiences along a continuum between reality and virtuality, often represented as an X-axis or slider.
- Virtual, Augmented, Mixed, and X-Reality: XR has three definitions: a general sensory-extension axis, an interpolation between reality and virtuality, and a cross-reality subset combining sensor/actuator networks with shared virtual worlds.
- Mediated Reality: All XR definitions use an X-axis spanning reality and virtuality, but the paper notes that this one-dimensional model does not cover technologies that modify reality.
- Mediated Reality: Mediated Reality adds a Mediality Y-axis to the Virtuality X-axis, covering deliberate modifications such as upside-down vision and HDR welding helmets.
- Unintentionally Mediated Reality: Head-mounted displays are reported as capable of both lasting harm and therapeutic effects, motivating caution about technology as an intermediary.
MEDIALITY, �
The Mediated Reality continuum represents both virtuality and reality modification, using separate axes to distinguish how much an experience is virtualized and altered.
- MEDIALITY, �: The continuum’s MIX X-axis runs from reality to virtuality, while the Mediality Y-axis runs from slightly to extremely modified.
- MEDIALITY, �: Stratton’s upside-down eyeglasses occupy the high-mediality, zero-virtuality region, whereas the EyeTap HDR welding helmet combines extreme mediation with moderate virtuality.
2 MULTIMEDIATED REALITY
The section develops multimediated and phenomenological realities that use physical sensing and display processes to make otherwise invisible phenomena directly perceivable. It also situates these systems among sensory attenuation, darkroom, submersive, and measurement applications.
- Sensory attenuation: Sensory attenuation is identified as an underexplored area spanning sleep masks, interactive bathing or flotation environments, and darkroom experiences.The section connects controlled sensory environments with interactive multimedia and reality-based media.
- Multimediated Reality Darkroom: Interactive darkroom experiments use sensors, amplifiers, and light bulbs to make otherwise invisible physical phenomena visible.The experiments began in the 1970s and included sweeping for surveillance-camera sightfields through video feedback.
- Multisensory mapping: Radio and sound waves can be mapped onto vision and touch, allowing their physical waveforms and interference patterns to be experienced at their real-world scale.The radio-wave example uses an antenna, XY plotter, and light bulb; multimediated systems can retain volumetric waveform interference patterns.
- Phenomenological Reality: Phenomenological Reality directly connects a physical quantity to the sensed quantity through physical means rather than computer simulation.A light bulb can simultaneously sense a quantity and display it, producing exact alignment without a separate tracking mechanism.
- Measurement and teaching: A sound-wave display measured a 7 cm wavelength for a 5 kHz tone and produced an experimental error of approximately 0.86 percent.The displayed waveform used 21 cycles across 1.5 meters and was compared with the theoretical speed of sound at 27 degrees C.
- Comparison with existing instruments: Unlike traditional instruments that remap quantities between senses, multimediated reality preserves direct physical alignment between measured quantities and their source world.This alignment supports direct teaching and measurement of wave propagation while keeping sound and radio waves visible in their existing environment.
3 MULTIMEDIATED REALITY IS MULTISCALE, MULTIMODAL, MULTISENSORY, MULTIVEILLANT, AND MULTIDIMENSIONAL
Multimediated reality extends interactive multimedia beyond synthetic visual experience by combining multiple scales, modalities, senses, and dimensions. It uses synthetic synesthesia to expose otherwise inaccessible wave properties and patterns.
- Multimediated reality treats technology as an extension of human minds and bodies through AI and human-in-the-loop intelligence.Its intelligence arises through a computational feedback loop involving the user.
- It spans wearable devices and environmental technologies such as smart rooms and darkrooms.
- Synthetic synesthesia maps existing or extra senses into perceptual outputs, enabling experiences such as seeing sound or radio waves.This makes the system multisensory and multimodal.
- Radio and sonar waveforms can be engaged directly as multidimensional signals, exposing properties such as multipolarization and multipath propagation.
- Phase can be displayed as color and magnitude as light quantity, making constructive and destructive sound-wave interference visible.The resulting display shows interference variations through light quantity and phase fronts through color.
- The system also supports underwater sound exploration, including sonar arrays and an aquatics facility where users can float among sound waves.
4 MULTIMEDIATED REALITY CONTINUUM
The Multimediated Reality Continuum replaces one-dimensional reality–virtuality descriptions with a multidimensional space that includes sensory deprivation, phenomenality, physical scale, veillance, and complex-valued data.
- Existing VR and AR taxonomies do not accommodate all presented systems, motivating a multidimensional reality continuum.
- The continuum begins with sensory deprivation, passes through sensory attenuation and Reality, and extends beyond Reality into Extended reality.
- Its dimensions include Reality, Virtuality, Phenomenality, Fluidity or Fluentity, physical scale, and sociopolitical Veillance.
- Multimediated reality spans environment and invironment scales, from wearables and prostheses to smart environments, while connecting surveillance with sousveillance.
- The framework also considers undigital sensing and actuation to reduce artefacts of digital computation and support undigital experiences.
- The proposed “*R” name uses the wildcard asterisk to denote the space of all realities, including complex-valued and multidimensional data.
- The continuum encompasses nested reality categories, with multimediated reality represented as a broader set containing the other forms.
5 SUMMARY AND CONCLUSIONS
The paper presents Multimediated Reality as a superset of existing reality categories and as an interactive framework for multidimensional, multisensory experiences. It also demonstrates how real but otherwise invisible phenomena can be made perceptible.
- Multimediated Reality is presented as a proper superset of mediated, mixed, augmented, and virtual reality.
- Its interactive multimedia framework is multidimensional, multisensory, and multimodal.
- The reality categories are mostly nested, except XR2, which is a proper subset of mixed reality.
- Multimediated reality can capture and display multidimensional complex-valued fields and make real but otherwise invisible phenomena perceptible.
- The taxonomy includes seven realities and their approximate dates of introduction.