All Reality: Virtual, Augmented, Mixed (X), Mediated (X,Y), and Multimediated Reality

1.1 Virtual, Augmented, Mixed, and X-Reality

• VR replaces reality with a simulated experience, while AR overlays computer-generated content on reality; mixed reality places experiences along a real-to-virtual continuum.VR can also use interactive playback of recorded media, as in the Aspen Movie Map.Source: §1.1 · p. 1
• XR has three distinct meanings: sensory extrapolation beyond reality, interpolation across reality and virtuality, or cross-reality linking sensor/actuator networks with shared virtual worlds.The section notes that XR is variously defined as a superset of mixed reality, equivalent to mixed reality, or a subset of it.Source: §1.1 · p. 2

1.2 Mediated Reality (X-Y Reality)

• Technology can mediate reality as an intermediary with the environment, either through deliberate design or as an unintended side effect.These two forms are considered separately in the following subsections.Source: §1.2 · p. 2

1.3 Deliberately mediated reality

• Deliberate mediation has been used in perceptual experiments, including Stratton’s upside-down glasses, and vision aids such as HDR welding helmets that reduce arc brightness and restore shadow detail.Other experiments used left-right reversing glasses, photographic-negative views, and computer-mediated reality.Source: §1.3 · pp. 2–3

1.4 Unintentionally Mediated Reality

• Video-see-through AR unintentionally mediates reality because both real and virtual objects pass through a display; HMDs can cause lasting harm or support treatment, warranting care.The cited examples include treatment for PTSD, phobias, and brain damage, as well as potential brain damage.Source: §1.4 · p. 3

1.5 The Mediated Reality (X,Y) Continuum

• The Mediated Reality continuum adds a mediality (Y) axis for modification to the virtuality (X) axis, locating Stratton’s glasses and HDR welding helmets differently.Stratton’s glasses involve little virtuality but substantial mediality; the EyeTap HDR welding helmet combines strong modification with a moderate amount of virtuality.Source: Figure 5 · p. 3, §1.5 · p. 3

2.1 Technologies for sensory attenuation

• Sensory attenuation technologies are underexplored, with proposed uses including interactive sleep masks, multimedia bathing and relaxation, and darkroom lightpainting.The passages note that existing continua do not directly address devices such as sunglasses, sleep masks, ear plugs, and sensory attenuation tanks.Source: §2.1 · pp. 3–4

2.2 Multimedia in Photographic Darkrooms

• In photographic darkrooms, moving lights create visible patterns through persistence-of-exposure, a practice related to lightpainting and directly viewed flow arts.The passage describes controlled lighting in darkness as a longstanding theatrical practice.Source: §2.2 · p. 4

2.3 Multimediated Reality Darkroom

• A bulb can reveal a camera’s sightfield through physical feedback, with the same filament sensing and displaying the signal in exact alignment without tracking.The authors call such direct physical displays Phenomenological Reality; a radio sensor mounted to a bulb can also map radio signals onto vision and touch.Source: §2.3 · p. 5, Figure 9 · p. 5
• A 5 kHz sound wave yielded 350 m/s from 7 cm cycles, about 0.86% error versus the theoretical 347 m/s at 27°C.The authors propose the setup as a teaching tool for making wave propagation directly visible.Source: §2.3 · p. 6
• Volumetric scanning stores wave measurements as 3D point clouds that can be reconstructed, manipulated, and analyzed from arbitrary angles and slices.The setup scans sound interference patterns through space and displays them using light, including patterns that can be animated at a slowed or zero apparent wave speed.Source: §2.3 · p. 6, Figure 12 · p. 7

2.4 Comparison with existing measuring instruments

• Multimediated reality provides direct physical alignment between measured quantities and their real-world sources, allowing sound or radio waves to be seen in perfect alignment at 1:1 scale.Sound or light speed can be represented as zero or a small number, making wave propagation easy to study. Humanistic Intelligence puts humans in the computational feedback loop, requiring reciprocal sensing between people and machines.Source: §2.4 · pp. 7, 9, Figure 14 · p. 9

3 MULTIMEDIATED REALITY IS MULTISCALE, MULTIMODAL, MULTISENSORY, MULTIVEILLANT, AND MULTIDIMENSIONAL

• Humanistic Intelligence treats technology as an intermediary in a computational feedback loop with people and their environment, extending their minds and bodies.The paper considers AI alongside human-in-the-loop AI, also called Humanistic Intelligence.Source: §3 · p. 9
• Multimediated reality spans wearable technology and environmental systems such as smart rooms, making it multiscale and multiveillant.The paper illustrates this scope with wearable technology and smart darkrooms.Source: §3 · p. 9

3.1 Multisensory Synthetic Synesthesia

• Synthetic synesthesia maps multimedia across existing senses, such as seeing sound, or adds sensory access beyond them, such as seeing or feeling radio waves.This makes multimediated reality multisensory and multimodal.Source: §3.1 · p. 9

3.2 Multidimensional Multimediated Reality

• Directly engaging radio and sound waveforms phase-coherently enables experience of multidimensional signal properties, including multipolarization and multipath propagation.Unlike systems that use waves only for tracking and position sensing, this approach makes the waveforms part of the output space.Source: §3.2 · p. 9
• For baseband radio waves, a color wheel represents complex numbers by encoding phase as color and magnitude as the quantity of presented light.In the sound-wave display, phase fronts appear as color variations, while constructive and destructive interference appears as changes in light quantity.Source: §3.2 · p. 9
• The system also supports exploration of underwater sound waves through a sonar array and an aquatics facility with underwater MR eyewear.The weightless state in water allows users to feel as if they are floating among the waves.Source: §3.2 · p. 9

4 MULTIMEDIATED REALITY CONTINUUM

• The continuum begins at absent sensory stimulation and extends beyond ordinary reality, combining dimensions such as Reality, Virtuality, Phenomenality, and Fluidity/Fluentity.For example, AR combined with Phenomenality yields PAR, while AR combined with Fluentity yields Submersive Reality; swimming underwater with a SWIM spans all four dimensions [20].Source: §4 · pp. 9–10, Figure 18 · p. 12
• The continuum’s dimensions can overlap rather than form an orthonormal basis, as Metaveillance, Wearability, and Phenomenality are closely related.Metaveillance combines surveillance and AI with sousveillance and HI; Wearability is closely correlated with Existentiality, and SWIM is an example of Phenomenality.Source: §4 · p. 10
• Undigital reality aims to avoid digital-computation artifacts by sensing above the Nyquist rate and using undigital sensors and effectors across all six signal-flow paths.This is especially important in Submersive Reality, where swimming’s fluidity makes quantization noise and pixelation unacceptable.Source: §4 · p. 10

4.1 Multimediated Reality is “*R” (All R)

• “*R” names the full Multimediated Reality continuum; “ZR” is an alternative when an asterisk is inconvenient, emphasizing multidimensional, potentially complex-valued data.The passage gives Z = X + iY as an example and notes that the space can extend beyond the two dimensions of the Argand plane.Source: §4.1 · pp. 10, 12
• The continuum spans sensory attenuation between sensory deprivation and reality, extrapolation beyond reality, and additional dimensions including Metaveillance, Wearability, Kineveillance, Imaginality, Therapality, and Digitality.Digitality represents undesirable quantization artifacts, while the paper advocates undigital computing [24].Source: Figure 18 · p. 12

• Multimediated Reality is presented as a proper superset of mediated, mixed, augmented, and virtual reality.Source: §5 · p. 13
• Its interactive multimedia is multidimensional, multisensory and cross-sensory, multimodal, and multidisciplinary across fields including arts, sciences, engineering, and health.Examples include synthetic synesthesia and direct interaction with complex-valued mathematical quantities [Gerson et al 2015].Source: §5 · p. 13
• The realities form mostly a nested set, except XR2, which is a proper subset of mixed reality limited to a specific kind of virtual world.Source: §5 · p. 13, Figure 20 · p. 13
• It can capture and display complex-valued fields and other quantities, making otherwise invisible features of the surrounding world perceptible as “real reality.”Source: §5 · p. 13