PaperlayerView exact revision on arXiv

Source-linked AI summary

Passing: An Endless Journey through Reconstructed Spacetime with AI-Generated Sound

Akira Takahashi, Chihiro Nagashima, Zhi Zhong, Shusuke Takahashi, Yuki Mitsufuji

arXiv:2609.27489v1cs.SDcs.AI

TL;DR

Passing addresses how sound and listening can operate when a monorail recording’s spatial and temporal relations are disrupted. It reconstructs the footage as nonlinear video and generates synchronized sound with SpecMaskFoley; the installation uses the sound as an interpretation, while artist, AI, and audience share creative agency.

  • Problem

    Reordered motion and spatial relations mean the reconstructed video has no single physically correct acoustic counterpart, so the task is to generate sound responsive to its altered flow.

    Source: §1 · p. 2

  • Method

    Passing resamples a monorail recording as a spatiotemporal volume, feeds the nonlinear visual flow to real-time SpecMaskFoley, and lets viewer presence influence the visual path.

    Source: §1 · p. 2, §3.1 · p. 3, §4 · p. 5

  • Results

    The installation generated 32-kHz stereo audio in real time; A/B loops tended to retain a train-interior atmosphere, while C/D transitions allowed stronger sonic variation.

    Source: §4.3 · p. 5, §5.1 · p. 6

  • Takeaways & Limitations

    Passing presents authorship as negotiated among artist, AI, and audience, with audience presence influencing the audiovisual path and the viewer constructing personal meaning.

    Source: §6 · p. 6

    Because the model lacks a body or lived spatial experience, its output may contain mismatches, ambiguities, or unexpected image-sound correspondences.

    Source: §5.2 · p. 6

Abstract

from arXiv · show

This paper introduces Passing, an interactive audiovisual installation that generates an endless journey from a single continuous monorail-window recording by reconstructing it as a spatiotemporal volume. Rather than replaying the footage linearly, the work resamples its spatial and temporal structure along nonlinear trajectories, producing a continuously passing landscape whose depth, speed, and temporal order become unstable. A camera-based viewer-presence detection system estimates whether a viewer is present in the viewing zone and uses this presence state to influence transitions among rendered video sequences. The resulting video stream is fed into SpecMaskFoley, a real-time video-to-audio synthesis model that generates a synchronized soundscape for the reconfigured image. The model is not used to reconstruct an objectively correct soundtrack, but functions as a speculative listener, proposing a possible auditory interpretation of a world whose conventional spatial and temporal premises have been disrupted. Passing distributes creative agency across the artist, who defines the rules of spacetime reconstruction; the AI model, which interprets the emergent visual flow as sound; and the audience, whose embodied presence influences the audiovisual trajectory. Through this structure, the work investigates how authorship and listening may be negotiated among human intention, machine inference, and audience interpretation. Artwork page: https://ryufurusawa.com/passing

1 Introduction

Passing reconfigures a monorail recording’s spacetime, unsettling the linear time and three-dimensional space on which ordinary perception relies. It asks how sound and listening can operate in that altered world, using AI-generated sound as an interpretation rather than a correct reconstruction.

1 Introduction

  • Reordered motion and spatial relations leave the reconstructed video without a single physically correct acoustic counterpart.The sound must respond to a visually coherent but physically impossible flow, rather than recover the original scene’s soundtrack.Source: §1 · p. 2
  • SpecMaskFoley [2] generates a new auditory interpretation for each moment of nonlinear video instead of reproducing an objectively correct soundtrack.The model is framed as a speculative listener responding to reconfigured spacetime.Source: §1 · p. 2
  • Creative agency is distributed among the artist, the AI model, and an audience whose presence or absence influences the visual path.The artist defines reconstruction rules, while the model interprets the resulting visual flow as sound.Source: §1 · p. 2

2 Related Works

Passing places its spacetime reconstruction in media-art lineages of altered recorded images, while using video-to-audio synthesis to interpret rather than restore the transformed world. It extends discussions of AI authorship by making audience presence part of the agency structure.

2.1 Spatiotemporal Manipulation in Media Art

  • Passing extends prior spacetime-manipulation practices by traversing a monorail recording as a volume, with viewer presence shaping playback of prepared trajectories.Figure 2 illustrates the transformation: rotating a cross-section by 90° exchanges the temporal axis with the horizontal image dimension.Source: §2.1 · p. 2, Figure 2 · p. 3

2.2 Speculative Video-to-Audio Synthesis

  • Unlike V2A work aimed at plausible sound reconstruction, Passing feeds reconfigured video to SpecMaskFoley [2] for a possible interpretation of the altered world.Here, synchronization supports artistic interpretation rather than objective acoustic reconstruction.Source: §2.2 · p. 2

2.3 Distributed Agency in AI Art

  • Passing extends AI-art discussions of distributed authorship by adding the audience to a three-way agency loop: presence shapes the visual material the AI interprets as sound.The artist defines the reconstruction rules; audience behavior indirectly changes the AI’s real-time input.Source: §2.3 · p. 3

3 Artwork and Interaction Design

Passing transforms one high-resolution monorail recording by resampling it as a spatiotemporal volume, then uses viewer presence to influence playback among rendered trajectories. A camera-mediated loop passes the selected visual sequence to real-time sound synthesis.

3.1 Reconstructing Spacetime from a Continuous Recording

  • Passing stacks frames from a continuous 4K, 480-fps monorail recording into a volume, then shifts and rotates cross-sections to resample its spatial and temporal structure.Different cuts can suspend, stretch, reverse, or displace latent movements; varied trajectories are pre-rendered and connected into an endless playback structure.Source: §3.1 · p. 3

3.2 Audience-Mediated Sequence Reordering

  • Viewer presence probabilistically enables transitions among video sequences; the viewer affects sound indirectly by changing the visual flow interpreted by the model.The altered image offers the sound model changes in motion direction, speed, event density, and transition rhythm.Source: §3.2 · p. 4

3.3 Installation Loop

  • A camera sends viewer-presence state to playback control, which selects a rendered clip and its conditioning sequence for synchronized stereo synthesis.The installation displays 4K HDR video at 120 fps; camera imagery is not recorded or stored, and the selected clip’s sound is routed through an audio interface and loudspeakers.Source: §3.3 · p. 4, Figure 3 · p. 4

4 Real-time Sound Generation

Passing uses SpecMaskFoley for synchronized, real-time sound generation from nonlinear video, with a model configuration adapted for the installation. Precomputed visual conditioning supports continuous generation, while exhibition tests informed the balance of synchronization, fidelity, and naturalness.

4.1 Model Architecture and Control

  • SpecMaskFoley [2] uses SpecMaskGIT [21] as its audio backbone, with ControlNet and FT-Aligner adapting temporal video features for synchronized sound generation.For Passing, the CLIP visual encoder was omitted; ControlNet provides temporal alignment and CLAP-based audio conditioning supplies semantic guidance.Source: §4.1 · p. 5, §4 · p. 5

4.2 Training Data and Adaptation

  • SpecMaskFoley was fine-tuned from SpecMaskGIT using VGGSound and approximately 35 min of custom monorail audiovisual data.Augmentation used spatial video crops, audio-channel manipulations, and temporally reversed pairs that preserved synchronization; source separation removed speech components from the training datasets to reduce unintended human-like voices.Source: §4.2 · p. 5

4.3 Real-time Deployment

  • Real-time generation lets the model synthesize sound anew for each moment of the nonlinear loop instead of replaying prepared audio.Rendered 4K HDR video ran at 120 fps, while precomputed 224 × 224 conditioning images ran at 25 fps, avoiding runtime feature extraction from displayed frames.Source: §4.3 · p. 5
  • The final installation generated 32-kHz stereo audio in real time, with stable train-interior atmosphere in A/B loops and stronger sonic variation in C/D transitions.The artist’s on-site stability tests and listening checks informed the final balance among synchronization, fidelity, and naturalness; displayed video and conditioning inputs were handled separately.Source: §4.3 · p. 5

5 Discussion

Passing mediates audience influence through changes to the visual path, while real-time sound responds to the resulting nonlinear flow. The work treats this instability and the model’s limitations as part of a negotiated, relational soundscape.

5.1 Mediated Observation and Listening

  • The viewer’s presence changes the visual path the model receives, rather than directly controlling sound through volume, pitch, or timbre.The artist defines the recording, transformation rules, and system constraints.Source: §5.1 · p. 6
  • The landscape remains recognizable as depth, speed, viewpoint, and temporal order become inconsistent; generated sound varies more in the distorted C/D transitions than in the stable A/B loop.Sound is synthesized in real time from the displayed visual flow rather than attached as a prerecorded track.Source: §5.1 · p. 6

5.2 Negotiated Soundscape and Relational Authorship

  • The artist iteratively adjusted generation parameters and balanced fidelity, synchronization, and naturalness to shape when the model’s inference could become artistically meaningful.The goal was neither a realistic soundtrack nor direct sonification of visual parameters.Source: §5.2 · p. 6
  • Because the model lacks a body and lived spatial experience, its mismatches and ambiguities are treated as productive frictions within conditional, relational creative agency.The model responds to visual motion and temporal change.Source: §5.2 · p. 6

6 Conclusion

Passing reconstructs a continuous recording as an endless audiovisual journey and uses real-time video-to-audio synthesis to explore listening in disrupted space and time. Its framework distributes authorship among artist, AI, and audience, within an exhibition-scale system where their contributions mutually condition one another.

6 Conclusion

  • Passing reconstructs a single continuous recording’s spacetime to generate an endless audiovisual journey.The work uses a real-time video-to-audio model to explore how an AI without a body responds when homogeneous space and linear time collapse.Source: §6 · p. 6
  • The framework negotiates authorship among the artist designing the world’s rules, the AI proposing real-time auditory interpretations, and the audience influencing the audiovisual path and constructing personal meaning.The paper presents Passing as a case study of creative agency redistributed through AI collaboration rather than automated or diminished.Source: §6 · p. 6
  • Passing integrates real-time video-to-audio synthesis into an exhibition-scale system where visual reconstruction, machine inference, and audience presence mutually condition one another during live playback.Source: §6 · p. 6

A.1 SpecMaskFoley Configuration and Architectural Modifications

The Passing configuration adapts SpecMaskFoley for interactive sound generation by combining ControlNet temporal alignment with CLAP semantic conditioning. It omits the CLIP visual encoder and prepares conditioning images in advance to support low-latency installation playback.

A.1 SpecMaskFoley Configuration and Architectural Modifications

  • In the Passing configuration, ControlNet provides temporal alignment while CLAP-based conditioning supplies semantic guidance through a conditional mask.Figure 4 distinguishes frozen modules with ice icons and trainable modules with fire icons.Source: Figure 4 · p. 9, §A.1 · p. 9
  • The CLIP visual encoder was omitted so semantic conditioning could rely on CLAP and ambient Tama Monorail recordings, anchoring sound in train-interior context while responding to reconfigured visual motion.The CLAP conditioning was not intended to reconstruct the original monorail soundtrack.Source: §A.1 · p. 9
  • Removing the CLIP visual encoder addressed its real-time inference bottleneck, supporting stable, low-latency synthesis for the evolving installation video stream.The final architecture uses ControlNet for temporal alignment and CLAP for semantic guidance.Source: §A.1 · p. 9
  • The system cached visual-conditioning images prepared at 224 × 224 pixels and 25 fps before playback, avoiding per-frame processing of the displayed 4K 120-fps video.The 32 kHz tokenizer, audio backbone, and vocoder were trained on AudioSet and VGGSound before adaptation with custom Tama Monorail recordings.Source: §A.1 · p. 9

A.2 Comparison of Video-to-Audio Synthesis Performance

The appendix compares SpecMaskFoley with representative video-to-audio methods on the VGGSound test set using audio-quality and synchronization measures. The comparison contextualizes the model’s suitability for real-time use in Passing, not the quality of Passing’s final soundscape.

A.2 Comparison of Video-to-Audio Synthesis Performance

  • FAD measures audio quality and DeSync measures temporal alignment on the VGGSound test dataset, with lower values better for both.Source: §A.2 · p. 10, Figure 5 · p. 10
  • The comparison supports SpecMaskFoley’s suitability as a real-time V2A component, rather than evaluating Passing’s final soundscape.The model adapts a pretrained SpecMaskGIT backbone through a ControlNet branch and a frequency-aware temporal feature aligner.Source: §A.2 · p. 10
  • SpecMaskFoley achieves competitive audio quality and synchronization performance without from-scratch V2A training.The comparison is against representative methods on the VGGSound test dataset.Source: Figure 5 · p. 10, §A.2 · p. 10

B.1 Digital Coordinate Transformation

Passing treats recorded footage as a spatiotemporal volume and changes how it is sliced over time. Trajectory variants expose altered motion while preserving the landscape’s overall right-to-left flow.

B.1 Digital Coordinate Transformation

  • Changing the slicing plane’s position and angle exchanges, stretches, or reverses spatial and temporal dimensions in images extracted from the recorded volume.Ordinary playback advances a cross-section through the volume in the original frame order.Source: §B.1 · p. 11
  • The transformation destabilizes depth, speed, viewpoint, and temporal order while retaining the visual continuity of recorded scenery.It reveals motion traces as simultaneity within a frame collapses, without synthesizing a new landscape.Source: §B.1 · p. 11

B.2 Cross-sectional Trajectory Generation

  • Each trajectory specifies the slicing plane’s path through the volume rather than replaying original frames in recorded order.Source: §B.2 · p. 11
  • Trajectories vary the slicing plane’s departure timing and rotational speed, changing how train, object, and local movements appear.Movements may appear suspended, stretched, reversed, or displaced.Source: §B.2 · p. 11
  • Trajectories preserve the landscape’s right-to-left flow, including when horizontal inversion is combined with reverse playback.This keeps the passing-scenery perception connected to bodily experience despite non-linear internal timing.Source: §B.2 · p. 11
  • The transformed clips are pre-rendered, reorganizing spatial and temporal structures already present in the footage rather than generating a landscape from scratch.Source: §B.2 · p. 11

B.3 Sequence Groups and Transition Graph

The installation organizes pre-rendered transformations of one monorail recording into default and branching sequences. Viewer presence can enable branches, while the combined playback structure sustains an endless loop from finite footage.

B.3 Sequence Groups and Transition Graph

  • The visual dataset derives from one continuous Tama Monorail window recording of the approximately 20-minute trip from Tachikawa to Tama Center.The source was captured at 4K 480-fps HDR; transformed clips were pre-rendered as 4K HDR for 120-fps playback.Source: §B.3 · p. 11
  • Groups A and B are the primary sequence groups; C and D provide distorted transition variations between forward and reverse-playback, mirrored material.A contains 131 approximately 10-second clips, B contains 131, C contains 58, and D contains 52.Source: §B.3 · p. 11
  • The default loop plays Group A forward in order, then Group B in reverse order, returning to A1.This alternates forward and reversed temporal conditions while maintaining right-to-left landscape flow.Source: §B.3 · p. 12
  • When a viewer is detected, the system may select a C or D branch; after a C clip it returns to B, and after D it returns to A.Without detected presence or a selected branch, playback continues on the default path; recently played branch clips are excluded for approximately 60 minutes.Source: §B.3 · p. 12, Figure 6 · p. 12
  • Combining the default A/B progression with presence-enabled C/D branches creates an endless video loop from a finite source recording.Different sessions form different paths through the pre-rendered transformations, and extended viewing is unlikely to repeat the same transition.Source: §B.3 · p. 12
Loading 2609.27489v1…