PaperlayerView exact revision on arXiv

Source-linked AI summary

GenStream: Semantic Streaming Framework for Generative Reconstruction of Human-centric Media

Emanuele Artioli, Daniele Lorenzi, Shivi Vats, Farzad Tashtarian, Christian Timmerer

arXiv:2609.18634v1cs.CVcs.AIcs.MM

TL;DR

Conventional codecs repeatedly encode pixel-level content, including persistent backgrounds, without using scene semantics. GenStream transmits keypoints, camera viewpoints, and a 3D venue representation for client-side generative reconstruction, reporting over 99.9% lower bandwidth than HEVC for the continuous stream. The current proof of concept shifts substantial computation to the client.

  • Problem

    Conventional codecs treat pixels uniformly and repeatedly encode similar content, limiting their use of structural redundancy in human-centric scenes.

    Source: §1 · p. 1

  • Method

    GenStream transmits performer keypoints and camera viewpoint metadata for client-side generative reconstruction and compositing into a pre-scanned 3D venue.

    Source: §1 · p. 2

  • Results

    Over 99.9% bandwidth reduction compared to conventional HEVC encoding is reported for GenStream’s continuous data stream.

    Source: §3 · p. 4, §6 · p. 7

  • Takeaways & Limitations

    GenStream provides a blueprint for streaming that prioritizes semantic content over pixel fidelity while preserving essential perceptual cues.

    Source: §6 · pp. 7–8

    The proof of concept requires a powerful client GPU, unlike typical smartphones or smart TVs, making client-side computational burden a major limitation.

    Source: §4.3 · p. 5

Abstract

from arXiv · show

Video streaming dominates global internet traffic, yet conventional pipelines remain inefficient for structured, human-centric content such as sports, performance, or interactive media. Standard codecs re-encode entire frames, foreground and background alike, treating all pixels uniformly and ignoring the semantic structure of the scene. This leads to significant bandwidth waste, particularly in scenarios where backgrounds are static and motion is constrained to a few salient actors. We introduce GenStream, a semantic streaming framework that replaces dense video frames with compact, structured metadata. Instead of transmitting pixels, GenStream encodes each scene as a combination of skeletal keypoints, camera viewpoint parameters, and a static 3D background model. These elements are transmitted to the client, where a generative model reconstructs photorealistic human figures and composites them into the 3D scene from the original viewpoint. This paradigm enables extreme compression, achieving over 99.9% bandwidth reduction compared to HEVC for the continuous data stream. We partially validate GenStream on Olympic figure skating footage and demonstrate potential for high perceptual fidelity under minimal data. While acknowledging the significant computational costs shifted to the client and challenges in generalization, GenStream opens new directions in volumetric avatar synthesis, canonical 3D actor fusion across views, and personalized viewing experiences, laying the groundwork for scalable, intelligent streaming in the post-codec era.

1 Introduction

GenStream replaces pixel-based streaming with semantic scene metadata and client-side generative reconstruction. Preliminary results report over 99.9% lower live-streaming bitrate than HEVC, under a pre-distribution assumption.

1 Introduction

  • Video streaming accounts for the majority of global Internet data traffic, while demand for immersive and interactive content is growing.The introduction notes that major live events deliver high-definition streams simultaneously to millions of viewers.Source: §1 · p. 1
  • Segment-based codecs repeatedly encode similar content because they reference only frames within each segment, wasting bandwidth on persistent backgrounds.The passage also identifies independently encoded views in multi-camera productions as an added source of redundancy.Source: §1 · p. 1
  • GenStream sends performer keypoints and camera viewpoint data for client-side generation and compositing into a pre-scanned 3D venue.The framework also proposes personalized performer styles and viewpoints.Source: §1 · p. 2
  • Over 99.9% lower live-streaming bitrate than baseline HEVC is reported when the venue model and generative-model weights are pre-distributed to clients.The framework shifts computational complexity to client-side synthesis.Source: §1 · p. 2

2 Related Work

Related work spans conventional codecs, learned and volumetric encoding, semantic streaming, and generative reconstruction. The paper positions GenStream at the intersection of traditional streaming, semantic video analysis, and generative synthesis.

2 Related Work

  • The paper positions its work at the intersection of traditional video streaming, semantic video analysis, and generative synthesis.It also draws inspiration from volumetric and model-based media transmission.Source: §2 · p. 2

2.1 Traditional Video Streaming and Codecs

  • Traditional codecs compress spatial and temporal redundancy but treat pixels without regard to scene semantics, with diminishing returns and rising complexity.The cited discussion notes increasing encoder-side energy and latency burdens for high-resolution or real-time use.Source: §2.1 · p. 2

2.2 Emerging Architectures for Video Encoding

  • Learned codec enhancements remain tied to legacy structures, while neural compression can be too computationally intensive for real-time or resource-limited use.Volumetric point-cloud, NeRF, and 3DGS work explores structured scene representations, while interpolation and super-resolution enhance received video without increasing bitrate.Source: §2.2 · p. 2

2.3 Object-Centric Representations and Semantic Streaming

  • Semantic streaming and object-centric tools separate salient agents from backgrounds, but broader entertainment, sports, and interactive streaming remains largely pixel-based.SemConf and related conferencing systems demonstrate generative reconstruction from sparse metadata in face-to-face scenarios.Source: §2.3 · pp. 2–3

2.4 Generative Reconstruction

  • Generative models can reconstruct photorealistic video from abstract inputs such as pose skeletons, semantic maps, or depth, supporting sparse-input streaming.Camera-motion estimation and neural scene representations offer routes to view-consistent 3D backgrounds and separately reconstructed agents, environments, and motion.Source: §2.4 · p. 3

3 Problem Description and Motivation

For figure-skating footage, the section frames reconstruction as combining a persistent 3D venue with dynamic skater representations and camera viewpoint data. Its estimate reports approximately 110 kB of stream data, while excluding one-time venue and model distribution.

3 Problem Description and Motivation

  • The evaluation uses a 50-second, 30-fps, 1080p ice-skating scene encoded with HEVC at 5.8 Mbps as its reference.The motivation is that skating footage typically combines a static or slowly evolving rink and stadium background with one or two skaters.Source: §3 · p. 3
  • Camera viewpoint T combines rotation R and camera center t; converting R to a quaternion reduces its representation from 12 values to 7.With the viewpoint and stadium point cloud, the client can reconstruct the background.Source: §3 · p. 3
  • Pixel crops of skaters create variable resolutions, synchronization challenges, or inefficient padding, motivating compact skeletal keypoints instead.The proposed scene representation includes keypoints, bounding boxes, and camera-viewpoint parameters.Source: §3 · pp. 3–4
  • Approximately 110 kB of transmitted data is reported, a 99.9% bandwidth reduction versus the original 134 MB HEVC video.The calculation covers the continuous stream only; venue-model and generative-weight distribution is a significant initial payload, and formal perceptual study remains future work.Source: §3 · p. 4

4 System Design

GenStream treats video as a generative instruction set rather than a sequence of pixels, combining object understanding, 3D scene modeling, and generative synthesis. Its server pipeline creates semantic layers from actors and backgrounds.

4 System Design

  • GenStream replaces server-side pixel transmission with a sparse generative codebook built from object understanding, 3D scene modeling, and generative synthesis.Figure 1 depicts separate server-side and client-side pipelines.Source: §4 · p. 4

4.1 Server-side GenStream

  • The server transforms raw video into dynamic object keypoints, background images, and camera viewpoints through four stages.The first stage detects and segments salient actors, selecting only the largest detected human per frame to reduce spectator misclassification; its mask yields foreground and actor-masked background RGBA layers.Source: §4.1 · p. 4, Salient Actor Extraction · p. 4

Salient Actor Extraction.

GenStream filters frames to salient actors, encodes their motion as keypoint trajectories, and uses actor-free backgrounds to support camera estimation and scene reconstruction.

Salient Actor Extraction.

  • Segmentation acts as an attention filter, retaining perceptually critical content while discarding clutter, crowds, and occlusions.Source: Salient Actor Extraction · p. 4
  • High-resolution actor keypoint trajectories form the structured motion encoding that guides client-side generative synthesis.Pose-estimation models track anatomical keypoints and encode them into a temporal stream.Source: Keypoint-based Pose Encoding · p. 4

Keypoint-based Pose Encoding.

  • Actor-free backgrounds are inpainted to support Structure-from-Motion camera estimation and coherent 3D scene reconstruction.This step is essential when organizers have not provided the venue's 3D model and camera viewpoints.Source: Background Inpainting and 3D… · p. 5

Background Inpainting and 3D Reconstruction.

GenStream packages keypoints and camera coordinates as metadata, then reconstructs full-body actors client-side with a conditional generative model.

Background Inpainting and 3D Reconstruction.

4.2 Client-side GenStream

Skeleton-Driven Actor Reconstruction.

The client composites synthesized actors into a 3D background using camera viewpoints and spatial metadata, producing video for playback.

Viewpoint-Aware Scene Recomposition.

  • Synthesized actors are reinserted frame by frame into the 3D background using camera viewpoints, spatial metadata, and evolving keypoints.The background may be custom-made and pre-sent or extracted by the camera-viewpoint inference algorithm.Source: Viewpoint-Aware Scene Recomposition · p. 5
  • The reconstructed sequence is sent to the player for smooth, low-latency playback, and the system is described as robust to bandwidth drops, jitter, or lost intermediate frames.The authors characterize it as more like a generative animation system than a classic streaming system.Source: Frame Synthesis and Playback · p. 5

Frame Synthesis and Playback.

GenStream replaces pixel blocks with poses, separates actor and background processing, and uses client-side generation to reduce bandwidth; this shifts substantial computation to the client.

4.3 Design Summary

  • GenStream replaces pixel blocks with poses, separates actor and background processing, and uses client-side generative models to recreate known figures while reducing bandwidth.Source: §4.3 · p. 5
  • The proof of concept requires a powerful client GPU, unlike typical smartphones or smart TVs, making client-side computation a major limitation.The authors suggest quantization, distillation, or edge-server offloading as future ways to mitigate the cost.Source: §4.3 · p. 5

5 Implementation

GenStream’s implementation separates the skater from the background, extracts actor keypoints, and inpaints removed regions to support scene reconstruction. The system was tested on Olympic figure-skating footage, while camera viewpoint recovery remained under scrutiny.

5 Implementation

  • The test set comprised over 40,000 frames in 48 scenes from four 2018 Olympic figure-skating performances, sharing a mostly stationary rink and one moving skater.Scenes lasted 9–55 seconds and were recorded at 30 fps and 1080p.Source: Content · p. 5

Content.

  • The Python pipeline ran on an Ubuntu server with a Xeon Gold 5218 CPU and Quadro RTX 8000 GPU; actor segmentation, keypoint extraction, inpainting, and data export were tested independently.Camera viewpoint recovery posed challenges and remained under scrutiny.Source: Setup · p. 5

Setup.

  • YOLOv11 detects people and the largest bounding box is retained; SAM 2 supplies a fine mask, while single-pass YOLOv11 offers faster but less robust segmentation.The mask separates actor and background images, enabling actor-specific compression and background reconstruction.Source: Actor Segmentation and Frame… · pp. 5–6

Actor Segmentation and Frame Decomposition.

  • Among Canny edges, grid sampling, and YOLOv11 Pose, GenStream selects the 17-landmark YOLO method for comparable downstream perceptual quality with a more compact representation.Keypoints are extracted from the segmented actor, avoiding background interference.Source: Keypoint extraction · p. 6

Background inpainting.

  • Inpainting fills actor-removed background regions for view synthesis; imperfect frame-to-frame consistency is tolerated because leaving moving actors can worsen COLMAP reconstruction.Removing actors helps reduce false feature matches and improve background-structure estimation.Source: Background inpainting · p. 6

3D Ice Rink Modeling.

The implementation combines a manually modeled Unity rink with partially implemented camera-recovery approaches and compact transmission of actor and camera data. It also trains a conditional GAN on paired keypoint-based and ground-truth images, while current rendering uses manually inferred camera parameters.

3D Ice Rink Modeling.

  • The Unity rink was manually reconstructed with placed lines, lighting, and reflections to match typical broadcast footage.The skater is overlaid on this 3D background in the rendering implementation.Source: 3D Ice Rink Modeling · p. 6, Client-Side Reconstruction · p. 7
  • COLMAP generates a sparse static-scene point cloud and per-frame camera intrinsics and extrinsics, but converting its coordinates and focal lengths to Unity remains unresolved.COLMAP produces plausible camera trajectories; feature fusion across scenes is needed to form a complete point cloud for pre-distribution.Source: 3D Ice Rink Modeling · pp. 6–7
  • Differentiable viewpoint optimization compares rendered views with inpainted frames, but is unsuitable for live streaming; optimization on the first frame plus SIFT-based relative poses propagates later viewpoints.The authors report this reduces computational burden while maintaining adequate tracking accuracy.Source: 3D Ice Rink Modeling · p. 7

Data export (Partially Implemented.

  • Extracted actor bounding boxes, keypoints, and camera viewpoints (or optionally COLMAP scene information) are saved to CSV and encoded for transmission; each keypoint coordinate uses 11 bits, with 2000 marking missed detections.Compressed data is delivered to the client at intervals aligned with target latency requirements of live video streaming applications.Source: Data export (Partially Implemented · p. 7

Generative Model Training Setup.

  • The conditional GAN is trained on paired keypoint-reconstructed and ground-truth person images, split 80%/20% for training and testing, with resizing, cropping, and mirroring augmentation.The paired source images are 128×128, 256×256, or 1024×1024.Source: Generative Model Training Setup · p. 7
  • Current Unity rendering places the generated skater at a matching spatial location using manually inferred camera parameters, a qualitative solution the authors say is not scalable.The renderer is configured to accept per-frame camera values once viewpoint recovery is solved.Source: Client-Side Reconstruction · p. 7

6 Conclusion

GenStream replaces dense pixel transmission with semantic instructions for client-side reconstruction, reporting over 99.9% bandwidth reduction while identifying computational and generalization challenges.

6 Conclusion

  • Over 99.9% bandwidth reduction versus conventional HEVC encoding is reported for transmitting sparse keypoints and camera metadata instead of dense pixels.The framework guides client-side generative reconstruction while preserving essential perceptual cues for immersive viewing.Source: §6 · p. 7
  • Significant client-side computational load is the most immediate challenge, motivating research into efficient generative models for resource-constrained devices.The work identifies this as a primary axis for future research.Source: §6 · p. 7
  • A future volumetric direction would replace 2D sprite-like actors with moving 3D skeletons driving volumetric avatars for viewpoint-consistent viewing.The envisioned free-viewpoint synthesis would let clients explore perspectives not captured by any camera, but requires canonical multi-scene point-cloud fusion and reliable 3D joints from single-camera 2D skeletons.Source: §6 · p. 7
  • Extending beyond figure skating to football or gymnastics would require handling richer interactions, dynamic cameras, frequent occlusions, and multi-actor synthesis.Efficiently fusing venue features into a compact unified point cloud is also important for pre-distributing 3D models to clients.Source: §6 · p. 8
  • GenStream is presented as a first step toward semantic, generative media delivery that offloads reconstruction to client-side models and transmits only salient content.The paper characterizes this as a scalable, perceptually meaningful approach for the post-codec era.Source: §6 · p. 8
Loading 2609.18634v1…