Source-linked AI summary
Active Inference for Slow and Fast Interaction Loops in Holographic-Type Communication
Kamran Fotovat, Kian Fotovat, Nika Hosseini, Martin Maier
TL;DR
Holographic-type communication needs new holograms when viewers leave a narrow viewing zone, but image-to-3D reconstruction can take tens of seconds. The paper separates mesh reconstruction from viewpoint rendering and uses active inference to allocate reconstruction. In a modeled 100-event session, this reduces mean latency from 14.8 to 4.4 s per event, and anticipation lowers it further to 1.7–2.9 s.
Problem
Viewpoint changes beyond a display’s narrow angular zone require new holograms, while image-to-3D reconstruction can take tens of seconds per asset.
Source: §I · p. 1Method
A slow loop reconstructs a persistent mesh on content changes, while a fast loop renders viewpoints from it; active inference allocates reconstruction timing, scope, and computation.
Source: §I · p. 1, §II · pp. 1–2Results
In a modeled 100-event session, mean latency fell from 14.8 to 4.4 s per event with loop separation, then to 1.7–2.9 s with anticipation.
Source: §IV · p. 4Takeaways & Limitations
Separating viewpoint rendering from reconstruction leaves content changes as the main latency cost, which the controller can anticipate and schedule.
Source: §VI · p. 4
Abstract
from arXiv · showhide
Holographic-type communication requires a new hologram when the viewer moves beyond the display's angular viewing zone, whereas image-to-3D reconstruction takes tens of seconds per asset. We separate these operations into a slow loop that reconstructs a persistent mesh and a fast loop that renders each viewpoint from the stored mesh. The fast loop takes 0.117 s independently of camera motion, compared with 13.8-14.8 s for a full reconstruction. An active-inference controller determines when to begin reconstruction from learned user-state transitions, whether to perform a full or regional update, and how much computation to allocate based on a monocular-depth detail measure. Regional updates preserve geometry outside the edited area. In a modeled 100-event session, separating the loops reduces mean latency per event from 14.8 to 4.4 s (3.4-fold); predictive reconstruction reduces it further to 1.7-2.9 s, depending on user routine strength.
I. INTRODUCTION
Because the display’s narrow angular zone makes viewpoint updates necessary while image-to-3D reconstruction takes tens of seconds, reconstructing for every view is impractical. The paper separates viewpoint rendering from content reconstruction and uses active inference to allocate reconstruction timing, scope, and computation.
I. INTRODUCTION
- Viewpoint changes require a new hologram outside the display’s narrow angular zone, but image-to-3D generators can take tens of seconds per asset.The problem is one of timing and control as well as optics: a new hologram must fit an interactive budget.Source: §I · p. 1
- Content changes, unlike viewpoint changes, require reconstruction, and their cost varies with edit extent and geometric detail.Beginning reconstruction during an idle period may also reduce the work left when a change is requested.Source: §I · p. 1
- The proposed architecture reuses a persistent mesh across viewpoints and reconstructs it when content changes.An active-inference controller anticipates likely changes, chooses full or regional reconstruction, and adjusts computation to geometric detail within one expected-free-energy formulation.Source: §I · p. 1
II. SYSTEM MODEL: SLOW AND FAST INTERACTION LOOPS
The system separates expensive mesh reconstruction from deterministic viewpoint rendering: the slow loop updates a persistent mesh, while the fast loop renders views from it. This makes viewpoint updates inexpensive and leaves content changes as the main reconstruction cost.
II. SYSTEM MODEL: SLOW AND FAST INTERACTION LOOPS
- A viewpoint change needs no new asset: deterministic rendering and propagation reused one mesh across 720 frames with no measurable degradation.The mesh is viewpoint-invariant; the fast loop renders intensity and depth from a new pose and propagates the field.Source: §II · p. 1
- Across six camera trajectories, median fast-loop latency varied by 2.9%, within 2.0–2.8% frame-to-frame variation, so no motion dependence was resolvable.Fresnel propagation dominates cost, which is set by the modulator rather than the scene.Source: §II · p. 1, Figure 2 · p. 2
- 0.117 s is the median viewpoint-update time, with the fast loop rendering from the stored mesh rather than invoking reconstruction.The pipeline assigns reconstruction to content changes and uses view rendering and Fresnel propagation for each viewpoint.Source: §II · pp. 1–2, Figure 1 · p. 2
- A viewpoint update is 118 times cheaper than the least expensive full refresh, while regional updates are about 1.4 times cheaper in wall time and 2.5–3.0 times cheaper in computation.Wall time includes a near-constant 8.1–8.6 s weight-transfer cost, in addition to computation that varies with step count.Source: §II · p. 2
- Regional reconstruction is expressible because TRELLIS uses a dense spatial grid whose tensor entries can be selected by region, and its stage step budgets are adjustable.The backend is selected for controllability rather than raw speed.Source: §II · p. 2
III. THE ACTIVE-INFERENCE CONTROLLER
The paper uses active inference because the controller must adapt to a new user without a training budget, while keeping latency and fidelity trade-offs explicit. It learns user transitions through a Dirichlet update that is inspectable and adapts within a few dozen events.
III. THE ACTIVE-INFERENCE CONTROLLER
- The controller has no training budget because it faces a new user from the first interaction, and exploratory actions cost that user waiting time.This motivates adapting from observed transitions rather than relying on prior user-specific training.Source: §III · p. 2
- Active inference keeps expected latency, residual geometric error, and transition counts explicit instead of compressing latency and fidelity into an engineered reward.The paper identifies this explicit trade-off as a reason for its controller choice.Source: §III · p. 2
- A Dirichlet update learns the user’s transitions, adapts within a few dozen events, and is inspectable with one interpretable rate.The update uses a learning rate and forgetting rate.Source: §III · p. 2
A. Detail Demand: Measured Before Reconstruction
The controller estimates geometric detail from a monocular depth map before reconstruction, avoiding image appearance as a proxy. Its detail score distinguishes objects and informs reconstruction step budgets.
A. Detail Demand: Measured Before Reconstruction
- A monocular depth map is converted to surface normals, band-passed, and scored by the fraction of surface carrying fine normal variation.Depth excludes albedo, while surface coverage prevents a few silhouette edges from standing in for detail spread across an object.Source: A. Detail Demand: Measured Before… · p. 2
- The geometric score ranks a carved bust at 0.133 and a dragon at 0.116, above the unornamented bust at 0.093 and apple at 0.060.The measure costs about 0.05 s, compared with tens of seconds for reconstruction.Source: A. Detail Demand: Measured Before… · p. 3
- The controller assigns 16 steps to a detailed object, 12 to a simple one, and 8 or 6 to regional updates.The accompanying habit-learning figure compares three routine strengths and four forgetting rates; faster forgetting recovers sooner after a routine change, while ω = 0.50 lowers the pre-change plateau below its ceiling.Source: A. Detail Demand: Measured Before… · p. 3, Figure 3 · p. 3
B. Scope: Regional Update
Regional reconstruction regenerates only the edited spatial region while retaining the surrounding geometry. The reported Chamfer distances show substantially greater preservation outside the edit, and cached geometry can shorten later updates.
B. Scope: Regional Update
- A spatially indexed dense grid lets a region be regenerated alone by re-noising cached latents and preserving the original latents outside the mask at each denoising step.The generator still attends to the surroundings, helping the new surface meet the old geometry.Source: B. Scope: Regional Update · p. 3
- Chamfer distance was 3.9 × 10−4 outside the edited region and 1.0 × 10−2 inside it, a 26-fold difference.The region is localized by a content-change difference map that is nonzero where the object no longer matches the mesh.Source: B. Scope: Regional Update · p. 3
- A high-budget asset retains its quality wherever it was not edited, while starting from a converged latent takes 6–8 steps against 12–16.This lets reconstruction quality be amortized over later updates.Source: B. Scope: Regional Update · p. 3
C. Timing: Anticipating the Refresh
The controller predicts the next action from learned user-state transitions so reconstruction can begin during an idle period. It updates its transition model over time and uses the posterior to inform policy choice.
C. Timing: Anticipating the Refresh
- The controller learns the user’s transition matrix through Bayesian updating under a Dirichlet prior with concentration counts.The policy prior is learned from transitions, and the controller adapts within a few dozen events.Source: C. Timing: Anticipating the Refresh · p. 3
- Setting the forgetting rate to ω = 0.90 raises accuracy over the 200 events after a routine change by a third for under 1% of steady-state accuracy.Faster forgetting rates erode steady-state accuracy.Source: C. Timing: Anticipating the Refresh · p. 3
- The speculative policy prior uses the transition posterior, with the user’s global marginal providing backoff for rarely visited states and reserved mass for unseen successors.The backoff strength is η, and α reserves mass for an unseen successor.Source: C. Timing: Anticipating the Refresh · p. 3
- Predicting the next action rather than a future-time state gives approximately 0.73 Bayes-optimal top-1 accuracy at a one-event horizon, versus approximately 0.19 at 13–24 steps.The transition matrix represents the probability of the next object state given the current state.Source: C. Timing: Anticipating the Refresh · p. 3
D. One Objective Over the Decisions
The expected-free-energy objective scores a policy’s timing, scope, and computation budget together. Its latency risk governs timing and scope, while residual geometric error governs the budget choice.
D. One Objective Over the Decisions
- A policy is the triple (when, scope, budget), and expected free energy scores all three decisions together.Risk carries timing and scope; ambiguity carries the computation-budget decision.Source: D. One Objective Over the Decisions · p. 3
- Risk measures deviation from the preference for low latency, while ambiguity is residual geometric error ε left by n steps on an object of detail d.The model omits an epistemic term because exploratory actions cost users waiting time and the model learns passively from events.Source: D. One Objective Over the Decisions · p. 3
- Expected latency uses predictor accuracy p, operation cost T, remaining idle time D, and the cost t_swap of adopting a finished result.The latency expression is never worse than not speculating because a miss costs exactly T; partial progress is retained when the guess is correct.Source: D. One Objective Over the Decisions · p. 3
- The modeled session contains 65% viewpoint, 18% regional, and 17% full events, with predictor accuracies of 0.39, 0.53, and 0.68 for the three routine strengths.The figure reports mean latency per event over a 100-event session and uses D = 12 s as its operating-point readout.Source: Figure 4 · p. 4
IV. RESULTS
The results show latency reductions from separating viewpoint rendering from reconstruction, with further savings from anticipating content changes; they also evaluate prediction across synthetic user-transition processes.
IV. RESULTS
- Predictor evaluation uses synthetic 12-state Markov processes with known transition matrices and scores each event before learning from it.This prequential evaluation sweeps hit rate rather than fitting it.Source: §IV · p. 4
- Eq. (4) reaches 95–100% of the closed-form Bayes-optimal one-step ceiling on first-order chains, converging within 200–400 transitions.Across second-order chains, semi-Markov dwells, drift, and cyclic workflows, it recovers 97–100% of the first-order optimum on every family and 208% under drift.Source: §IV · p. 4
- 14.8 s per event falls to 4.4 s with dual loops, then to 2.9 s for weakly routine and 1.7 s for strongly routine users with anticipation.The modeled session contains 100 events; the dual-loop reduction is a factor of 3.4.Source: §IV · p. 4
- At D = 0 s, all habit curves meet the dual-timescale line at 4.4 s; at D = 20 s, they reach 2.7, 2.1, and 1.5 s.The curves flatten once D exceeds the cost being hidden; the 12 s operating point is on their steep portion.Source: §IV · p. 4
- At D = 8 s, anticipation returns 1.45× on a full refresh, 1.73× on a regional update, and 2.18× at a routine user.The passage attributes the combined gain to the interaction between scope and timing through D/T.Source: §IV · p. 4
V. FUTURE WORK
Future work targets a calibrated reconstruction-error model and extensions that exploit overlap between the two loops and broaden the deployment setting.
V. FUTURE WORK
- Calibrating ε(n, d) against step count and detail demand could replace discrete budget selection with direct allocation and expose a continuous fidelity–latency trade-off.The current decision selects among the budgets in Table I.Source: §V · p. 4
- Running the slow loop continuously behind viewpoint rendering could increase the fraction of content changes already in progress when requested.The two loops can be scheduled against one another because viewpoint updates do not invoke the generator.Source: §V · p. 4
- A resident-model deployment could widen the separation between budgets, while faster hardware compounds with the controller as T shrinks.The present wall-time difference between budgets is 0.08 s, compared with a 3.1-fold spread in computation.Source: §V · p. 4
- The predictor’s transition learning over an abstract user state is described as extending unchanged to multi-user sessions.Source: §V · p. 4
VI. CONCLUSION
The conclusion attributes lower HTC interaction latency to separating viewpoint rendering from reconstruction and coordinating reconstruction timing, scope, and budget with active inference.
VI. CONCLUSION
- A viewpoint update takes 0.117 s, compared with 13.8–14.8 s for a full reconstruction.The fast loop renders viewpoints from a persistent mesh updated by the slow loop when content changes.Source: §VI · p. 4
- The active-inference controller coordinates when to reconstruct, whether to use full or regional reconstruction, and how much computation to allocate.The three decisions are made against one objective rather than tuned independently.Source: §VI · p. 4
- In the modeled 100-event session, mean latency falls from 14.8 to 4.4 s per event, then to 1.7–2.9 s with anticipation.The reported gains are established component-wise and through the latency model of (6), so the ratios are properties of the structure rather than any one deployment.Source: §VI · p. 4